vmware-samples / vmware-architecture-migration-tool

The VMware Architecture Migration Tool (VAMT) is designed to provide an easy and automated process to cold migrate machines between clusters of different architecture types within the same vCenter or across vCenter Server instances.

VMware Architecture Migration Tool

Getting Started

Visit our QuickStart guide and explore our Table of Contents below to review the project's documentation.

Table of Contents

• Getting Started
• Purpose
• Known Limitations
• Design
• Prerequisites
• Execution
• Contributing
• License

Purpose

The VMware Architecture Migration Tool (VAMT) is designed to provide an easy and automated process to cold migrate machines between clusters of different architecture types within the same vCenter or across vCenter Server instances. In an effort to provide a useful and intuitive tool, the following features have been implemented:

• Change Window Support - Ability to schedule migration within a time frame and stop new migrations if the defined time frame is exceeded.
• Process Throttling - Control of how many parallel migrations tasks that vSphere will be asked to execute.
• Syslog Support - Ability to send logs to a syslog server.
• Email Notifications Support - A report of the logs and individual VM migration status can be sent to an email address.
• Best Effort Migrated VM Success Validation - The tool will wait for VMware Tools to load successfully after the VM has been migrated.
• Extensibility Stubs - The tool provides a stub of functions to add customization before and after VM is migrated.
• Maintain the VM's UUID - In an effort to ensure external tools still recognize the VM after migration the UUID is maintained.
• Rollback - Ability to rollback a migration after initial migration execution.

Known Limitations

Below is a list of known limitations. This by no means can be a complete list as there may be limitations that have not been considered.

• Designed to be executed from Windows only.
• Rollback process will send VM back to its original vCenter, ESXi host, resource pool, datastore(s), and network(s). Not supporting cluster for rollback allows the widest array of supported scenarios. If rollback target needs to be modified, it can be updated in the VM properties/attributes.
• Migrations will start up until the end of the change window and run until complete, possibly past the defined window.
• Cold migrations initiated by this script are constrained by the limitations and requirements of vSphere & PowerCLI.
  • For example, initiating a migration between major vSphere versions (i.e., vSphere 7 -> 8) could have special requirements or limitations that cannot be documented here.

Design

Visit Design.md to review the details of VAMT's design including:

• Overview
• Design Philosophy
• Conceptual Model
• Logic Flow Diagrams

Prerequisites

Visit Script_Prerequisites.md to review the prerequisites for running the VAMT.

Execution

Visit Script_Usage.md to review the details for running VMwareArchitectureMigrationTool.ps1.

Contributing

The vmware-architecture-migration-tool project team welcomes contributions from the community. Before you start working with this project please read and sign our Contributor License Agreement (https://cla.vmware.com/cla/1/preview). If you wish to contribute code and you have not signed our Contributor License Agreement (CLA), our bot will prompt you to do so when you open a Pull Request. For any questions about the CLA process, please refer to our FAQ. For more detailed information, refer to CONTRIBUTING.md.

License

VMware Architecture Migration Tool is available under the GPL v3.0 license. Please see LICENSE.

vmware-architecture-migration-tool's Issues

Error seen during migration: "Detected invalid tag state" during OCI testing

Describe the bug

This seems to be due to a transient error in VMware.

This is the error:
[Error] 02/22/2023 17:41:03 - VM move job for 'vm069' failed with errors:
Caught exception in migration job on line '57':
Detected invalid tag state '' on 'vm069'. This is likely the result of a concurent job running on the VM elsewhere.

Reproduction steps

1. Migrate 100 VMs between the AMD and Intel cluster
2. Randomly see this error in what appears to be due to timing

Expected behavior

Migration should succeed on all 100 VMs.

Additional context

Logs can be found in the shared jumphost on OCI at: C:\Users\opc\Desktop\VAMT\vmware-architecture-migration-tool\vamt_runlogs\20230222-173117

SslThumbprint error received when testing out multiple datastore migration

Describe the bug

Received the following error from VAMT when migrating using multiple datastores in JSON file:

[Info] 03/05/2023 21:40:40 - Successfully to set 'VAMT : failed' tag on 'Win2012r2-Kevin-1' [Error] 03/05/2023 21:40:40 - VM move job for 'Win2012r2-Kevin-2' failed with errors: Caught excecption in migration job on line '223': The property 'SslThumbprint' cannot be found on this object. Verify that the property exists and can be set.
[Info] 03/05/2023 21:40:40 - Refreshing VM 'Win2012r2-Kevin-2' object

Reproduction steps

1. Specify multiple datastores in JSON input file
2. Begin migration

Below is an illustration of the JSON file being used for the migration. With a single datastore this works.

No error:
[
{
"vmname": "Win2012r2-Kevin-1",
"target_vc": "vsphere.maas",
"target_hostpoolcluster": "green",
"target_portgroup": ["VLAN158-Green", "VLAN158-Green", "VLAN158-Green", "VLAN819-Green"],
"target_datastore": ["vsanDatastore"]
},

SslThumbprint error:
[
{
"vmname": "Win2012r2-Kevin-1",
"target_vc": "vsphere.maas",
"target_hostpoolcluster": "green",
"target_portgroup": ["VLAN158-Green", "VLAN158-Green", "VLAN158-Green", "VLAN819-Green"],
"target_datastore": ["vsanDatastore", "ATCPOC0006595VOL001_30TB"]
},

Expected behavior

Migration succeeds with multiple datastores

Additional context

This was found on WWT LaaS test setup during multi-vcenter, multi-NIC, and multi-datastore validation.

MINOR bug: Error message says CSV file when input is a JSON

Describe the bug

When testing out a migration, I accidentally used the same VM name in a JSON input file. Error message is:

`[Error] 02/23/2023 08:11:29 - The following VM names were found more than once (vm200) in the inputs CSV file.
The following VM names were found more than once (vm200) in the inputs CSV file.
At C:\users\opc\Desktop\VAMT\vmware-architecture-migration-tool\VMwareArchitectureMigrationTool.ps1:354 char:5

• throw $message
  

• ~~~~~~~~~~~~~~
  

  • CategoryInfo : OperationStopped: (The following V...nputs CSV file.:String) [], RuntimeException
  • FullyQualifiedErrorId : The following VM names were found more than once (vm200) in the inputs CSV file.`

Reproduction steps

1. Put invalid entry into JSON file
2. Attempt migration

Expected behavior

Error message should probably just say "input VM list file" or something like that.

Additional context

No response

Script runtime with > 1 hour migration duration not showing correct value

Describe the bug

[Info] 02/23/2023 15:39:11 - Script run summary:
Script start: '02/23/2023 14:23:30'
Script runtime: 15 minutes
Script completion: '02/23/2023 15:39:11'
Total VM targets in migration run: 400
VM migration jobs with final status 'Completed': 400
[Info] 02/23/2023 15:39:11 - Final report:

Reproduction steps

1. Run migration
2. Compare Script start, Script completion, an dScript runtime.

Expected behavior

Should see correct time

Additional context

No response

Dependencies on logDir and logFileNamePrefix during standalone module function testing

The following variables used in the Write-Log function located in the VAMT.psm1 PS module are not being set when functions are called standalone:
logDir
logFileNamePrefix

The function I am testing with is Initialize-VIServer, which then calls Write-Log, which has those two dependencies, so I get prompted for those values during testing of the execution.

I'm getting prompted each time multiple times in that function when it calls Write-Log when trying to call that Initialize-VIServer function from the VAMT.psm1 module. Down inside that call, it then makes calls to other functions in that module, which all work. BUT on the Write-Log cmdlet it has 2 parameters that are mandatory for the call, and those values are not passed in to the initialize function in this modular standalone use case, which leads the script to prompt for that log input both times.
Initialize function header:
function Initialize-VIServer {
param(
[Parameter(Mandatory)]
[ValidateNotNullOrEmpty()]
[String[]]$vCenters,
[Parameter()]
[PSCredential]$cred,
[Parameter()]
[String]$credentialDirectory
)
Write-Log call from within the .psm1
function Write-Log {
[CmdletBinding()]
param(
.
.
[Parameter(Mandatory)]
[ValidateNotNullOrEmpty()]
[String]$logDir,
.
.
[Parameter(Mandatory)]
[ValidateNotNullOrEmpty()]
[String]$logFileNamePrefix,
Here is the output from the run after I fixed or worked around my other bugs, where I import the module and then call the function. You can see it prompt me each time it tries to write the log, because these 2 variables are missing:
logDir
logFileNamePrefix
There are many ways we could skin the cat, but in general that is really one of the more useful modular functions. I'm fine with whatever approach you want to use, but some that came to mind are that 1) we could add a default log location and prefix down in the call, 2) we could add a parameter to skip logging altogether, 3) we could remove mandatory from the arguments and then use their presense as the trigger not to log if they are null, etc. My inclination is to make logging optional, but let me know what you think. I've created a branch off main for testing this, so can try out fixes as needed, if you want to just direct me in what you feel is the right way.
PS C:\users\admin\test\vamt-tools> .\connect.ps1

cmdlet Write-Log at command pipeline position 1
Supply values for the following parameters:
logDir: C:\users\admin\test
logFileNamePrefix: test

cmdlet Write-Log at command pipeline position 1
Supply values for the following parameters:
logDir: C:\users\admin\test
logFileNamePrefix: test

cmdlet Write-Log at command pipeline position 1
Supply values for the following parameters:
logDir: C:\users\admin\test
logFileNamePrefix: test
[Info] 09/07/2022 11:40:38 - Saved new credential for 'vsphere.maas'. User: '[email protected]'

cmdlet Write-Log at command pipeline position 1
Supply values for the following parameters:
logDir: C:\users\admin\test
logFileNamePrefix: test
[Info] 09/07/2022 11:40:44 - Logging in to vCenter vsphere.maas with User: [email protected]
Removing all snapshots in VAMT-Demo-Kevin

Add additional input parameters for tuning script

Is your feature request related to a problem? Please describe.

Would like to be able to tune some more of the internal parameter knobs. Note that the defaults below are fine as defaults, but would like to be able to tweak these for different environments:

$Script:vamtOsShutdownTimeout = 600 #seconds
$Script:vamtOsPowerOnTimeout = 900 #seconds
$jobControllerRefreshInterval = 15 #seconds

Describe the solution you'd like

Add parameters for CLI like this one for the requested values:

vmware-architecture-migration-tool/VMwareArchitectureMigrationTool.ps1

Line 62 in c03a2b8

[Parameter()] <# number of retries that should be attempted if the VM has active vCenter Tasks preventing the migration #>

in for the variables shown.

Describe alternatives you've considered

No response

Additional context

No response

"VM move job for '<VM>' ended with unsupported state Blocked" error hit when running PS/PCLI script on Windows 22H2/2H21 hosts

##UPDATE: Was able to reproduce on new 21H2 build of Windows 10, so apparently it is not just the version. Possibly the PowerCLI vs. PowerShell version or something, but am able to reproduce as described later.

Describe the bug

"VM move job for '' ended with unsupported state Blocked" error hit when running PS/PCLI script on new Windows 22H2 host, not seen on older Windows 21H2 host.

Problem was reported when setting up a new VAMT Windows execution host at customer site, was able to reproduce the problem in our lab.

Followed the execution policy settings here: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.security/set-executionpolicy?view=powershell-7.3

As well as the debug info below:
The error message "Command execution is blocked" in PowerShell typically occurs when PowerShell's execution policy restricts the running of scripts and/or the execution of certain commands. PowerShell execution policies are designed to help prevent the execution of malicious scripts and ensure system security.

There are different execution policies in PowerShell, each with varying levels of restriction:

Restricted: This is the default execution policy. It doesn't allow the execution of scripts or configuration files.
AllSigned: Only signed scripts from trusted publishers are allowed to run.
RemoteSigned: Local scripts can run, but scripts downloaded from the internet must be signed by a trusted publisher.
Unrestricted: All scripts can run without any restrictions. This is not recommended unless you trust all the scripts you run.
Bypass: No execution policy is applied. All scripts can run without restrictions.

If you see the "Command execution is blocked" error message, it means the current execution policy is restricting the execution of the specific command or script you are trying to run.

To change the execution policy, you'll need to open a PowerShell session with administrative privileges (Run as Administrator) and use the Set-ExecutionPolicy cmdlet. For example, to set the execution policy to "RemoteSigned," you can run:

powershellCopy codeSet-ExecutionPolicy RemoteSigned

Keep in mind that adjusting the execution policy can have security implications, so it's essential to choose an appropriate policy based on your needs and the level of trust you have in the scripts you'll be running.

If you are in a restricted environment where you don't have the necessary permissions to change the execution policy, you may need to contact your system administrator to get the proper permissions or have them run the necessary commands for you.

And this as well:
Apart from the PowerShell execution policy, there are a few other potential causes for the "Command execution is blocked" error in PowerShell:

Group Policy: In a domain environment, a Group Policy may be in place that enforces a specific PowerShell execution policy for all computers within the domain. If your system is part of a domain and has Group Policy settings applied, these settings could override the local execution policy settings.
Antivirus or Endpoint Protection Software: Some antivirus or endpoint protection software may have features that block the execution of certain commands or scripts. These security measures might perceive specific PowerShell commands as potentially harmful and prevent them from executing.
Script Blocking Software: Similarly, third-party script-blocking software may be installed on the system, which prevents the execution of PowerShell scripts or specific commands to enhance security.
User Account Control (UAC): If you are running PowerShell with standard user privileges on a Windows system with User Account Control enabled, certain operations might be blocked due to restricted permissions.
Execution Policy Set by Script: If you have a script that sets the execution policy and it's run at the start of your PowerShell session, it could be responsible for enforcing a restrictive execution policy.
Insufficient Privileges: Certain PowerShell commands require elevated privileges (Run as Administrator) to execute. If you are running PowerShell without administrative rights and attempting to execute a command that requires elevated privileges, you might encounter this error.
Missing or Corrupted PowerShell Components: In some cases, missing or corrupted PowerShell components could cause issues with script execution.
File System Permissions: If the script or the file you are trying to execute has restrictive file system permissions, it may block the execution.

To troubleshoot the issue, consider the following steps:

Check the current execution policy by running Get-ExecutionPolicy in PowerShell and ensure it's set to a suitable level.
If you suspect Group Policy settings may be affecting the execution policy, contact your domain administrator to inquire about the applied policies.
Review the settings of your antivirus or endpoint protection software to see if it's blocking PowerShell commands.
Try running PowerShell as an administrator to see if elevated privileges are needed for the specific command.
Review the scripts or commands you are trying to execute and ensure they are correct and not blocked by other security software or permissions.

If none of these steps resolve the issue, it might be beneficial to seek assistance from your system administrator or IT support team to diagnose and resolve the problem more thoroughly.

Reproduction steps

1. Load Windows 10 22H2 to use as execution host
2. Load PowerCLI
3. Run VAMT migration as administrator

See errors like this:

Info] 08/19/2023 18:39:54 - There are currently 0 moves in progress and 10 moves waiting to start.
[Info] 08/19/2023 18:39:54 - New batch of moves: Rocky9-HPE-1, CentOS7-HPE-2, Rocky9-HPE-2, Ubuntu2004-HPE-2, Ubuntu2004-HPE-1
[Error] 08/19/2023 18:40:21 - VM move job for 'CentOS7-HPE-2' ended with unsupported state Blocked. Considering this job failed.
[Info] 08/19/2023 18:40:22 - Refreshing VM 'CentOS7-HPE-2' object
[Info] 08/19/2023 18:40:22 - Preparing to set 'VAMT : failed' tag on 'CentOS7-HPE-2'
[Info] 08/19/2023 18:40:22 - Successfully to set 'VAMT : failed' tag on 'CentOS7-HPE-2'
[Error] 08/19/2023 18:40:22 - VM move job for 'Rocky9-HPE-1' ended with unsupported state Blocked. Considering this job failed.
[Info] 08/19/2023 18:40:22 - Refreshing VM 'Rocky9-HPE-1' object
[Info] 08/19/2023 18:40:22 - Preparing to set 'VAMT : failed' tag on 'Rocky9-HPE-1'
[Info] 08/19/2023 18:40:23 - Successfully to set 'VAMT : failed' tag on 'Rocky9-HPE-1'
[Error] 08/19/2023 18:40:23 - VM move job for 'Rocky9-HPE-2' ended with unsupported state Blocked. Considering this job failed.
[Info] 08/19/2023 18:40:23 - Refreshing VM 'Rocky9-HPE-2' object
[Info] 08/19/2023 18:40:23 - Preparing to set 'VAMT : failed' tag on 'Rocky9-HPE-2'
[Info] 08/19/2023 18:40:23 - Successfully to set 'VAMT : failed' tag on 'Rocky9-HPE-2'
[Error] 08/19/2023 18:40:23 - VM move job for 'Ubuntu2004-HPE-1' ended with unsupported state Blocked. Considering this job failed.
[Info] 08/19/2023 18:40:23 - Refreshing VM 'Ubuntu2004-HPE-1' object
[Info] 08/19/2023 18:40:24 - Preparing to set 'VAMT : failed' tag on 'Ubuntu2004-HPE-1'
[Info] 08/19/2023 18:40:24 - Successfully to set 'VAMT : failed' tag on 'Ubuntu2004-HPE-1'
[Error] 08/19/2023 18:40:24 - VM move job for 'Ubuntu2004-HPE-2' ended with unsupported state Blocked. Considering this job failed.
[Info] 08/19/2023 18:40:25 - Refreshing VM 'Ubuntu2004-HPE-2' object
[Info] 08/19/2023 18:40:25 - Preparing to set 'VAMT : failed' tag on 'Ubuntu2004-HPE-2'
[Info] 08/19/2023 18:40:25 - Successfully to set 'VAMT : failed' tag on 'Ubuntu2004-HPE-2'
[Info] 08/19/2023 18:40:25 - There are currently 0 moves in progress and 5 moves waiting to start.
[Info] 08/19/2023 18:40:25 - New batch of moves: Win2019-HPE-2, CentOS7-HPE-1, Win2019-HPE-1, Win2012r2-HPE-1, Win2012r2-HPE-2
[Error] 08/19/2023 18:40:46 - VM move job for 'Win2012r2-HPE-1' ended with unsupported state Blocked. Considering this job failed.
[Info] 08/19/2023 18:40:46 - Refreshing VM 'Win2012r2-HPE-1' object
[Info] 08/19/2023 18:40:46 - Preparing to set 'VAMT : failed' tag on 'Win2012r2-HPE-1'
[Info] 08/19/2023 18:40:46 - Successfully to set 'VAMT : failed' tag on 'Win2012r2-HPE-1'
[Error] 08/19/2023 18:40:46 - VM move job for 'Win2019-HPE-1' ended with unsupported state Blocked. Considering this job failed.
[Info] 08/19/2023 18:40:46 - Refreshing VM 'Win2019-HPE-1' object
[Info] 08/19/2023 18:40:47 - Preparing to set 'VAMT : failed' tag on 'Win2019-HPE-1'
[Info] 08/19/2023 18:40:47 - Successfully to set 'VAMT : failed' tag on 'Win2019-HPE-1'
[Error] 08/19/2023 18:40:47 - VM move job for 'Win2019-HPE-2' ended with unsupported state Blocked. Considering this job failed.
[Info] 08/19/2023 18:40:47 - Refreshing VM 'Win2019-HPE-2' object
[Info] 08/19/2023 18:40:47 - Preparing to set 'VAMT : failed' tag on 'Win2019-HPE-2'
[Info] 08/19/2023 18:40:48 - Successfully to set 'VAMT : failed' tag on 'Win2019-HPE-2'
[Error] 08/19/2023 18:40:48 - VM move job for 'CentOS7-HPE-1' ended with unsupported state Blocked. Considering this job failed.
[Info] 08/19/2023 18:40:48 - Refreshing VM 'CentOS7-HPE-1' object
[Info] 08/19/2023 18:40:48 - Preparing to set 'VAMT : failed' tag on 'CentOS7-HPE-1'
[Info] 08/19/2023 18:40:48 - Successfully to set 'VAMT : failed' tag on 'CentOS7-HPE-1'
[Info] 08/19/2023 18:40:48 - There are currently 1 moves in progress and 0 moves waiting to start.

Expected behavior

Migration should work, and it does when running the exact same scripts on a Windows 10 21H2 build, so odds are good that it is some default policy or security behavior that has changed between these Windows 10 builds that is now blocking this.

Additional context

This one works:

This one does not:

Need to have a cover timer for overall script operation, individual VM migration timeout

Describe the bug

There are times seen relatively infrequently, but enough that all of the folks testing have seen it, where the script comes down to the last VM but just seems to get stuck in a loop waiting on that to complete. I've let it go overnight and it still keeps going, so we probably need to come up with a reasonable cover timer for either the overall script or an individual VM (or both) so that it breaks out of that condition.

[Info] 02/24/2023 20:33:39 - There are currently 2 moves in progress and 0 moves waiting to start.
[Info] 02/24/2023 20:33:54 - There are currently 2 moves in progress and 0 moves waiting to start.
[Info] 02/24/2023 20:34:09 - VM move job for 'vm001' is complete.
[Info] 02/24/2023 20:34:09 - There are currently 1 moves in progress and 0 moves waiting to start.
[Info] 02/24/2023 20:34:25 - There are currently 1 moves in progress and 0 moves waiting to start.
[Info] 02/24/2023 20:34:40 - There are currently 1 moves in progress and 0 moves waiting to start.
[Info] 02/24/2023 20:34:55 - There are currently 1 moves in progress and 0 moves waiting to start.
[Info] 02/24/2023 20:35:10 - There are currently 1 moves in progress and 0 moves waiting to start.

Reproduction steps

1. Start a migration
2. Watch migration, gets stuck at the last VM
3. Happens randomly, doesn't seem to be related to scale or environment.

Expected behavior

Script eventually succeeds or aborts with a message.

Additional context

Saw this in the OCI environment, logs are at: C:\Users\opc\Desktop\VAMT\vmware-architecture-migration-tool\vamt_runlogs\20230224-191312

Add number of VM retries to migration summary

Is your feature request related to a problem? Please describe.

Would be good to see whether there were any retries in the summary as well as in the detailed log listing. For instance, when there is a retryable error as below:

[Info] 02/23/2023 08:25:17 - New batch of moves: vm208, vm209, vm184, vm185, vm212 [Error] 02/23/2023 08:25:34 - VM move job for 'vm212' failed with errors: [Warn] 02/23/2023 08:25:34 - Error is eligible to be re-tried. Setting retry status to try again later for 'vm212'. [Info] 02/23/2023 08:25:34 - There are currently 19 moves in progress and 273 moves waiting to start.

The summary will show the following when complete if the retry succeeded:
[Info] 02/23/2023 09:16:52 - Script run summary: Script start: '02/23/2023 08:19:39' Script runtime: 57 minutes Script completion: '02/23/2023 09:16:52' Total VM targets in migration run: 300 VM migration jobs with final status 'Completed': 300 [Info] 02/23/2023 09:16:52 - Final report:

And the number of attempts is shown in the detailed logging:
`tgt_compute tgt_network job_result tgt_storage tgt_folder tag_state job_state src_vcenter attempts tgt_vm

---

oci01-w01-consolidated01 VM Network Successfully moved VM 'vm001'. vsanDatastore MigrationTest complete Completed vcenter-wuovkm-sddc.sddc.iad.oci.oraclecloud.com 1 vm001
oci01-w01-consolidated01 VM Network Successfully moved VM 'vm002'. vsanDatastore MigrationTest complete Completed vcenter-wuovkm-sddc.sddc.iad.oci.oraclecloud.com 1 vm002`

For ease of visibility, would be good to see the number of VMs with retries.

Describe the solution you'd like

Something like the following:

[Info] 02/23/2023 09:16:52 - Script run summary: Script start: '02/23/2023 08:19:39' Script runtime: 57 minutes Script completion: '02/23/2023 09:16:52' Total VM targets in migration run: 300 VM migration jobs with final status 'Completed': 300 VMs with retries: 1 [Info] 02/23/2023 09:16:52 - Final report:

Describe alternatives you've considered

Thought about writing a separate parsing script to scrape the log for output metadata, but would be easier to do and see in the summary.

Additional context

No response

Add counts to summary data at the end of the migration

During various larger scale VAMT testing runs using VMmark3 benchmark in 20 tile (380 VM) and 21 tile (399 VM) scenarios, we discovered that it would be useful to have the counts added to the summary data.

This is an example of the summary data that is sent out at the end:

[Info] 11/05/2022 00:38:12 - Script run summary:
        Script start: '11/04/2022 23:39:12'
        Script runtime: 58 minutes
        Script completion: '11/05/2022 00:38:12'
[Info] 11/05/2022 00:38:12 - Final report:

Would like to add something like:

Total VM targets in migration run: X
VMs successfully migrated: Y
VMs that failed migration: Z

Error during rollback if a vm or vms failed the original migration

transcript-20230124T1523575856.log

Add option to bypass Tag checking for govmomi/vcsim support for FAE training material.

Is your feature request related to a problem? Please describe.

We are creating an online certification process and running the VAMT code against a containerized simulator is required for beginner and intermediate training. The vcsim simulator does not have full tag functionality and we need to come up with a work around that supports the current feature set.

Describe the solution you'd like

create a parameter defined in the @options that will bypass/return success on any tag validation or tag creation calls.

Describe alternatives you've considered

adding full tag support or at least hacking govmomi/vcsim to return 1

Additional context

No response

Add more retryable errors to script

Is your feature request related to a problem? Please describe.

In the current vcenter test environment for vsphere 8 we are using with VMFS datastores, vcenter encounters the following error:

[Error] 03/21/2023 18:40:40 - VM move job for 'Test2' failed with errors: Caught excecption in migration job on line '148': 3/21/2023 6:40:25 PM Move-VM The operation for the entity "Test2" failed with the following message: "Could not complete network copy for file /vmfs/volumes/640f46db-56785310-2b16-84160caa86ec/Test2/Test2_5.vmdk".

20230321-171248_VMFSTransientFailure.zip

There is also an error that looks to be retryable relative to tag state:

3/20/2023 2:36:48 PM Get-TagAssignment There was no endpoint listening at https://vsphere.maas/sdk that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. 3/20/2023 2:36:48 PM Get-TagAssignment Could not find TagCategory with name 'VAMT'. 3/20/2023 2:36:48 PM Get-TagAssignment TagCategory parameter: Could not find any of the objects specified by name. 3/20/2023 2:36:48 PM Get-Tag There was no endpoint listening at https://vsphere.maas/lookupservice/sdk that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. 3/20/2023 2:36:48 PM Get-Tag Could not find TagCategory with name 'VAMT'. 3/20/2023 2:36:48 PM Get-Tag TagCategory parameter: Could not find any of the objects specified by name. 3/20/2023 2:36:48 PM Get-Tag Tag with name 'complete' was not found using the specified filter(s). Caught excecption in migration job on line '799': Cannot bind argument to parameter 'Tag' because it is null.

20230320-142245_TagNotFound.zip

And this was one from a little while back that also had to do with invalid tag state:

20230222-173117_InvalidTagState.zip

Describe the solution you'd like

When this migration is ran again a second time for both of these errors, it always succeeds, so would like to add both of these to the list of retryable errors.

Describe alternatives you've considered

Just running the script again is the current workaround. But since we have the retry mechanism seems like a better approach.

Additional context

I've added log file zips for each one to this issue

Unknown errors encountered during 100 VM test in Oracle cloud migration environment but no detail showing in error log

Describe the bug

Saw the following error example numerous times during several 100 VM test runs
[Error] 02/20/2023 06:32:14 - VM move job for 'vm077' failed with errors:
[Error] 02/20/2023 06:50:52 - VM move job for 'vm002' failed with errors:

The output showed this:
oci01-w01-consolidated01 VM Network Successfully moved VM 'vm001'. vsanDatastore MigrationTest complete Completed vcenter-intel-sddc.sddc.iad.oci.o...
oci01-w01-consolidated01 VM Network Job completed with unknown result. See scripting logs for details. vsanDatastore MigrationTest failed Failed vcenter-intel-sddc.sddc.iad.oci.o...
oci01-w01-consolidated01 VM Network Successfully moved VM 'vm003'. vsanDatastore MigrationTest complete Completed vcenter-intel-sddc.sddc.iad.oci.o...

oci01-w01-consolidated01 VM Network Successfully moved VM 'vm076'. vsanDatastore MigrationTest complete Completed vcenter-intel-sddc.sddc.iad.oci.o...
oci01-w01-consolidated01 VM Network Job completed with unknown result. See scripting logs for details. vsanDatastore MigrationTest failed Failed vcenter-intel-sddc.sddc.iad.oci.o...
oci01-w01-consolidated01 VM Network Successfully moved VM 'vm078'. vsanDatastore MigrationTest complete Completed vcenter-intel-sddc.sddc.iad.oci.o...

Reproduction steps

1. Kick off 100 VM migration between SDDCs with migration
2. During migration, some of the VMs fail to migrate
3. Look into the error logs, no error showing

These are the options used:
$options = @{
action = "migrate"
vCenters = @("vcenter-intel-sddc.sddc.iad.oci.oraclecloud.com","vcenter-wuovkm-sddc.sddc.iad.oci.oraclecloud.com")
vcCredentials = @($intelVcenterCredObject, $amdVcenterCredObject)
inputFilePath = "C:\Users\opc\Desktop\VAMT\vmware-architecture-migration-tool\example\TestMigrateItoA.json"
parallelTaskCount = 10
debugLogging = $true
}

Expected behavior

All migrations succeed

Additional context

Austin is adding some retries to test