VCF on VxRail Upgrade cheat sheet

This document lists out proactive steps customers can take to prevent known issues happening during VoV upgrade, and lists out some potential issues customers might face during this process and the remediation steps for the same.

Pre-Checks (Days or Weeks before the Upgrade)

Pre-checks are used to:

• Review VCF System Health.
• Determine current code and target code levels.
• Assess the size of the environment:
  • Number of WLD’s.
  • Number of ESXi Nodes in the environment.
• Provides some guidance on expected duration of the VCF on upgrade.
• Start the bundle download process using either:
  • Skip Level Tool with -d option.
  • Offline Bundle Transfer Tool and a list of bundles.

1. Gather basic system information
   1. 1. Count the number of Workload Domains (WLD) present
      2. Count the number of hosts in total on the SDDC manager (will require the following steps per WLD)
   2. Verify current VCF / SDDC Manager version
      • Navigate to Lifecycle > Inventory page on the SDDC Manager UI
   3. Verify intended destination
      1. Based on the destination being targeted, determine if this will be single jump or multi jump upgrade.
      2. Check with Supported versions of VMware Cloud Foundation on VxRail, to confirm the build is available.
   4. Verify compatibility
      • Please reference VMware VCF compatibility matrix: Supported versions of VMware Cloud Foundation on VxRail and Correlating VMware Cloud Foundation version with the versions of its constituent products
      • For specific version requests (ESXi builds, vCenter Server builds etc..) please reference the release note documentation for specific VCF builds.
   5. Check for any optional software present on the SDDC Manager
      1. Navigate to Settings > Licensing
      2. Check for any software that not covered under the LCM process
      3. Software components covered by LCM are:
         VMware vCenter Server Appliance (VCSA) (PSC license included in this)
         VMware vSAN
         VMware NSX-T Data Center
         VMware vSphere ESXi
         VMware vRealize Suite Lifecycle Manager
         Items specifically not covered Kubernetes (Pivotal) KPS or VMware Horizon.
   6. Verify the systems health
      1. Log in to the SDDC manager UI
      2. Navigate to Inventory
      3. Click on a Domain and click the Updates/Patches tab
      4. Click Precheck to validate the appliance readiness for the upgrade.
         Click View Status to see the update status for each component and the tests performed. Expand a test by clicking the arrow next to it to see further details.
         If any of the tests fail, fix the issue and click Retry Precheck. The precheck results are displayed below the Precheck button. Ensure that the precheck results are green. Repeat steps A-D for each Domain present (WLD)
   7. Gather environment information via CLI:
      1. Open an SSH session to the SDDC manager using IP or FQDN of the manager VM
      2. Login using username ‘vcf‘,  and your password
      3. Run the following commands and record ALL output:
         This command gets the current SDDC Version:

         /opt/vmware/sddc-support/sos –version

         This command returns the MGMT and WLD Domain information:

         psql –host=localhost -U postgres -d platform -c “select d.type,count(distinct(d.name)) as wld_count from host_and_domain e,domain d where e.domain_id = d.id group by d.type order by wld_count;”

         This command returns the list of ESXi hosts in each Domain:

         psql –host=localhost -U postgres -d platform -c “select d.name,d.type as domain_type,d.status as domain_status,h.hostname, h.version, h.status from host h,host_and_domain e, domain d where h.id=e.host_id and e.domain_id = d.id group by d.name,d.type,d.status,h.hostname,h.version,h.status order by domain_type;”

   8. Check DEPOT status to make sure Depot Status show’s Success for serial upgrades.
      Note: Skip Level upgrades must have the Depot disconnected before the Skip Level upgrade is started.

      curl http://localhost/lcm/depot/statuses | json_pp

      Example:
      root@example-vcf [ /home/vcf ]# curl http://localhost/lcm/depot/statuses | json_pp
      % Total % Received % Xferd Average Speed Time Time Time Current
      Dload Upload Total Spent Left Speed
      100 144 0 144 0 0 422 0 –:–:– –:–:– –:–:– 422
      {
      “VXRAIL_DEPOT” : {
       “status” : “SUCCESS”,
       “message” : “Depot Status: Success”
        },
       “VCF_DEPOT” : {
        “status” : “SUCCESS”,
        “message” : “Depot Status: Success”
        }
        }

   9. Check Space on SDDC

      Check /nfs/vmware/vcf/nfs-mount/ free space by executing command:

      df -h on SDDC Manager CLI

      At least 100GB is required to upgrade, clear space if less is available.

   10. Run CLI Health checks from SDDC manager (SOS).
       1. 1. Open an SSH session to the SDDC manager using IP or FQDN of the manager VM
          2. Login using username ‘vcf‘,  and your password
          3. SU to root and run the health check tool (SOS Utility) with the following commands:
             /opt/vmware/sddc-support/sos –health-check –domain-name ALL 
             (specify a single domain name or use ALL to check all domains.)
             /opt/vmware/sddc-support/sos –get-vcf-summary
   11. Run vSAN health checks for all clusters and resolve any issues.
   12. Clear any old alerts in vCenter Server.

Pre-Checks (Upgrade Day)

It’s assumed at this stage that the bundles have all been downloaded to SDDC Manager with -d option or uploaded using Offline Bundle Transfer Tool.

1. Verify the systems health
   1. Log in to the SDDC manager UI
   2. Navigate to Inventory
   3. Click on a Domain and click the Updates/Patches tab
   4. Click Precheck to validate the appliance readiness for the upgrade.

      Click View Status to see the update status for each component and the tests performed. Expand a test by clicking the arrow next to it to see further details.

      If any of the tests fail, fix the issue and click Retry Precheck. The precheck results are displayed below the Precheck button. Ensure that the precheck results are green. Repeat steps A-D for each Domain present (WLD)

2. Check DEPOT status to make sure Depot Status show’s Success for serial upgrades.

   Note: Skip Level upgrades must have the Depot disconnected before the Skip Level upgrade is started.
   curl http://localhost/lcm/depot/statuses| json_pp

   Example:

   root@example-vcf [ /home/vcf ]# curl http://localhost/lcm/depot/statuses | json_pp
   % Total % Received % Xferd Average Speed Time Time Time Current
     Dload Upload Total Spent Left Speed
     100 144 0 144 0 0 422 0 –:–:– –:–:– –:–:– 422
     {
     “VXRAIL_DEPOT” : {
     “status” : “SUCCESS”,
     “message” : “Depot Status: Success”
     },
     “VCF_DEPOT” : {
     “status” : “SUCCESS”,
     “message” : “Depot Status: Success”
     }
     }

3. Check Space on SDDC

   Check /nfs/vmware/vcf/nfs-mount/ free space by executing command:

   df -h on SDDC Manager CLI

   At least 100GB is required to upgrade, clear space if less is available.

4. Run CLI Health checks from SDDC manager (SOS).
   1. Open an SSH session to the SDDC manager using IP or FQDN of the manager VM
   2. Login using username ‘vcf’, and your password
   3. SU to root and run the health check tool (SOS Utility) with the following commands:

      /opt/vmware/sddc-support/sos –health-check –domain-name ALL 
      (specify a single domain name or use ALL to check all domains.)

      /opt/vmware/sddc-support/sos –get-vcf-summary

      Manual check:

      1. Login to sddc manager using vcf credential and evaluate to root privileges.

      2. Enter the privileged user credentials.

         If credentials are valid the mystic and root passwords will be shown for all VxRail clusters in the VCF environment.

5. Perform workaround in the below KB to ensure there is enough space on VxRail Manager VM for the code bundle.

   https://www.dell.com/support/kbdoc/en-ie/000181710/dell-emc-vxrail-how-to-migrate-vcf-upgrade-bundle-in-vxrail-manager

6. Take snap shots of the following components:
   • SDDC Manager
   • PSC’s (Platform Service Controllers)
   • vCenter Servers
7. If not using SLT tool move to step 12. Otherwise, for Skip Level Tool Upgrades, open a command prompt in the bin directory.

   Run SLT with -d -u -s (this will check the downloads and start the upgrade; the -s flag skips the vCenter snapshot as a snapshot has already been taken.)

   sddcmanager-skip-level-upgrade.bat -d -u -s

8. Once the SLT completes you will need to perform the following step to make the bundles appear in the SDDC Manager Repository:

   sddcmanager-skip-level-upgrade.bat -e

9. Check the health again in SDDC Manager before continuing.
10. Open the SDDC Manager UI and go to Inventory > Workload Domains > View Details > MGMT > Update/Patches.

    Note: Upgrade must be completed on the MGMT Domain before starting a WLD.

11. Click Precheck to validate that SDDC Manager is ready to be updated.
12. Click View Status to see the update status for each component and the tests performed. Expand a test by clicking the arrow next to it to see further details.

    If any of the tests fail, please fix the issue and click Retry Precheck. The precheck results are displayed below the Precheck button. Ensure that the precheck results are green before proceeding. A failed precheck may cause the update to fail.

13. Click Update Now to start the update.
    
14. Monitor the upgrade in the Update in Progress section in the workload domain detail page.
15. When the update completes, click Finish.
16. Repeat the process until all components have been upgraded in MGMT Domain. Once completed move into the WLD domains and repeat step 12 for the WLD.
17. Once all components are upgrade there should be no more Available Updates and all component versions should match the BOM for the target code.
18. If NSX-V is in the environment,
    • Make sure Port 1234 is opened inside the environment. This has prevented upgrades from occurring.
19. If NSX-T is in the environment,
    1. Check and Resolve all critical and high severity alarms first, then perform a reboot.
    2. Reboot the NSX-T Managers, one by one. The managers should take between 5 to 8 minutes to reboot and sync
    3. Check for NSX – T endpoints on the hosts and resolve any issues. The below screenshot shows an example of endpoints with existing issues.

 

Post-Checks

1. Verify the systems health
   1. Log in to the SDDC manager UI
   2. Navigate to Inventory
   3. Click on a Domain and click the Updates/Patches tab
   4. Click Precheck to validate the appliance readiness for the upgrade.

Click View Status to see the update status for each component and the tests performed. Expand a test by clicking the arrow next to it to see further details.

2. Reconnect the Repository if a Skip Level Upgrade was performed.
3. Check the vSAN Health in vCenter.
4. Run CLI Health checks from SDDC manager (SOS).
   1. Open an SSH session to the SDDC manager using IP or FQDN of the manager VM
   2. Login using username ‘vcf‘, and your password
   3. SU to root and run the health check tool (SOS Utility) with the following commands:

/opt/vmware/sddc-support/sos –health-check –domain-name ALL 
(specify a single domain name or use ALL to check all domains.)

/opt/vmware/sddc-support/sos –get-vcf-summary

5. Run VxVerify on all VxRail Clusters and ensure the plugin works as expected.
6. Run vSAN Health Checks from vCenter Server.
7. Check SDDC Manager for free space and delete any old bundles.

Appendix A

Skip Level Tool Upgrades:

Download the latest Skip Level Upgrade procedure, follow the steps to download:

3.10.1.2
https://my.vmware.com/group/vmware/downloads/details?downloadGroup=VCF3101_TOOLS&productId=1001 

4.1.0.0
https://my.vmware.com/group/vmware/downloads/details?downloadGroup=VCF410_TOOLS&productId=1061

1. Extract the zip and ensure there is at least 100GB free space.
2. Disconnect the Repo’s otherwise the Skip Level Tool will fail.
3. Open a command prompt with admin privileges and navigate to /bin
4. Run sddcmanager-skip-level-upgrade -d

This will download the packages and upload them to the SDDC Manager. The information in Red will be required.

E:Skip_Level_Toollcm-tools-prod.tarbin>sddcmanager-skip-level-upgrade.bat -d -s

2021-01-13 06:39:39.509-0600 INFO  Welcome to the VMware Cloud Foundation SDDC Skip Level Upgrade Tool, Version: 3.10.1-vcf31012RELEASE-17289780
2021-01-13 06:39:39.509-0600 INFO  It supports SDDC Manager services upgrade from version 3.5.X, 3.7.X, 3.8.X, 3.9.X, 3.10.X to 3.10.1.2
2021-01-13 06:39:39.510-0600 INFO  Log file: E:Skip_Level_Toollcm-tools-prod.tarskiplevelupgrade.log
Please enter the FQDN of SDDC Manager VM: SDDC FQDN
Please enter the primary user on SDDC Manager VM: vcf
Please enter the basic auth user on SDDC Manager VM: admin
Please enter password of the primary user:
Please enter root password of SDDC Manager VM:
Please enter basic auth password of SDDC Manager VM:

2021-01-13 06:40:42.955-0600 INFO  Target VCF version is set to 3.10.1.2
2021-01-13 06:40:42.955-0600 INFO  Perform pre-validation before SDDC Manager Skip Level Upgrade
2021-01-13 06:40:43.126-0600 INFO  Pre-validation succeeded
2021-01-13 06:40:49.850-0600 WARN  Download for bundle with ID 3c04ec22-5982-4dc9-8a0a-4dc6bd8c16ac is not in success state
2021-01-13 06:40:49.866-0600 WARN  Download for bundle with ID 58259e63-6c21-4d88-b58a-281240de52a2 is not in success state
2021-01-13 06:40:49.866-0600 WARN  Download for bundle with ID 528047d2-a844-4a51-90e1-f63a9ea42ca5 is not in success state
2021-01-13 06:40:49.881-0600 WARN  Download for bundle with ID 06d763d9-13c6-4e92-aa5f-e61949b7d227 is not in success state
2021-01-13 06:40:49.881-0600 WARN  Download for bundle with ID 4e388e85-aa48-4ca8-a064-6510f5f19571 is not in success state
2021-01-13 06:40:49.897-0600 WARN  Download for bundle with ID VXRAIL4-7-525-26769857DL101531_VxRail-4-7-525-Composite-Upgrade-Package-for-4-7-x-zip is not in success state
Please enter the depot user: [email protected]
Please enter the depot user password:
Please enter the directory for downloaded bundles: E:VCF_Download
Please enter the Dell EMC depot user: [email protected]
Please enter the Dell EMC depot user password:
2021-01-13 06:41:59.801-0600 INFO  Download partner bundles from Dell EMC Depot
2021-01-13 06:42:07.060-0600 INFO  Download required bundles and copy them to SDDC Manager VM

Troubleshooting issues with Skip Level Tool:

If the Skip Level fails to connect to VMware depot, verify credentials by logging into myVMware.com using the credentials.

If credentials are valid please check if a proxy server is configured:

On SDDC manager

cat /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties

lcm.depot.adapter.proxyEnabled=true <<  Proxy Configured
lcm.depot.adapter.proxyHost=proxy IP address
lcm.depot.adapter.proxyPort=proxy port

If there is a proxy server configured, then you will need to update the Skip Level Tool so it uses the same server to connect to the internet.

Go to the folder where the Skip Level Tool have been unzipped and navigate to conf folder.

Edit the application-skiplevelupgrade.properties file and copy the settings from above to:

################### Depot proxy properties ########################
lcm.depot.adapter.proxyEnabled=true
lcm.depot.adapter.proxyHost=proxy ip address
lcm.depot.adapter.proxyPort=proxy port

Save the file and try the Skip Level Tool again.

If you are still having issues, check the firewall settings.

Offline LCM Download Overview

This method should be used when concurrent connection to SDDC Manager and the Internet is not possible. This process manually downloads each VCF bundle using the lcm-bundle-transfer-util command.  After all bundles are downloaded, they are copied to SDDC Manager VM and staged for the upgrade activity.

Offline Bundle Transfer Tool:

Download the latest Offline Bundle Transfer Tool procedure, and follow the steps to download the latest Bundle Transfer Utility, it’s inside Skip Level zip file.

The skip level upgrade tool can be downloaded from myvmware > Product Downloads > VMware Cloud Foundations Downloads > Drivers & Tools.

Validate the checksum using certutil command as follows:

The offline LCM bundle download process consists of 3 phases:

1. Download VCF Bundles with lcm-bundle-transfer-util
2. Download VxRail composite bundle from Dell Support site
3. Copy and Stage the files on SDDC Manager VM

Appendix B:

Download VCF Bundles (Offline)

1. Login to the external host where the lcm-tools-prod bundle has been installed.
2. Open a console prompt and navigate to the location of lcm-tools.

cd C:lcm-tools-prodbin

3. Download the LCM bundle with the following command:

C:lcm-tools-prodbinlcm-bundle-transfer-util -download “withCompatibilitySets” -outputDirectory C:Temp -depotUser [email protected] -bundle bundle-12404

Example Output:

Enter Myvmware user password:
Validating the depot user credentials…
Downloading bundle: bundle-16154
Download Progress of bundle tar : /evo/vmw/bundles/bundle-16154.tar : 0.1 MB, Average Speed: 0.78 Mbps, Total Size:  : 139.0 MB
Download Progress of bundle tar : /evo/vmw/bundles/bundle-16154.tar : 21.6 MB, Average Speed: 10.10 Mbps, Total Size:  : 139.0 MB
Download Progress of bundle tar : /evo/vmw/bundles/bundle-16154.tar : 79.3 MB, Average Speed: 12.91 Mbps, Total Size:  : 139.0 MB
Successfully downloaded bundle: bundle-16154.
Completed downloading bundle to directory: C:Temp

4. Repeat the previous step to download all the required bundles.

Download the VxRail 4.7.525 Composite Bundle

 Download the required VxRail Composite Bundle directly from the Dell Support site: https://www.dell.com/support/home/en-us/product-support/product/vxrail-software/drivers

Copy and Stage the files on SDDC Manager VM

1. Login to SDDC Manager VM using SSH utility and login as vcf user and then su to root.
2. Change directory with the command:

cd /nfs/vmware/vcf/nfs-mount/bundle

3. Create the “tmp” directory with the command:

mkdir tmp

4. Update permissions of the folder with commands:

chown vcf_lcm:vcf tmp

chmod 755 tmp

5. Using an SCP utility such as WinSCP, copy the files from the C:Temp to SDDC Manager:

All files from C:Tempbundles to /nfs/vmware/vcf/nfs-mount/bundle/tmp/

All files from C:Tempmanifests to /nfs/vmware/vcf/nfs-mount/bundle/tmp/

6. Locate the partnerBundleMetadata.json and softwareCompatibilitySets.json files from the lcm-tools location:

   ..lcm-tools-prodconfskiplevelupgradepartnerBundleMetadata.json

   ..lcm-tools-prodconfskiplevelupgradesoftwareCompatibilitySets.json

   These files will be copied to SDDC Manager VM in the next step.

7. Using an SCP utility such as WinSCP, copy the VxRail composite bundle and the two JSON files to SDDC Manager VM to /home/vcf
8. Login to SDDC Manager VM using SSH as “vcf” user and then su to root.
9. Run the following commands to stage the files for the skip level upgrade:
10. 1. Change directories:
       cd /home/vcf
    2. Rename the VxRail bundle filename (see table below for filenames for all VxRail Bundles – 4.7.525 used in this example):
       mv VXRAIL_COMPOSITE-4.7.525-26769857_for_4.7.x.zip DL101531_VxRail-4.7.525-Composite-Upgrade-Package-for-4.7.x.zip
    3. Copy files to the correct location:

       cp -p DL101531_VxRail-4.7.525-Composite-Upgrade-Package-for-4.7.x.zip /nfs/vmware/vcf/nfs-mount/bundle/depot/local/bundles

       cp -p partnerBundleMetadata.json /nfs/vmware/vcf/nfs-mount/bundle/depot/local/bundles

       cp -p softwareCompatibilitySets.json /nfs/vmware/vcf/nfs-mount/bundle/depot/local/bundles

    4. Update permissions for VxRail bundle files:

       chown -R vcf_lcm:vcf /nfs/vmware/vcf/nfs-mount/bundle/depot/local/bundles

       chmod -R 777 /nfs/vmware/vcf/nfs-mount/bundle/depot/local/bundles

    5. Update permissions for VCF bundle files:

chown -R vcf_lcm:vcf /nfs/vmware/vcf/nfs-mount/bundle/tmp/
chmod -R 777 /nfs/vmware/vcf/nfs-mount/bundle/tmp/

Code	Filename
4.7.111	DL93390_VxRail_4.7.111_Composite_Upgrade_Package_for_4.7.x.zip
4.7.211	DL94267_VxRail_4.7.211_Composite_Upgrade_Package_for_4.7.x.zip
4.7.212	DL95074_VxRail-4.7.212-Composite-Upgrade-Package-for-4.7.x.zip
4.7.300	DL95561_VxRail-4.7.300-Composite-Upgrade-Package-for-4.7.x.zip
4.7.410	DL96732_VxRail-4.7.410-Composite-Upgrade-Package-for-4.7.x.zip
4.7.511	DL99171_VxRail-4.7.511-Composite-Upgrade-Package-for-4.7.x.zip
4.7.515	DL100174_VxRail-4.7.515-Composite-Upgrade-Package-for-4.7.x.zip
4.7.525	DL101531_VxRail-4.7.525-Composite-Upgrade-Package-for-4.7.x.zip
7.0.000	DL99643_VxRail-7.0.010-Composite-Upgrade-Package-for-7.0.x.zip
7.0.100	DL100842_VxRail-7.0.100-Composite-Upgrade-Slim-Package-for-7.0.x.zip
7.0.101	DL101581_VxRail-7.0.101-Composite-Upgrade-Slim-Package-for-7.0.x.zip

Appendix C:

Basic Auth user:

The Basic Auth user is defined at install and the default username is admin. If the password is unavailable, it  will need to be reset to run the skip level tool.

Use the following command to change the password when logged in as root on SDDC Manager:

/opt/vmware/vcf/commonsvcs/scripts/auth/set-basicauth-password.sh admin <password>