RPS VCF on VxRail Run Book

Meet the EMEA startups and scaleups redefining their industries with cloud technology
March 25, 2021
Four ways CSPs can harness data, automation, and AI to create business value
March 25, 2021
Meet the EMEA startups and scaleups redefining their industries with cloud technology
March 25, 2021
Four ways CSPs can harness data, automation, and AI to create business value
March 25, 2021
Show all
0

Pre-Checks (Tech Consult) Days or Weeks before the Upgrade

A Tech Consult is used to:

  • Review VCF System Health.
  • Determine current code and target code levels.
  • Assess the size of the environment:
    • Number of WLD’s.
    • Number of ESXi Nodes in the environment.
  • Provide customers with some guidance on expected duration of the VCF on VxRail Upgrade.
  • Start the bundle download process using either:
    • Skip Level Tool with -d option.
    • Offline Bundle Transfer Tool and a list of bundles.
  1. Gather basic system information
    • Navigate to Inventory
    • Count the number of Workload Domains (WLD) present (Document findings as this is required for TC template)
    • Count the number of hosts in total on the SDDC manager (will require the following steps per WLD)
  2. Verify current VCF / SDDC Manager version
    • Navigate to Lifecycle –> Inventory page on the SDDC Manager UI
  3. Verify intended destination with customer
    • If the requested destination is not mentioned in the SR or RCM task, please ask customer what destination they are targeting.
    • Based on the destination being targeted, determine if this will be single jump or multi jump upgrade.
    • Check with https://kb.vmware.com/s/article/67854,to confirm the code Customer has requested has been released already.
    • As there is a lead time between VCF releases and the VCF on VxRail Release please check the following links to ensure VCF on VxRail version is supported:
  4. Verify compatibility
  5. Check for any optional software present on the SDDC Manager
    • Navigate to Settings –> Licensing
    • Check for any software that not covered under the LCM process (you will need to notify customer within post TC email report)
    • Software components covered by LCM are:
      VMware vCenter Server Appliance (VCSA) (PSC license included in this)
      VMware vSAN
      VMware NSX-T Data Center
      VMware vSphere ESXi
      VMware vRealize Suite Lifecycle Manager
      Items specifically not covered Kubernetes (Pivotal) KPS or VMware Horizon.
  6. Verify systems health
    • Log in to the SDDC manager UI
    • Navigate to Inventory
    • Click on a Domain and click the Updates/Patches tab
    • Click Precheck to validate the appliance readiness for the upgrade.
      Click View Status to see the update status for each component and the tests performed. Expand a test by clicking the arrow next to it to see further details.
      If any of the tests fail, fix the issue and click Retry Precheck. The precheck results are displayed below the Precheck button. Ensure that the precheck results are green. Repeat steps a-d for each Domain present (WLD)
  7. Gather environment information via CLI:
    • Open a putty session to the SDDC manager using IP or FQDN of the manager VM
    • Login using username ‘vcf‘, password is customer provided (su as root)
    • Run the following commands and record ALL output:
      This command gets the current SDDC Version:

      /opt/vmware/sddc-support/sos –version

      This command gets the MGMT and WLD Domain information for SR notes:

      psql –host=localhost -U postgres -d platform -c “select d.type,count(distinct(d.name)) as wld_count from host_and_domain e,domain d where e.domain_id = d.id group by d.type order by wld_count;”

      This command gets the list of ESXi hosts in each Domain:

      psql –host=localhost -U postgres -d platform -c “select d.name,d.type as domain_type,d.status as domain_status,h.hostname, h.version, h.status from host h,host_and_domain e, domain d where h.id=e.host_id and e.domain_id = d.id group by d.name,d.type,d.status,h.hostname,h.version,h.status order by domain_type;”

  8. Check DEPOT status to make sure Depot Status show’s Success for serial upgrades.
    Note: Skip Level upgrades must have the Depot disconnected before the Skip Level upgrade is started.

    curl http://localhost/lcm/depot/statuses | json_pp

    Example-
    root@example-vcf [ /home/vcf ]# curl http://localhost/lcm/depot/statuses | json_pp
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 144 0 144 0 0 422 0 –:–:– –:–:– –:–:– 422
    {
    “VXRAIL_DEPOT” : {
     “status” : “SUCCESS”,
     “message” : “Depot Status: Success”
      },
     “VCF_DEPOT” : {
      “status” : “SUCCESS”,
      “message” : “Depot Status: Success”
      }
      }

  1. Check Space on SDDC

    Check /nfs/vmware/vcf/nfs-mount/ free space by executing command:

    df -h on SDDC Manager CLI

    if the space is less than 100 GB we can either solve that or open a collab for that to fix it prior the upgrade in order to have free space for downloading the required bundles.

  2. Run CLI Health checks from SDDC manager (SOS).
    • Open a putty session to the SDDC manager using IP or FQDN of the manager VM
    • Login using username ‘vcf‘, password is customer provided
    • SU to root and run the health check tool (SOS Utility) with the following commands:
      /opt/vmware/sddc-support/sos –health-check –domain-name ALL 
      (specify a single domain name or use ALL to check all domains.)
      /opt/vmware/sddc-support/sos –get-vcf-summary
  3. Run vSAN health checks for all clusters and resolve any issues.
  4. Clear any old alerts in vCenter Server.
  5. Initiate code download
    RPS Team will initiate code transfer with customer but due to the length of time required for the code to download RPS will disengage once it’s confirmed that download is progressing.
    • For Serial Upgrades, example 3.10.1.1 à 3.10.1.2 use the SDDC GUI to download the code.
      • Open SDDC Manager UI and go to Repository > Bundles.
      • Multiple LCM bundles will begin to show up and be available for download. It may take several hours for all bundles to become available. Refer to the VMware support matrix and BOM to determine which packages are required.
      • Find one of the required bundles and click Download Now. Repeat this step for all required bundles
    • If the SDDC Manager is connected to the Internet and a Skip Level Upgrade is required, please follow VCF On VxRail Code Transfers (procedure in Solve). This will enable a customer to download the code bundles in advance and automatically upload to SDDC Manager. Multiple LCM bundles will need to be downloaded and it may take several hours for all bundles to finish downloading. Refer to the VMware support matrix and BOM to determine which packages are required
      Inform Customer to monitor the download progress and retry the download if any failures.
      Contact RPS Team if transfer is still showing failures after retrying.
    • If the SDDC IS NOT connected with Internet. Please follow Offline Bundle Transfer (procedure in Solve) to start the offline code download.
      Multiple LCM bundles will need to be downloaded and it may take several hours for all bundles to finish downloading. Inform Customer to copy downloaded bundles to SDDC Manager under /nfs/vmware/vcf/nfs-mount. 
      Customer will need to monitor the download progress at regular intervals retry the download if there are any failures.
      Contact RPS Team if transfer is showing failures after retrying.

Pre-Checks (Upgrade Day)

It’s assumed that the bundles have all been downloaded to SDDC Manager with -d option or uploaded using Offline Bundle Transfer Tool.

  1. Verify systems health
    • Log in to the SDDC manager UI
    • Navigate to Inventory
    • Click on a Domain and click the Updates/Patches tab
    • Click Precheck to validate the appliance readiness for the upgrade.
      Click View Status to see the update status for each component and the tests performed. Expand a test by clicking the arrow next to it to see further details.

      If any of the tests fail, fix the issue and click Retry Precheck. The precheck results are displayed below the Precheck button. Ensure that the precheck results are green. Repeat steps a-d for each Domain present (WLD)

  2. Check DEPOT status to make sure Depot Status show’s Success for serial upgrades.
    Note: Skip Level upgrades must have the Depot disconnected before the Skip Level upgrade is started.
    curl http://localhost/lcm/depot/statuses | json_pp
    Example-
    root@example-vcf [ /home/vcf ]# curl http://localhost/lcm/depot/statuses | json_pp
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 144 0 144 0 0 422 0 –:–:– –:–:– –:–:– 422
    {
     “VXRAIL_DEPOT” : {
     “status” : “SUCCESS”,
      “message” : “Depot Status: Success”
     },
     “VCF_DEPOT” : {
     “status” : “SUCCESS”,
     “message” : “Depot Status: Success”
      }
      }
  3. Check Space on SDDC

    Check /nfs/vmware/vcf/nfs-mount/ free space by executing command:
    df -h on SDDC Manager CLI

    if the space is less than 100 GB we can either solve that or open a collab for that to fix it prior the upgrade in order to have free space for downloading the required bundles

  4. Run CLI Health checks from SDDC manager (SOS).
    • Open a putty session to the SDDC manager using IP or FQDN of the manager VM
    • Login using username ‘vcf‘, password is customer provided
    • SU to root and run the health check tool (SOS Utility) with the following commands:

      /opt/vmware/sddc-support/sos –health-check –domain-name ALL 
      (specify a single domain name or use ALL to check all domains.)
      /opt/vmware/sddc-support/sos –get-vcf-summary

  5. For each VxRail Manager on the VCF environment run VxVerify and Debugger script
    Please refer to VxRail documentation on how to run each script from VxRail Manager:
    • Check and see if this is a Stretched cluster as it will require us to upgrade Witness Host manually and document this in the upgrade plan.
      Please copy ALL output to the SR as evidence.
  6. Check the VxRail Manager password for illegal characters. Ask the customer if any of the following characters are in the mystic or root password: $ ` !
    Note: ! is ok but only as the last character in the password.

Manual check:

  • Login to sddc manager using vcf credential and evaluate to root privileges.
  • Run lookup_passwords command and select VXRAIL_MANAGER
  • Enter the privileged user credentials.

If credentials are valid the mystic and root passwords will be shown for all VxRail clusters in the VCF environment.

  1. Perform workaround in the below KB to ensure there is enough space on VxRail Manager VM for the code bundle.
    https://www.dell.com/support/kbdoc/en-ie/000181710/dell-emc-vxrail-how-to-migrate-vcf-upgrade-bundle-in-vxrail-manager
  2. Take snap shots of the following components:
    • SDDC Manager
    • PSC’s (Platform Service Controllers)
    • vCenter Servers
    • VxRail Managers
  3. If not using SLT tool move to step 12.
    For Skip Level Tool Upgrades, open a command prompt in the bin directory.
    Run SLT with -d -u -s (this will check the downloads and start the upgrade; the -s flag skips the vCenter snapshot as a snapshot has already been taken.)
    sddcmanager-skip-level-upgrade.bat -d -u -s
  4. Once the SLT completes you will need to perform the following step to make the bundles appear in the SDDC Manager Repository:
    sddcmanager-skip-level-upgrade.bat -e
  5. Check the health again in SDDC Manager before continuing.
  6. Open the SDDC Manager UI and go to Inventory > Workload Domains > View Details > MGMT > Update/Patches.
    Note: Upgrade must be completed on the MGMT Domain before starting a WLD.
  7. Click Precheck to validate that SDDC Manager is ready to be updated.
  8. Click View Status to see the update status for each component and the tests performed. Expand a test by clicking the arrow next to it to see further details.

    If any of the tests fail, please fix the issue and click Retry Precheck. The precheck results are displayed below the Precheck button. Ensure that the precheck results are green before proceeding. A failed precheck may cause the update to fail

  9. Click Update Now to start the update.
  1. Monitor the upgrade in the Update in Progress section in the workload domain detail page.
  2. When the update completes, click Finish.
  3. Repeat the process until all components have been upgraded in MGMT Domain. Once completed move into the WLD domains and repeat step 12 for the WLD.
  4. Once all components are upgrade there should be no more Available Updates and all component versions should match the BOM for the target code.

Post-Checks (To be further defined)

  1. Verify systems health
    • Log in to the SDDC manager UI
    • Navigate to Inventory
    • Click on a Domain and click the Updates/Patches tab
    • Click Precheck to validate the appliance readiness for the upgrade.
      Click View Status to see the update status for each component and the tests performed. Expand a test by clicking the arrow next to it to see further details.
  2. Reconnect the Repository if a Skip Level Upgrade was performed.
  3. Check the vSAN Health in vCenter.
  4. Run CLI Health checks from SDDC manager (SOS).
    • Open a putty session to the SDDC manager using IP or FQDN of the manager VM
    • Login using username ‘vcf‘, password is customer provided
    • SU to root and run the health check tool (SOS Utility) with the following commands:
      /opt/vmware/sddc-support/sos –health-check –domain-name ALL
      (specify a single domain name or use ALL to check all domains.)

      /opt/vmware/sddc-support/sos –get-vcf-summary

      Please copy ALL output to the SR as evidence.

  5. Run VxVerify on all VxRail Clusters and ensure Plugin works as expected.
  6. Run vSAN Health Checks from vCenter Server.
  7. Check SDDC Manager for free space and delete any old bundles.

Skip Level Tool Upgrades:

Download the latest Skip Level Upgrade procedure from Solve and follow the steps to download the latest. Follow the steps to download the code from VMware site:

Current working links are:
3.10.1.2
https://my.vmware.com/group/vmware/downloads/details?downloadGroup=VCF3101_TOOLS&productId=1001 

4.1.0.0
https://my.vmware.com/group/vmware/downloads/details?downloadGroup=VCF410_TOOLS&productId=1061

  1. Extract the zip on the customer machine and ensure there is 100GB free space.
  2. Disconnect the Repo’s otherwise the Skip Level Tool will fail.
  3. Open a command prompt with admin privileges and navigate to /bin
  4. Run sddcmanager-skip-level-upgrade -d

    This will download the packages and upload them to the SDDC Manager. The information in Red will be required.

    E:Skip_Level_Toollcm-tools-prod.tarbin>sddcmanager-skip-level-upgrade.bat -d -s

    2021-01-13 06:39:39.509-0600 INFO  Welcome to the VMware Cloud Foundation SDDC Skip Level Upgrade Tool, Version: 3.10.1-vcf31012RELEASE-17289780
    2021-01-13 06:39:39.509-0600 INFO  It supports SDDC Manager services upgrade from version 3.5.X, 3.7.X, 3.8.X, 3.9.X, 3.10.X to 3.10.1.2
    2021-01-13 06:39:39.510-0600 INFO  Log file: E:Skip_Level_Toollcm-tools-prod.tarskiplevelupgrade.log
    Please enter the FQDN of SDDC Manager VM: SDDC FQDN
    Please enter the primary user on SDDC Manager VM: vcf
    Please enter the basic auth user on SDDC Manager VM: admin
    Please enter password of the primary user:
    Please enter root password of SDDC Manager VM:
    Please enter basic auth password of SDDC Manager VM:

    2021-01-13 06:40:42.955-0600 INFO  Target VCF version is set to 3.10.1.2
    2021-01-13 06:40:42.955-0600 INFO  Perform pre-validation before SDDC Manager Skip Level Upgrade
    2021-01-13 06:40:43.126-0600 INFO  Pre-validation succeeded
    2021-01-13 06:40:49.850-0600 WARN  Download for bundle with ID
    3c04ec22-5982-4dc9-8a0a-4dc6bd8c16ac is not in success state
    2021-01-13 06:40:49.866-0600 WARN  Download for bundle with ID 58259e63-6c21-4d88-b58a-281240de52a2 is not in success state
    2021-01-13 06:40:49.866-0600 WARN  Download for bundle with ID 528047d2-a844-4a51-90e1-f63a9ea42ca5 is not in success state
    2021-01-13 06:40:49.881-0600 WARN  Download for bundle with ID 06d763d9-13c6-4e92-aa5f-e61949b7d227 is not in success state
    2021-01-13 06:40:49.881-0600 WARN  Download for bundle with ID 4e388e85-aa48-4ca8-a064-6510f5f19571 is not in success state
    2021-01-13 06:40:49.897-0600 WARN  Download for bundle with ID VXRAIL4-7-525-26769857DL101531_VxRail-4-7-525-Composite-Upgrade-Package-for-4-7-x-zip is not in success state
    Please enter the depot user: [email protected]
    Please enter the depot user password:
    Please enter the directory for downloaded bundles: E:VCF_Download
    Please enter the Dell EMC depot user: [email protected]
    Please enter the Dell EMC depot user password:
    2021-01-13 06:41:59.801-0600 INFO  Download partner bundles from Dell EMC Depot
    2021-01-13 06:42:07.060-0600 INFO  Download required bundles and copy them to SDDC Manager VM

Troubleshooting issues with Skip Level Tool:

If the Skip Level fails to connect to VMware depot, please have the customer verify credentials by logging into myVMware.com using the credentials.

If credentials are valid please check if a proxy server is configured:

On SDDC manager

cat /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties

lcm.depot.adapter.proxyEnabled=true <<  Proxy Configured
lcm.depot.adapter.proxyHost=proxy IP address
lcm.depot.adapter.proxyPort=proxy port

If there is a proxy server configured, then you will need to update the Skip Level Tool so it uses the same server to connect to the internet.

Go to the folder where the Skip Level Tool have been unzipped and navigate to conf folder.

Edit the application-skiplevelupgrade.properties file and copy the settings from above to:

################### Depot proxy properties ########################
lcm.depot.adapter.proxyEnabled=true
lcm.depot.adapter.proxyHost=<proxy ip address>
lcm.depot.adapter.proxyPort=<proxy port>

Save the file and try the Skip Level Tool again.

If still having issues, ask customer to check their firewall settings and open a support SR if further troubleshooting is required.

Offline LCM Download Overview

This method should be used when concurrent connection to SDDC Manager and the Internet is not possible. This process manually downloads each VCF bundle using the lcm-bundle-transfer-util command.  After all bundles are downloaded, they are copied to SDDC Manager VM and staged for the upgrade activity.

Offline Bundle Transfer Tool:

Download the latest Offline Bundle Transfer Tool procedure from Solve and follow the steps to download the latest Bundle Transfer Utility, it’s inside Skip Level zip file.

The skip level upgrade tool can be downloaded from myvmware > Product Downloads > VMware Cloud Foundations Downloads > Drivers & Tools.

Validate the checksum using certutil command as follows:

The offline LCM bundle download process consists of 3 phases:

  • Download VCF Bundles with lcm-bundle-transfer-util
  • Download VxRail composite bundle from Dell Support site
  • Copy and Stage the files on SDDC Manager VM

 

Download VCF Bundles (Offline)

  1. Login to the external host where the lcm-tools-prod bundle has been installed.
  2. Open a console prompt and navigate to the location of lcm-tools.

    cd C:lcm-tools-prodbin

  3. Download the LCM bundle with the following command:

    C:lcm-tools-prodbinlcm-bundle-transfer-util -download “withCompatibilitySets” -outputDirectory C:Temp -depotUser [email protected] -bundle bundle-12404

    Example Output:

    Enter Myvmware user password:
    Validating the depot user credentials…
    Downloading bundle: bundle-16154
    Download Progress of bundle tar : /evo/vmw/bundles/bundle-16154.tar : 0.1 MB, Average Speed: 0.78 Mbps, Total Size:  : 139.0 MB
    Download Progress of bundle tar : /evo/vmw/bundles/bundle-16154.tar : 21.6 MB, Average Speed: 10.10 Mbps, Total Size:  : 139.0 MB
    Download Progress of bundle tar : /evo/vmw/bundles/bundle-16154.tar : 79.3 MB, Average Speed: 12.91 Mbps, Total Size:  : 139.0 MB
    Successfully downloaded bundle: bundle-16154.
    Completed downloading bundle to directory: C:Temp

  4. Repeat the previous step to download all the required bundles.

Download the VxRail 4.7.525 Composite Bundle

Copy and Stage the files on SDDC Manager VM

  1. Login to SDDC Manager VM using SSH utility and login as vcf user and then su to root.
  2. Change directory with the command:
    cd /nfs/vmware/vcf/nfs-mount/bundle
  3. Create the “tmp” directory with the command:
    mkdir tmp
  4. Update permissions of the folder with commands:
    chown vcf_lcm:vcf tmp
    chmod 755 tmp
  5. Using an SCP utility such as WinSCP, copy the files from the C:Temp to SDDC Manager:
    All files from C:Tempbundles to /nfs/vmware/vcf/nfs-mount/bundle/tmp/
    All files from C:Tempmanifests to /nfs/vmware/vcf/nfs-mount/bundle/tmp/
  6. Locate the partnerBundleMetadata.json and softwareCompatibilitySets.json files from the lcm-tools location:

    ..lcm-tools-prodconfskiplevelupgradepartnerBundleMetadata.json
    ..lcm-tools-prodconfskiplevelupgradesoftwareCompatibilitySets.json

    These files will be copied to SDDC Manager VM in the next step.

  7. Using an SCP utility such as WinSCP, copy the VxRail composite bundle and the two JSON files to SDDC Manager VM to /home/vcf
  8. Login to SDDC Manager VM using SSH as “vcf” user and then su to root.
  9. Run the following commands to stage the files for the skip level upgrade:
    • Change directories:
      cd /home/vcf
    • Rename the VxRail bundle filename (see table below for filenames for all VxRail Bundles – 4.7.525 used in this example):

      mv VXRAIL_COMPOSITE-4.7.525-26769857_for_4.7.x.zip DL101531_VxRail-4.7.525-Composite-Upgrade-Package-for-4.7.x.zip

    • Copy files to the correct location:

      cp -p DL101531_VxRail-4.7.525-Composite-Upgrade-Package-for-4.7.x.zip /nfs/vmware/vcf/nfs-mount/bundle/depot/local/bundles

      cp -p partnerBundleMetadata.json /nfs/vmware/vcf/nfs-mount/bundle/depot/local/bundles

      cp -p softwareCompatibilitySets.json /nfs/vmware/vcf/nfs-mount/bundle/depot/local/bundles

    • Update permissions for VxRail bundle files:

      chown -R vcf_lcm:vcf /nfs/vmware/vcf/nfs-mount/bundle/depot/local/bundles

      chmod -R 777 /nfs/vmware/vcf/nfs-mount/bundle/depot/local/bundles

    • Update permissions for VCF bundle files:

      chown -R vcf_lcm:vcf /nfs/vmware/vcf/nfs-mount/bundle/tmp/
      chmod -R 777 /nfs/vmware/vcf/nfs-mount/bundle/tmp/

Code Filename
4.7.111 DL93390_VxRail_4.7.111_Composite_Upgrade_Package_for_4.7.x.zip
4.7.211 DL94267_VxRail_4.7.211_Composite_Upgrade_Package_for_4.7.x.zip
4.7.212 DL95074_VxRail-4.7.212-Composite-Upgrade-Package-for-4.7.x.zip
4.7.300 DL95561_VxRail-4.7.300-Composite-Upgrade-Package-for-4.7.x.zip
4.7.410 DL96732_VxRail-4.7.410-Composite-Upgrade-Package-for-4.7.x.zip
4.7.511 DL99171_VxRail-4.7.511-Composite-Upgrade-Package-for-4.7.x.zip
4.7.515 DL100174_VxRail-4.7.515-Composite-Upgrade-Package-for-4.7.x.zip
4.7.525 DL101531_VxRail-4.7.525-Composite-Upgrade-Package-for-4.7.x.zip
7.0.000 DL99643_VxRail-7.0.010-Composite-Upgrade-Package-for-7.0.x.zip
7.0.100 DL100842_VxRail-7.0.100-Composite-Upgrade-Slim-Package-for-7.0.x.zip
7.0.101 DL101581_VxRail-7.0.101-Composite-Upgrade-Slim-Package-for-7.0.x.zip

Basic Auth user:

The Basic Auth user is defined at install and the default username is admin. If the customer doesn’t know the password, we will need to reset it so that we can run the skip level tool.

Use the following command to change the password when logged in as root on SDDC Manager:

/opt/vmware/vcf/commonsvcs/scripts/auth/set-basicauth-password.sh admin <password>

Related posts

March 21, 2024

Simpler Licensing with VMware vSphere Foundation and VMware Cloud Foundation 5.1.1

Read more
January 24, 2024

VMware Skyline Advisor Pro Proactive Findings – January 2024 Edition

Read more
January 16, 2024

Skyline Advisor Pro: Introducing Inventory Export Reports

Read more

Leave a Reply

Your email address will not be published. Required fields are marked *