VCF on VxRail: Async Patch Tool

What is the Async Patch Tool?

VMware released the async patch tool in 2022 as a way to apply component patching in a more flexible manner. As stated in the tool’s release notes – The Async Patch Tool is a utility that allows you to apply critical patches to certain VMware Cloud Foundation components (NSX Manager, vCenter Server, and ESXi) outside of VMware Cloud Foundation releases. The Async Patch Tool also supports ESXi and VxRail Manager patching of VMware Cloud Foundation on VxRail and is supported with VMware Cloud Foundation 4.2.1 and later. If we slightly rephrase the statement from the release notes – it ultimately provides users, the ability to patch their systems without taking on the burden of upgrading every component in the VCF Bill of Materials (BOM).

It is always very important to check Applying individual product updates to VCF using Async Patch Tool (AP Tool) (88287) for the specific patch you plan to apply. There may be knowledge base articles associated to it that require additional steps (just like the one you will see in the post).

VCF Async Patch Tool Options

We won’t be going through all the options that are available while using the Async Patch Tool. However, as you can see in the image below, VMware has done a nice job of documenting the command options and providing explanations for each.

It’s important to point out that the Async Patch Tool itself is developed and released asynchronously from the VCF version itself and the general guidance from VMware is that you must use the latest available Async Patch Tool. That being said, as newer Async Patch Tool versions are released the command options may change over time.

Online vs. Offline Patching

Online

This method is intended for customers whose systems have access to the internet. It does not require a jump host and the procedure is to install the Async Patch Tool on SDDC Manager only.

Offline

This method is intended for customers whose systems have no access to the internet and requires a jump host with access to the internet, as well as SDDC Manager. Files are downloaded locally and then transferred to SDDC Manger.

Enable Patch vs. Enable VCF Upgrade

Enable Patch

The intended use of the enable patch (or -e, –enableAsyncPatch) switch is for applying patches that will deviate your system outside of the VCF Bill of Materials (BOM). The primary driver for this type of patch is generally critical security vulnerabilities or bug fixes that cannot wait for a full VCF release. Once the patch is enabled via console access to SDDC Manager, the patch will show available to apply in the SDDC Manager UI.

• Pre-checks and post checks are always encouraged.
• New workload domains created after the async patching process will not contain the newly applied async patch. The user will need to async patch each new domain after it is created.
• Future upgrades to a full VCF BOM will require the use of the enable VCF Upgrade switch, on supported versions.
• After using enable patch, always remember to deactivate the patch once complete.

Enable VCF Upgrade

The intended use of the enable VCF Upgrade (or -r, –enableVCFUpgrade) switch is to “recover” the VCF instance to a full VCF BOM after the enable patch switch has been run on the environment. This is because one or more of the VCF components have deviated from the VCF BOM. This switch is only supported to be used with VCF 4.x to VCF 4.y releases and not VCF 5.0 or newer.

Patching Different Components

When enabling patches using the Async Patch Tool there are five different components that are possible to patch.

• SDDC Manager – hot patch only
• NSX Manager
• vCenter Server
• ESXi
• VxRail Manager

While VMware lists the async patch product type as “VxRail Manager”, what’s really being applied is the full VxRail bundle. That’s important to distinguish because not just the VxRail Manager appliance gets patched. Instead, SDDC Manager calls the VxRail Manager API to initiate a typical VxRail LCM and VxRail Manager will apply upgrades to all completed updated in the VxRail bundle.

Patching VCF on VxRail 4.5.1 with VxRail 7.0.451

The steps that we’ll be going through in this post will apply an async patch to a VCF on VxRail system that requires a CVE remediation that was fixed in VxRail version 7.0.451. The environment is currently running VCF on VxRail 4.5.1 with VxRail version 7.0.450 and we will be async patching to VxRail version 7.0.451.

Note: I did not run a pre-check or post-check in my environment since it is a lab environment. It would be very prudent for production environments to run both checks.

Setup

First you will need to setup the environment. We’ll be using the offline method which is configured on a Windows jump host. I won’t be going through these steps, but you should have more than enough to get setup using either Using VCF’s Async Patch Tool on Windows or Using VCF’s Async Patch Tool with PowerShell.

List Available Patches

After the Async Patch Tool environment has been setup, we will want to look at the list of available patches. In the screenshot you can see all the available patches. If you wish to filter these results by SKU type and/or product type, you can so using these switches –sku –productType.

VxRail Async Patching Only – Partner Bundle Metadata File

When applying patches for VxRail we need to make sure that SDDC Manager has the information about what Dell URL to get the correct bundle from, compatible versions, bundle size, checksum, and much more. This is accomplished behind the scenes via a partner bundle metadata file (PBM). When using the online method there’s no need to download a new copy of the PBM. When using the offline method however, we need to download the JSON file and place it in the output directory created during environment setup. All the steps for this process can be found using Pre-requisite before enabling VxRail async patch to VMware Cloud Foundation environments using Async Patch Tool (AP Tool) (91830).

Download VxRail Bundle 7.0.451

Now that we’ve seen all the available patches, we can see that we want to work with VxRail-7.0.451-Composite-Upgrade-Slim-Package-for-7.0.x.zip. Using the information from the –listAsyncPatch switch we can use the -d and –patch switches to download the proper content to the Windows jump host. The format used for the download and enable patch inputs for product type and version are product:version as seen in VMware’s documentation. We will use VX_MANAGER:7.0.451-28053212 along with sku VCF_ON_VXRAIL. There are two depot accounts that are required. One is used for connectivity to the VMware depot which is referred to as –du (Depot User) and the other is used for connectivity to Dell support which is referred to as –pdu (Partner Depot User). Lastly, we’ve used an output directory using the –outputDirectory switch which will download all bundles and artifacts to this jump host directory. You will notice apToolBundles is used in a lot of documentation examples and some of my recent posts. The example is this post use AP-Output, so just keep that in mind.

There will be prompts to confirm that you are using the latest Async Patch Tool, as well as user passwords for both the depot user and the partner depot user. Once these confirmations are made the required bundles and artifacts will be downloaded to the jump host output directory. After the downloads complete you will see output stating success.

Upload Patches to SDDC Manager

Now that the downloads have completed, we need to transfer them the Async Patch Tool installation artifacts we extracted during setup, as well as the downloaded bundles and artifacts. The apToolBundles directory does not yet exist on SDDC Manager, so you will be copying the entire output directory and setting permissions on it, not just the copying the contents of the output directory.

• Copy the entire output directory to /nfs/vmware/vcf-mount/apToolBundles on SDDC Manager.
• Navigate to /nfs/vmware/vcf/nfs-mount/apToolBundles
• Set permissions – chmod -R 755 apToolBundles
• Set ownership – chown -R vcf:vcf apToolBundles

• Make a directory for the Async Patch Tool installation – mkdir /home/vcf/asyncPatchTool
• Copy the entire contents of C:\Users\admin\Downloads\asyncPatchTool to /home/vcf/asyncPatchTool
• Set permissions – chmod -R 755 asyncPatchTool
• Set ownership – chown -R vcf:vcf asyncPatchTool

Enable and Apply Async Patch

Next, we will enable and apply the patch on SDDC Manager. SSH into SDDC Manager using the vcf user. You will use the enable patch switch discussed earlier to enable the patch, so that we can login to the SDDC Manager UI and apply it. We will use a lot of similar input as the download command because we are now enabling that bundle on SDDC Manger – for example VX_MANAGER:7.0.451-28053212 and –outputDirectory. We also need to specify –sddcSSOUser (likely administrator@vsphere.local), the –sddcSSHUser which will be the vcf user, and –it OFFLINE for instance type being offline. You will be prompted to confirm:

• Using the latest Async Patch Tool
• Met the prerequisites for the environment
• SDDC Manager vcf user password
• SDDC Manager root user password
• SSO password
• Confirm to continue

After SDDC Manager has updated all of backend services, manifests, etc. the SDDC Manager UI can be used to upgrade the VxRail system.

Deactivate Patches

Finally, now that the upgrade has completed successfully the Async Patch Tool can be deactivated to allow for future upgrades. This is a simple process that only requires one command

• Navigate to /home/vcf/asyncPatchTool/bin – cd /home/vcf/asyncPatchTool/bin
• Deactivate patches – ./vcf-async-patch-tool –disableAllPatches –sddcSSOUser SSOuser –sddcSSHUser vcf

Helpful Links

• Async Patch Tool Help Options
• Async Patch Tool 1.0.0.2 Documentation
• Applying individual product updates to VCF using Async Patch Tool (AP Tool) (88287)
• VCF on VxRail Support Matrix
• Pre-requisite before enabling VxRail async patch to VMware Cloud Foundation environments using Async Patch Tool (AP Tool) (91830).