VCSA 7 u1 update woes
“Don’t be hasty master Meriadoc”
I was being hasty while updating my own lab vCenter Appliance to 7.0 U1a patch. I started the process from the VCSA gui as usual, but the download was interrupted at 3% (perhaps from a fault of my own), leaving the install in a messed up state. Refreshing the gui would only result in a window telling me my update failed, prompting me to run the pre-checks again. The pre-checks would go through, but it would never go to the download phase again, instead looping back to the failed install message, and the pre-check. Over and over again. Rebooting the appliance had no impact in this. To add insult to injury, the install had masked most of the vCenter services in the appliance, preventing their startup and breaking most of vCenter along with it.
I started debugging from the masked services, running:
systemctl unmask vmware-vmon
The above is an example of the syntax; in my case the list of services was numerous. After that, I ran systemctl –stop –all, and –start –all to make sure all the correct services were running. vCenter was now in a semi-working state (i.e. I could log on and manage VMs), but the appliance management was still in the failed update / pre-check loop.
For this, I found the following clue: https://kb.vmware.com/s/article/67179 for a similar situation of a botched update. Same search also led me to this thread: https://communities.vmware.com/thread/644420. The core issue is there is a file which tells VCSA that an update or install is still in progress. The file is called: /etc/applmgmt/appliance/software_update_state.conf. I couldn’t find meaningful instructions on how to modify the file, so I took a backup and deleted it. After that, restarted the applmgmt service, which allowed me to get back into the VCSA management gui.
In the meanwhile I also performed the update to 7.0U1a via ssh, but this didn’t clear the botched install oddly enough, so the gui was still complaining. The install via gui goes like this.
From the appliance shell, not the bash shell run:
software-packages stage –url –acceptEulas
Reboot for good measure after the installer says it is done. If you instead have an ISO, you can use –iso in place of –url. Source: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vcenter.upgrade.doc/GUID-8466F019-C57C-4344-9E15-8CFF74A6E4C2.html
After this things were running smoothly again, although the patch in question doesn’t contain anything super critical for me at least. Release notes for 7.0 u1a here: https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-vcenter-server-70u1a-release-notes.html. After the install your vCenter will list version 7.0.1 build 17005016 and the appliance will show 7.0.1.00100 build 17004997. Don’t forget to patch your hosts, check for vmware tools and vm compatibility upgrades while you are messing about. And be mindful that as always, any vm compatibility updates are a one-way-street without snapshots, though you can also create a new VM using the existing disks, or use VMware converter to downgrade.