Update Manager

If you’re running ESXi deployed from custom HP ESXi installation images but you’re not patching the HP utilities inside with each ESXi update, you may be setting yourself up for trouble in the future. And simply pointing to the VMware repositories won’t update these custom tools — you also have to point towards the HP vibs repository to get all of the latest updates. So, lets look at how to do that…

Updates & Patching for ESXi

VMware has done a good job of providing administrators ways to keep their ESXi deployments up to date – both with offline bundles and with VMware Update Manager for patching the ESXi systems. Either method is pretty simple and HP leverages these same tools for its custom vibs.

For HP, administrators should point to http://vibsdepot.hp.com for the latest and most up to date information about the correct recipe of VMware and HP tools and driver versions. This is also the repository for automated updates that can be plugged directly into VMware Update Manager – or can be manually downloaded and loaded into Update Manager or used as offline bundles.

Why not just deploy VMware’s default ESXi?

Some administrators adopted the HP customized ESXi image early and others have held out until the latest ProLiant Gen8 hardware was released. The latest ProLiants need storage and network controller drivers included in the customized HP ESXi image to successfully deploy ESXi on Gen8. So now, a growing community has deployed additional HP utilities into their ESXi as well. In my own experience, that’s not been an issue, however I did hit a problem a couple months back where excessive logging from one HP tool filled up my ESXi RAMdisk. Filling the RAMdisk made the system unmanageable even though the VM’s continued to operate on the host. So like any other software, you need to patch those drivers, tools and utilities from HP if they’re present in your ESXi.

Patching options for HP customized deployments

As mentioned earlier, the HP updates are deployed using the same VMware provided tools as all other ESXi patches. Administrators can use VMware Update Manager for a more automated deployment, ESXCLI for command line or scripted installations or they can use Image Builder tools to package the HP vibs into images that they are creating for deployment. Since I’m primarily focusing on patching, Update Manager and ESXCLI would be our two go-to tools.

Update Manager Repository

For Update Manager, the easiest way to get the HP updates into Update Manager and onto hosts is to point to a new repository in the Update Manager tool. Adding an additional repository in Update Manager must be done through the Configuration tab of the Update Manager settings using the traditional Windows vSphere Client. It doesn’t appear to be configurable in the vSphere Web Client.

Navigate to Solutions and Applications and choose Update Manager.
On the Configuration tab, under Settings, click Download Settings.
In the Download Sources pane, select Direct connection to Internet.

Click Add Download Source.
In the Add Download Source window, type the new download source URL. For the HP repository, the URL will be http://vibsdepot.hp.com/hpq/<release date>/index.xml.
Tip: You may navigate to http://vibsdepot.hp.com/hpq/ to see a directory list of all the release dates.
Click Validate URL to verify that the URL is accessible.

Click OK and then Apply.

Once you have setup the custom repository, click Download and let those updates be imported into Update Manager. Once imported, rescan your hosts and now the HP customized tools will also be remediated during patching.

Update Manager Manual Import

Alternatively, you can download the individual zip files of the updates from the http://vibsdepot.hp.com website and import the individual updates into your local Update Manager library. Once imported, make sure these updates are included in a baseline and attached to a host and rescan the host. The new patches will show up and you’ll be able to remediate the hosts with these patches.

ESXCLI

For users who want to script, who want or need to use ESXCLI, one additional line will pull down and deploy the updates from an HP release to your ESX hosts:

[code]esxcli software vib install -d http://vibsdepot.hp.com/hpq/<release date>/index.xml[/code]

Just replace the <release date> in the URL with a correct release date to point to an active repository.

Limited Time Only

With both Update Manager and ESXCLI, keep in mind that you’ve configured and/or remediated against a single, point-in-time HP repository, so you will also need to add new repositories as the patches change and new updates are released from HP. You may navigate to http://vibsdepot.hp.com/hpq/ to see a directory list of all the release dates. Configuring against a single release is the recommended HP method to ensure you get the correct patches according to the recipe for your ESXi patch level.

Recipes

HP also releases a list of recipes for which version of tools and drivers work with which version of ESXi and its patch levels. These recipes give administrators additional validation that the tools loaded have been tested with the ESXi patch level and everything should run correctly together. The recipes are available on the same site with the patches and downloads at http://vibsdepot.hp.com. The PDF recipes highlight more than just the software aspect of the ESXi and HP tool integration – it also highlights the firmware levels desired by ProLiant make and model to ensure that everything has been tested together.

Update: VMware has published a KB article related to this issue. It is available at http://kb.vmware.com/kb/2018897.

Several months ago, our VMware Update Manager simply stopped updating from its Shavlik repository. It seemed to only affect Windows patches being downloaded and the VMware patches continued without any issues. After a few months of troubleshooting with VMware support, a resolution was located and implemented in our environment and its a pretty simple solution.

Problem background

For a bit of background, the primary issue we noticed is that when trying to download updates from the repository, I received a generic error “Cannot download patch definitions” in the vCenter tasks. After looking into it, I found that if I unchecked the Windows patch option in the Patch Download Settings section of Update Manager’s configuration tab, the error would not appear. It was only when Windows patches were enabled that the error happened.

I uninstalled and reinstalled Update Manager against the same database and reinstalled a second time with a new database. Neither of these steps fixed the problem and so I opened a case with VMware support.

With support, I began digging into the log files for Update Manager and found more details in the files. We noticed the Update Manager would pull down a .cab file from the Shavlik servers. After the successful download, Update Manager immediately logged an error, “Error downloading new Windows updates: Cannot de-obfuscate Shavlik metadata file.” Although the error gives a little more detail, it still wasn’t enough to point to a concrete cause. It took VMware a couple of months, but finally, a simple solution was found.

The Solution

The issue is due to an untrusted certificate associated with the signed .cab file being downloaded. Since the file is digitally signed, the untrusted certificate was causing the issue. At some point earlier this year, the Shavlik Windows patches began using a new Verisign certificate that was not installed on my Windows Server 2008 server (not R2). To fix the problem, follow these steps:

Using Firefox (a VMware recommendation), download the following cabinet file: https://xml.shavlik.com/data/pd5.cab

Right click the pd5.cab file and go to Properties.
Go to the Digital Signatures tab, click on the name of the signer, which should be Shavlik Technologies, and click the Details button.
The Digital Signature Details window will appear. Click the View Certificate button.

The Certificate window will appear. Go to the Certification Path tab. Go to the root Verisign certificate and make sure that the Certificate status says “This certificate is OK.”
Click the certificate labeled “Verisign Class 3 Code Signing 2010 CA” and click View Certificate. Ensure that this certificate is valid and trusted. Click the Install Certificate button and the Certificate Import Wizard will appear.
Click Next and choose the Place all certificates into the following store option. Click Browse and select the Trusted Root Certification Authorities store. Click Next and then click Finish.

At this point, the missing trusted CA certificate is installed and your download process should begin working again. In my environment, Update Manager is running under original Windows Server 2008 version and not under the 2008 R2 version. I suspect many people have vCenter and Update Manager running on Windows Server 2008 R2, which may not encounter this issue (I am not sure). VMware Support indicated that a KB article will be published for this issue. I will try to followup and link it once it is published.

Patching HP customized ESXi installs with latest agents and drivers