We recently made the purchase to upgrade our two blade enclosures to HP’s Flex-10 Ethernet technology and last week, with the help of a partner, we performed the install and upgrade of the Virtual Connect domain.
HP Virtual Connect Flex-10 allows an administrator to take a physical 10G connection and partition it into up to 4 more appropriately sized logical NICs presented to a server blade. Virtual Connect Flex-10 incorporates all of the benefits of Virtual Connect (previously explained here) while allowing for additional logical NICs. A Flex-10 Ethernet interconnect module with two onboard Flex-10 NICs on a blade will allow for 8 total NICs – something formerly only available if all 8 interconnect bays and the two mezzanine slots were populated with Ethernet modules. This greatly reduces the amount of cabling required at the blade enclosure and condenses the amount of connections required.
The Flex-10 interconnect modules include SFP connectors which are capable of copper or fiber connections depending on the GBIC installed.
Upgrade Process
We expected going into the installation that we could need to recreate the Virtual Connect domain (a domain is the configuration, server profiles and settings for a Virtual Connect deployment on a specific enclosure or set of enclosures). Since we only had 5 server blades installed and operating, we weren’t overly concerned. Our primary concerns were to ensure that the same Virtual Connect MAC and WWID settings were reinstalled and that our recreated server profiles matched their MAC address and WWID settings from the prior installation. So, to ensure we had all the information, we made printouts of the server profiles, Ethernet networks and fiber channel configuration and then proceeded to installing the modules.
We shutdown all of the existing blades from service. We removed the existing VC-Ethernet modules from Bays 1 and 2 and we installed the new VC-Ethernet Flex-10 modules in these bays. The VC-Eth modules in Bays 1 and 2 contain the domain configuration. Once removed, the domain had to be completely recreated. The new modules were at a higher firmware (2.12) versus the original modules (2.01).
What surprised us during installation was that we were able to successfully restore our original Virtual Connect domain configuration onto the Flex-10 modules. Once restored, we were able to see all configuration and the only thing showing an error were the Ethernet networks and shared uplink sets. These didn’t have any active ports assigned to the uplinks. The new VC-Eth Flex 10 modules have a different port numbering to differentiate from the original VC-Eth modules. After reassigning ports to the uplink sets, all of the Ethernet networks appeared to be back online. The firmware differences didn’t appear to be a problem after initial restore.
After booting servers onto the enclosure (all are boot-from-SAN and booted fine), we determined that we were having some network connectivity issues. After troubleshooting, we upgraded VC firmware on the modules. This proceeded without problems, but upon reboot, the VC-Ethernet Flex-10 modules would not come back online. After troubleshooting, we installed upgraded firmware on the enclosure. The newer firmware on all seemed to resolve all issues and the enclosure was back online. Final time to upgrade was around 5 hours, but much of that was waiting on Virtual Connect modules to reboot and wondering why they wouldn’t come back online.
Our second enclosure was expected to be a simple slam dunk after our largely successful and simple upgrade procedure — especially since we knew about firmware before on this round, but unexpected problems were there. This enclosure had a mid-plane replacement early in its life and apparently there was confusion with the Virtual Connect domain and the enclosure serial number. We received errors during restore and even after we forced the restore and upgraded firmware, the Ethernet networks never came back and responded.
This forced us into a recreate scenario. After recreating this VC domain by hand, everything came online and worked as wanted. I still believe that this is the cleaner way to handle the upgrade, although the restore worked on the first enclosure and presented very few problems.
Next… Configuring Flex-10 FlexNICs
