failure

Last night, we undertook replacing our mid-plane in the blade enclosure that had problems last month. It was the first recommendation from HP support after several hours of working with various teams, but both our internal team and our HP field service guys didn’t feel it was the cause. Turns out, we may have been very wrong with our initial hesistance, but I’m getting ahead of myself.

After a month of continued support and working the case, we escalated the case to a level where an engineer reviewed all the steps, troubleshooting, and case information to ensure nothing had been missed and to help diagnose the issue. He came back to the original conclusion – that after all else was eliminated, the mid-plane must be the culprit. So, we scheduled the replacement.

The actual hardware replacement went smoothly and took less than an hour to complete. The midplane is a lot more bulky the I expected when I first saw it. It is a single piece of hardware with interconnects on both sides that connect blades to interconnect bays, power sources to power consumers and LCD display to the logic. But, I guess I was surprised that it was a good 2 to 3 inches thick. In my mind, I expected a single piece of copper sitting in the middle – yes I realize now that’s stupid.

Something interesting occurred after replacing the mid-plane. Apparently, Virtual Connect did not see the system serial number that it expected and so it reverted to its default configuration. So, word of advice to anyone replacing a mid-plane. Leave your VC modules ejected so that you don’t lose your domain configuration. From talking with support, Virtual Connect needs constant communication with the OA to function (another dependancy we were not aware of). The serial number stored reported by the OA from the enclosure is also very important to VC. Its part of the configuration file for VC. It all makes logical sense, but it was not spelled out in the support document detailing the mid-plane replacement.

After unsuccessfully attempting to restore my backup for the Virtual Connect domain, I opted to build it from scratch by hand. It took about an hour and half to do for my 5 blades, but I feel better about it. I am still worried about not being able to restore my VC domain configuration, but I attribute that problem to the hardware replacement.

I can’t believe it, but its been almost a month since my last post. And what a month its been around my work. This has been one of the busiest and most difficult months that I can remember with the company. I have my hands in several different technologies, VMware and our blades are just two of my primary responsiblities. Over the past month, though, we’ve experienced a catastrophic failure of one of our blade enclosures. The failure has only occurred once, but the fall-out from this has taken almost a month to work out. And honestly, we’re still not through working out the kinks.

Of course, my story has to begin on Friday the 13th… Sometime around 9:00am, we started getting calls for both our SQL 2005 database cluster and our Exchange cluster. After investigation, we found that the active nodes were both in the same enclosure and a third ESX host in the same was experiencing problems, too. The problems were affecting both network and disk IO on the blades. All of our blades are boot from SAN, so the IO had to be a fiber-channel issue.

Several hours later, we were finally able to get enough response out of the nodes to be able to force a failover of services for Exchange, shortly followed by SQL 2005. As I worked with HP support, nothing improved on the affected servers. We were finally diagnosed with a problem mid-plane on the enclosure.

While waiting for the mid-plane to be dispatched to the field service folks, I requested that we go ahead and do a complete power-down on the enclosure and bring it up clean. This required physically removing power from the enclosure after powering down everything that I could from the onboard administrator.

After the reboot, everything looked much healthier. The blades came back to life and everything began operating as expected. After intense discussions on the HP side, we reseated our OA’s and the sleeve that they plug into on the back side of the enclosure. Net outcome was the same – everything still operating well. The OA’s nor the sleeve were loose, so we doubted that was the cause.

One nugget I learned from HP support (please vett this information on your own), is that the Virtual Connect interconnect modules require communication with the onboard administrators (OA’s). I’m still not sure I fully understand, but HP support did tell us that if VC lost communication to the OA, its possible that it caused our problems. If this is so, this smells like very, very bad engineering and design…

Continued investigation on HP’s part has pointed us back to the original diagnosis – a faulty mid-plane. Only by default did we return to that conculsion, however. This is the only piece of hardware common to the problems. Our only other conclusion was that this was a very bad, “hiccup” — which obviously buys us no real peace of mind…

So, sometime soon, we will be replacing the mid-plane of our enclosure. I have, of course, lost some faith in the HP blade ecosystem. We have plans to migrate our corporate VMware cluster onto blades, as well as some Citrix and other servers. Losing an enclosure like this has un-nerved those plans. We were fortunate to have drug our feet to only have 3 blades populated and serving anything at the time this happened. I will post updates as we move forward…

Mid-plane is replaced on our blade enclosure

Month of silence, because of a blade enclosure