Category:

Virtualization

Patching HP customized ESXi installs with latest agents and drivers

by Philip Sellers April 22, 2014

written by Philip Sellers

If you’re running ESXi deployed from custom HP ESXi installation images but you’re not patching the HP utilities inside with each ESXi update, you may be setting yourself up for trouble in the future. And simply pointing to the VMware repositories won’t update these custom tools — you also have to point towards the HP vibs repository to get all of the latest updates. So, lets look at how to do that…

Updates & Patching for ESXi

VMware has done a good job of providing administrators ways to keep their ESXi deployments up to date – both with offline bundles and with VMware Update Manager for patching the ESXi systems. Either method is pretty simple and HP leverages these same tools for its custom vibs.

For HP, administrators should point to http://vibsdepot.hp.com for the latest and most up to date information about the correct recipe of VMware and HP tools and driver versions. This is also the repository for automated updates that can be plugged directly into VMware Update Manager – or can be manually downloaded and loaded into Update Manager or used as offline bundles.

Why not just deploy VMware’s default ESXi?

Some administrators adopted the HP customized ESXi image early and others have held out until the latest ProLiant Gen8 hardware was released. The latest ProLiants need storage and network controller drivers included in the customized HP ESXi image to successfully deploy ESXi on Gen8. So now, a growing community has deployed additional HP utilities into their ESXi as well. In my own experience, that’s not been an issue, however I did hit a problem a couple months back where excessive logging from one HP tool filled up my ESXi RAMdisk. Filling the RAMdisk made the system unmanageable even though the VM’s continued to operate on the host. So like any other software, you need to patch those drivers, tools and utilities from HP if they’re present in your ESXi.

Patching options for HP customized deployments

As mentioned earlier, the HP updates are deployed using the same VMware provided tools as all other ESXi patches. Administrators can use VMware Update Manager for a more automated deployment, ESXCLI for command line or scripted installations or they can use Image Builder tools to package the HP vibs into images that they are creating for deployment. Since I’m primarily focusing on patching, Update Manager and ESXCLI would be our two go-to tools.

Update Manager Repository

For Update Manager, the easiest way to get the HP updates into Update Manager and onto hosts is to point to a new repository in the Update Manager tool. Adding an additional repository in Update Manager must be done through the Configuration tab of the Update Manager settings using the traditional Windows vSphere Client. It doesn’t appear to be configurable in the vSphere Web Client.

Navigate to Solutions and Applications and choose Update Manager.
On the Configuration tab, under Settings, click Download Settings.
In the Download Sources pane, select Direct connection to Internet.
Click Add Download Source.
In the Add Download Source window, type the new download source URL. For the HP repository, the URL will be http://vibsdepot.hp.com/hpq/<release date>/index.xml.
Tip: You may navigate to http://vibsdepot.hp.com/hpq/ to see a directory list of all the release dates.
Click Validate URL to verify that the URL is accessible.
Click OK and then Apply.

Once you have setup the custom repository, click Download and let those updates be imported into Update Manager. Once imported, rescan your hosts and now the HP customized tools will also be remediated during patching.

Update Manager Manual Import

Alternatively, you can download the individual zip files of the updates from the http://vibsdepot.hp.com website and import the individual updates into your local Update Manager library. Once imported, make sure these updates are included in a baseline and attached to a host and rescan the host. The new patches will show up and you’ll be able to remediate the hosts with these patches.

ESXCLI

For users who want to script, who want or need to use ESXCLI, one additional line will pull down and deploy the updates from an HP release to your ESX hosts:

[code]esxcli software vib install -d http://vibsdepot.hp.com/hpq/<release date>/index.xml[/code]

Just replace the <release date> in the URL with a correct release date to point to an active repository.

Limited Time Only

With both Update Manager and ESXCLI, keep in mind that you’ve configured and/or remediated against a single, point-in-time HP repository, so you will also need to add new repositories as the patches change and new updates are released from HP. You may navigate to http://vibsdepot.hp.com/hpq/ to see a directory list of all the release dates. Configuring against a single release is the recommended HP method to ensure you get the correct patches according to the recipe for your ESXi patch level.

Recipes

HP also releases a list of recipes for which version of tools and drivers work with which version of ESXi and its patch levels. These recipes give administrators additional validation that the tools loaded have been tested with the ESXi patch level and everything should run correctly together. The recipes are available on the same site with the patches and downloads at http://vibsdepot.hp.com. The PDF recipes highlight more than just the software aspect of the ESXi and HP tool integration – it also highlights the firmware levels desired by ProLiant make and model to ensure that everything has been tested together.

April 22, 2014 3 comments

Compute Datacenter Software Storage Virtualization

HP announces OneView integration with vCenter Server

by Philip Sellers February 10, 2014

written by Philip Sellers

oneviewheader

Today at VMware Partner Exchange (PEX), HP announced vCenter Server integration for its HP OneView systems management package. Using REST API calls to HP OneView, the vCenter plugin is now able to initiate actions in HP OneView directly from the vSphere Web Client. HP OneView, introduced last September, is a central platform of the Converged Systems group and it is the glue that takes disparate components and makes them into a cohesive solution. The current version has support for HP BladeSystem, Virtual Connect and Gen8 ProLiant blades and rack-mount systems. Proliant G7 blades are also supported. It has a roadmap for adding support with HP networking and HP storage in the future. HP OneView provides both deployment, management and monitoring capabilities so that administrators can build out solutions and maintain solutions.

HP Grow Cluster - vCenter with OneView

As part of a demo of the integration, HP is showing off the ability to completely automate the build and deployment of ESXi nodes in a new cluster and into existing clusters. By utilizing OneView’s engine, a VMware administrator can kick off the orchestration of a new hardware build directly from vSphere Web Client using a simple “HP Grow Cluster” menu item (pictured above). The menu item kicks off a wizard drive workflow (at right) that calls the tasks in OneView via its REST API. The vCenter plugin allows the administrator to pick a reference server profile, create multiple server profiles from the template and then uses HP Server Provisioning to deploy an ESXi host build plan.

At the core of OneView is the software-defined model of IT – where the underlying hardware is abstracted from the software-based configuration and settings. The reference server profiles define a base level of hardware and combined with settings, including firmware levels, BIOS settings and the ProLiant server model and generation preferences. Once the reference profile is deployed, it becomes a server profile assigned to physical hardware. Like Virtual Connect server profiles, these are portable and can be migrated between physical hardware.

Customers who are familiar with HP’s Insight Control for vCenter will notice many similarities between the OneView integration and the Insight Control integration. The OneView plugin for vCenter shows network diagrams of port mappings and configuration as well as end-to-end storage mapping from the VM to the LUN and storage array. The current iteration of OneView relies on the same Insight Control for vCenter storage plugin. OneView has yet to adopt storage control and management directly into its platform, though it seems safe to expect its coming given HP’s direction.

The majority of this announcement is that VMware administrators can now kick off OneView tasks directly from the vSphere Web Client, monitor and view the progress and completion of the tasks it spawns. OneView is still in early in its life, but the interface is simple and intuitive. Its not overly complex at this time and its build with modern concepts, like search, deeply integrated into the core product. Extensions like the vCenter plugin are great enhancements for administrators who are accustomed to working in a native platform and gain access to the capabilities of OneView directly from a familiar console. This release continues on HP’s track record of delivering usable functionality directly to VMware administrators from vCenter.

The HP team has a video demonstration of the HP Grow Cluster feature in the vCenter Web Client posted at hp.com. It is definitely worth a watch to see the entire process orchestrated with just a few clicks and settings.

February 10, 2014 0 comments

Datacenter Storage Virtualization

Fixing ESXi when RAMdisks run out of disk space

by Philip Sellers January 27, 2014

written by Philip Sellers

I ran into an interesting problem a few weeks back with one of my ESXi hosts. While trying to do some normal things – like vMotion – I noticed an error recorded for the tasks – nothing that seemed to point to a lot of detail – just “A general system error occurred.” On further investigation, I found that the underlying message was an out of disk space message while trying to proceed with a Storage vMotion.

Observed errors

Attempting vMotion – “A general system error occurred:”

Attempting Storage vMotion – “/var/log/vmware/journal/xxxx error writing file. There is no space left on the device.”

Troubleshooting steps

Go to Configuration tab on host in vCenter client, go to Security Profile, click Properties link on the Services section.
Scroll down to SSH and highlight – click options – click start to start SSH service.
Use putty or reflections to ssh to the host.
If you get a connection rejected – root filesystem ramdisk is probably full.
Go to console (either through KVM or OA for blades)
F2 to login, login, arrow down to Troubleshooting Options, select Enable ESXi Shell.
Press ALT-F1 to change to management shell and login (same root credentials).

Run ‘vdf -h’ and look for root filesystem – should look like:

Ramdisk                   Size      Used Available Use% Mounted on
root                       32M        3M       28M  10% --

If it is 0M available and 100% used, that’s the problem. Try to clear up space:
```
cd /var/log/
 ls -la
```
Check size of the hpHelper.log file – likely pretty large. Reset the file, if large.
```
> hpHelper.log
```

Host is back online and working and it looks like the most likely culprit was the HP agents inside of the custom ESXi image provided by HP. It seems in some circumstances that the hpHelper.log file can become very large, filling the RAMdisk and causing the issues. Its a first for me and I have not observed the issues on any of my other ESXi hosts running on Proliant rack-mount or blade servers.

Helpful Knowlegebase Articles

HP KB – HP ProLiant Gen8 Servers – ESXi 5: The /var/log/hpHelper.log File Can Grow Very Large and Fill the ESXi 5 RAMDisk
VMware KB – vCenter Server task migration fails with the error: Failed to create journal file provider, Failed to open for write (1019598)
VMware KB – Investigating disk space on an ESX or ESXi host (1003564)
VMware KB – RAM disk is full (2001550)
VMware KB – ESXi RAM disk full due to a locked virtual machine log file (2057975)

January 27, 2014 4 comments

Datacenter Storage Virtualization

VMware releases VSAN and the power of policies

by Philip Sellers September 19, 2013

written by Philip Sellers

VMware Virtual SAN (VSAN)
Diagram by VMware

VMware is pushing back on storage vendors as it seeks to show the world the full power of software-defined storage. A couple weeks ago, VMware announced the VMware Virtual SAN (VSAN), a new software-defined storage platform tightly coupled with vCenter server. The new platform requires both hard disks and solid state disks to create a pool of redundant, shared storage within a cluster. The hard disks are the core of the storage and the SSD are used for buffering and read caching but do not contribute to the total amount of storage. And although the name has SAN in it, the traditional SAN that is bread and butter of its parent company, EMC, could not be more different than VSAN.

In many ways, this is similar to the VMware Virtual Storage Appliance, but VSAN is much more integrated into vCenter and the vSphere platform. The requirement of SSD is also new and will greatly contribute to overall responsiveness for the storage. But more importantly, the new software defined storage platform is an example of the possibilities yet to come from tightly integrating storage with vSphere.

The real power of what VMware has released is not the storage technology itself or the simplicity that it can be deployed (single-click simplicity like HA and DRS). The real power is the addition of Storage Policies completely throughout vSphere and VSAN. The policies are the real capability that can be leveraged across multiple types of storage. As I said earlier, I think VMware is simply showing all storage vendors this is what is possible. The VSAN is certainly not a SAN replacement for most enterprise customers, but there are certainly remote office or small office use cases where this will fit nicely without the need of purchasing a purpose built storage array — use cases, by the way, where virtual storage appliances (VSA) currently fit.

But back to the policies. Introduced in vSphere 5.0, Storage Policies allow administrators to tag and set classing of storage for different pools that are available within a vSphere cluster. The Storage Policies allow an administrator to abstract the back-end storage and provide a list of capabilities to end-users or operators who are creating new virtual machines – and choose a policy during initial deployment to meet their needs. You see, what VMware has discovered is that the end users don’t care if the back-end storage is a 3PAR or a VMAX – a Synology or a Drobo. What they care about is a class of service provided, response times and how those translate to their application and its performance. They also care about the cost of charge-back, if an organization has implemented those types of billing scenarios. Policies are all a way of defining those attributes back to the end user and abstracting the actual storage from those decisions.

I am sure many VMware customers are like myself. When I started the virtualization project, most systems ran on local storage. The SAN was reserved for our most mission critical systems and most of those were clustered. Today, however, vSphere has moved all of those workloads onto the SAN. It has increased my resiliency and ability to survive a disaster, but its also driven storage costs up exponentially. Yes, VMware is owned by EMC – goal to sell disks accomplished.

But the VSAN must run a bit counter to the EMC culture, so back to my original point. In many ways, I think the VSAN is simply a proof of concept of what is possible with software defined storage. It is a way for VMware to show the rest of the world what we can do. VMware is very good at fostering its ecosystem and partner vendors, so I think this will spur new innovation.

And so, for a third time, back to policies. With the introduction of VSAN, VMware is extending this capability of Storage Policies to allow administrators to assign a policy to a VM on creation and then the policy lives with the VM allowing policy based management of the VM over time. Not only can it be setup initially with the VM, it can follow and adapt as the back-end storage changes over time. With the addition of storage pools and maintenance mode for storage, VMware has automated a lot of the manual labor around managing storage. By creating the pools of storage and moving datastores in and out of them, administrators and move or remove storage without interrupting service of virtual machines. And also by applying tags and policy around those storage pools, it allows for the addition and removal of datastores as required to onramp or offramp storage as purchasing or lease cycles dictate. All of this while maintaining service levels and agreements for customers and workloads – whether those customers are internal or external to your organization.

Beyond backend maintenance of storage, say the attributes of a particular application or VM changes over time. Say it now requires a higher tier of storage – well thats an easy policy change as well on the VM and with the magic of storage vMotion and fully-automated storage DRS, the vSphere platform will perform the necessary moves to increase performance and move the workload onto the appropriate storage de jour.

In many ways, this is all done with the idea of reducing the menial tasks that an administrator has to perform. Its mean to kill the tracking spreadsheets and other mechanisms we have traditionally relied upon for classing storage or moving workloads appropriately. It all goes to making the storage layer of vSphere smarter and more automated.

Now, the last piece of policy that really sets VSAN apart from the competition. Within VSAN, the vSphere administrator can dictate the type of RAID, the number of stripes and copies that are present for a particular VM or datastore. For the first time, the core vSphere platform is controlling storage down the stack and automating how it operates. vSphere policy now defines where and how replication occurs, how and where stripes of data are implemented, and on a per VM basis, quality of service attributes to prevent noisy neighbor problems. And combining those together gives context and visibility up the stack in vSphere, too. The two way communication of the policy and the storage implementation means that the administrator can more granularly dictate how the storage should act to support the requirements of the virtual workload.

So, I have to say, after a week in San Francisco at VMworld 2013, the VSAN is the one piece I am most excited to test and evaluate. If for no other reason, you should sign up for the VSAN beta to see where the future of storage is going. I think the writing is on the wall of every vendor is talking software-defined, the VSAN represents the first software-defined storage product with such a deep level of integration into vSphere. I fully expect to see other vendors like Nexenta and HP StoreVirtual quickly adopting this paradigm – maybe even making it to market with product offerings around the same time that VSAN becomes generally available. API calls and other integration points should be very simple for software storage implementations. I’m excited to see where this next step takes us.

September 19, 2013 0 comments

Virtualization

VMware community unites to raise money for cancer research

by Philip Sellers September 16, 2013

written by Philip Sellers

20130913150122-1600-Podcaster_mounted One tweet has the power to really ignite something, as Gabriel Chapman (@Bacon_is_king) saw last week. A quick, but personal post, about his mom’s cancer diagnosis a year after losing his father to the disease has quickly turned into a grassroots effort to raise money for cancer research in the family’s name.

The VMware community is very tight. And it doesn’t surprise me to see such a warm response and so much support from the community for one of its members. Within minutes a proposal was thrown out on Twitter and the Podcasting for Cancer movement began. So, here’s your chance to support someone in the industry and to help an amazing cause at the same time. Proceeds are going to the American Cancer Society. The initial goal is $5000 donation and the organizers have just broken that amount – but how much further can this go?

Most everyone has been touched by cancer in some way – whether it was an uncle lost to the disease 3 years ago, like me, or whether its your dad and now mom like Gabriel. And you can do something to help… I wouldn’t ask you to donate if I hadn’t already done so myself. Read all about it and get involved here – http://www.indiegogo.com/projects/podcasting-for-cancer.

September 16, 2013 0 comments

Virtualization

HP Virtual Performance Viewer offers heat map for virtual performance

by Philip Sellers September 11, 2013

written by Philip Sellers

A couple weeks ago at VMworld, I stopped by the HP booth to see what they had new for this year and I was introduced to HP Virtualization Performance Viewer (vPV). Presented as a virtual triage tool, the idea behind vPV is to quickly let a virtualization administrator identify host spots of CPU or Memory that deviate from their baseline so they can pinpoint and fix problems quickly.

vPV comes in two editions – a free edition supporting up to 200 VM objects and a paid for edition with no limit.

The interface is clean and shows an overview of the entire environment, color coded – with shades of greens being good and oranges and red representing the problem areas. What is even better is you are able to drill down into a specific host or VM by double clicking one of the blocks. Drill down is simple and acts as it should, filtering the results to a specific cluster, resource pool, datastore or host to view what is happening inside of that object. This provides quick information about why a particular object is deviating from its norm.

A simple toggle radio button on the right side allows users to switch between vSphere and Hyper-V in the interface.

Screenshot of the HP Virtualization Performance Viewer dashboard — click for larger image

Hypervisor Support

vPV documentation says the system supports both VMware vSphere, Microsoft Hyper-V, KVM and Xen hypervisors. In addition, it supports OpenStack OS in a private cloud deployment and HP Cloud Application Platform as a Service (Paas).

In the free edition that I downloaded, I was able to see where I could configure against VMware vCenter or Microsoft System Center Virtual Machine Manager, but not against the other hypervisors or hosts listed. vCenter requires no agent. For Hyper-V, it appears SCVMM is required and to communicate with SCVMM an additional data collector must be installed. I did not see a requirement for SCVMM in the documentation, however.

HP is very big on the message of customer choice and support across heterogeneous virtualized environments. This is another product supporting the message, in my opinion.

Installation

HP documentation says that deployment should take about 15 minutes and I found that to be true. vPV is packaged as a virtual appliance, so the installation into VMware was quick and simple from an OVA template. After spinning up the OVA, configuration against vCenter was simple, too, except I had too many VM objects for the free edition and so vPV presented an error. I was able to begin testing against a subset of VMs by creating a special username and limiting permissions to specific hosts, so the free tool can be useful even in a larger environment against a subset of virtual guests.

Free versus Licensed

The free edition of vPV does not provide any security for the application. If you can get the URL for your browser, you are in and able to make configuration changes. The licensed edition includes AD integration and password protection for the application. The licensed edition also includes 3 months of data retention, while the free version only includes 1 day.

Integration

The licensed version of vPV has integration points against other HP management software packages – HP Performance Manager and HP Business Software Manager (BSM).

Support

For the free edition, community support is available from an HP support forum.

Video Overview – How to install vPV in under 15 minutes

More Information

September 11, 2013 0 comments

Datacenter Storage Virtualization

VMware unwraps vSphere 5.5 – What’s in it for your Datacenter?

by Philip Sellers September 9, 2013

written by Philip Sellers

Almost two weeks ago now, VMware unwrapped vSphere 5.5 to the public. The release, which ships later this year, includes a lot of enhancements and some new features for the datacenter. I won’t lie – 5.5 has a lot more in it for the datacenter than 5.1 enhancements. And I left VMworld 2013 much more excited about the direction of the virtual datacenter than I did at the end of VMworld 2012. So what are the things that sys admins will really care about in vSphere 5.5?

Expected enhancements – bigger maximums and more Web Client

As expected, the new release of vSphere increases maximums. The ESXi host maximums were doubled from vSphere 5.1 to vSphere 5.5 in all cases, except for the maximum LUN size, which got a surprising 32X bump in limit to 64TB. The number of logical CPU’s doubled to 320, the number of virtual CPU’s doubled to 4096 and the amount of RAM has doubled to 4TB per host. Those represent some pretty incredible numbers. vSphere 5.5 also introduces Virtual Hardware version 10. The new hardware version is introduced to support newer CPU architectures as well as a new virtual-SATA Advanced Host Controller Interface (AHCI) with up to 4 controllers and 30 devices per controller. So, the new AHCI supports 120 total devices connected to a single VM.

Also expected, vSphere 5.5 drives users more towards into the vSphere Web Client for management. All new capabilities are controlled from the vSphere Web Client. This follows suit with vSphere 5.1, whose new capabilities like shared-nothing vMotion, were available only in the Web Client. One thing I am hearing repeatedly on social media is that the vSphere 5.5 Web Client is noticeably more responsive, which is great. I am also hearing the new Virtual Hardware 10 VMs can only be managed through the Web Client, which could cause heartburn for stand-alone hosts, but the final release may change this functionality. The beta version of the Windows VI Client throws an error if you try to edit a Virtual Hardware 10 VM. The error says the version 10 VM must be managed through the Web Client. This just means that resiliency of the vCenter servers continue to be critical parts of virtual deployments as it is a required part of managing the infrastructure.

And speaking of vCenter Server, the vCenter Server Virtual Appliance (VCSA) continues to become a more capable alternative to a Windows server deployment of vCenter. With vSphere 5.5, the VCSA has a retooled embedded Postgres database capable of supporting 100 hosts and 3000 VMs, although those numbers may actually change for better or worse. During one Ask the Expert session during the week, a member of the VMware Labs team said they were using a single VCSA to support as many as 500 hosts, but mileage my vary. This is one of those numbers that may change during final release.

I’ll file SSO enhancements under expected as well. While most people outside of VMware did not know what VMware intended to do with SSO, one session at VMworld pretty much summed it up by saying that the new version of vSphere SSO releasing with vSphere 5.5 will be what should have been released the first time. To many adminstrators, myself included, the version of SSO introduced with vSphere 5.1 just didn’t seem up to VMware standards. It seemed to be a third party tool that was included but not fully streamlined into the vSphere implementation.

The vSphere 5.5 SSO version includes a number of streamlined improvements. AD integration is much improved and includes the ability to tie it to multiple forests, which is needed for many larger environments – or anywhere a merger has occurred. There is no longer a need for an external database. And there is a multi-master replication model built into this new version.

Storage improvements

There are many storage improvements included in vSphere 5.5. One that I am very happy about is the streamlined and scriptable UNMAP process used to identify and clear out free space on a VMFS datastore so that your storage array can re-thin and free the space on the back-end. This is a big one for me with 3PAR. I want to keep my LUNs on a diet and this new ESXCLI command will go a long way towards helping me do that. In addition to the new command line, the back-end functionality has changed with UNMAP, hopefully making it more acceptable to run during production hours and moving towards a time when automatic UNMAP during deletes might return to default functionality. UNMAP can now stagger the reclamation process so that it doesn’t saturate a backend array with IO.

Another nice feature, from an administrative standpoint, is the automatic removal of permanent device loss (PDL) state feature. This is especially handy in the event of array failures, which have been known to happen. In the event of a failure, automatically clearing a LUN in PDL will keep the administrator from having to perform rolling reboots to completely clean up the mess. The handling of PDL conditions has improved greatly over the versions I have used from 3.0 forward. There as a time that a PDL could completely hose an ESX host and a reboot was the only way to recover.

vSphere 5.5 also features enhancements around SSD and Flash storage. The ESXi hosts now have the ability to work with add-add and removal of SSD devices, allowing a graceful handling of SSD storage as it becomes increasingly more common. In the world of Flash, a new read cache feature is introduced allowing for the aggregation of Flash storage into a pool that can be consumed by VMs for the purpose of accelerating read intensive workloads. Last, but not least, 16Gb Fiber Channel support, end to end, is available in vSphere 5.5.

Network enhancements

In the networking world, vSphere Distributed Switches just got a needed boost of capabilities. Traffic filtering has been added allowing administrators to filter the types of traffic. QOS has been added to the mis allowing for prioritization of traffic from the vSwitch up to the physical switch layer. Security officers and admins troubleshooting will both be happy to see the addition of a host level packet capturing tool that is the equivalent of tcpdump to Linux admins. 40Gb NIC support is also being introduced with vSphere 5.5.

Mission critical workloads

One of the message points from VMware during VMworld 2013 was about making vSphere a platform that can run any mission critical workload. The maximum enhancements and other resiliency features are all targeted towards that goal, but in particular one additional capability was introduced with vSphere 5.5 that is meant to assist with virtualizing latency sensitive workloads, like VOIP. A new Latency Sensitivity setting is now available on VMs which help reduce latency for necessary virtual workloads. When set to a ‘high’ setting, the hypervisor actively tries to reduce latency by reserving memory, dedicating CPU cores and disabling network feature prone to latency.

Tying it all together…

There is really so much more in vSphere 5.5 – a couple great places to get more detail is available from Kyle Gleed in his A Summary of What’s New in vSphere 5.5 post and in the What’s New in vSphere 5.5 document on the VMware website.

All in all, I left the Monday morning keynote of VMworld 2013 feeling a little let down. Many of the enhancements that will really improve the admin’s life didn’t get any stage time or mention during the keynote – one reason for the delay in posting details – I had to find details… Two of the big deals that were given stage time this year were the VMware VSAN, a software defined virtual storage implementation baked heavily into vSphere, and NSX, the network virtualization platform that VMware acquired with Nicira. I hope to post on these two in depth in the coming weeks.

I was very happy to see a bit of a return by VMware to its core audience and products – centered around datacenter virtualization. I had the opportunity to talk to a lot of other bloggers and industry folks while I was at the conference and I found several who shared my view that VMware had strayed from its roots in years past chasing the cloudy dream — a dream that we felt was a bit beyond the masses. And while cloud adoption is growing, most enterprise datacenters are still reliant upon VMware and its vSphere platform to running daily operations. Its good to see that VMware did not abandon us all and that they are are still working very hard to build the best and most resilient datacenter platform available.

Questionable decisions in years past – like the nasty 4 letter word – VRAM – can now be jokes and chuckles. With so much attention given to vCloud and programming layers in the past couple years, it was welcoming to see things added to vSphere, where I live and work. In my opinion, this year VMware had to deliver something valuable to all of its customers to starve off competition and justify the continued spend for Service and Subscription (SnS) contracts. In many ways, I think they delivered. I’ll save my thoughts on where they still miss for another post.

September 9, 2013 0 comments

Datacenter Storage Virtualization

Keeping VMFS datastores thin on 3PAR StoreServ

by Philip Sellers July 31, 2013

written by Philip Sellers

Earlier this year, I moved a large part of my infrastructure onto HP’s 3PAR StoreServ 7400. As promised, our initial migration netted us over 50% savings through the thin technologies baked into the 3PAR StoreServ platform. But one my biggest concerns was how well could 3PAR maintain its lean status over months of operation with my vSphere running there. (Looking for how to free space on 3PAR, jump down to 2 Commands to Reclaim section below.)

After several months of running on 3PAR, I can report that it does a noble job identifying and thinning the storage under my vSphere environment. But, its not perfect. 3PAR can only identify what’s truly zeroed and on a VMFS5 datastore, that equates to space that has not been utilized. vSphere doesn’t clean up its VMFS datastore in vSphere 5.1. And this behavior has nothing to do with 3PAR.

History of Space Reclamation in vSphere

With vSphere 5, VMware introduced a new primitive using the SCSI UNMAP command to its VMware APIs for Array Integration (VAAI). Billed as VMware’s Block Space Reclamation feature, it allowed vSphere to communicate with the storage array and identify which space was no longer occupied by a VM and could be reclaimed. VMware shipped vSphere 5 with the EnableBlockDelete advanced storage feature enabled by default.

Unfortunately, the Block Space Reclamation feature induced overhead on the arrays and caused varying performance impacts on systems, in some cases causing severe performance degradation. Ultimately, VMware decided to disable the EnableBlockDelete advanced feature by default and disabled is the recommendation for production environments. Fast forward to vSphere 5.1 and VMware has not reversed its recommendation.

So where does that leave customers with bloated filesystems? The good news is that you can run a manual UNMAP using the vmkfstools command on a particular LUN. Effectively, the manual UNMAP writes a large balloon file on the filesystem which is used to zero out the space that was previously occupied so that the storage array can identify it as free space.

On 3PAR, once the vmkfstools command is run, 3PAR then begins its discovery of freespace through the Zero Detect feature and over the next several hours will begin reporting true free space.

2 Commands to Reclaim

The command is easy for initiating an UNMAP in vSphere. But, you have to do a bit of prep work. UNMAP is supported with 3PAR StoreServ arrays running HP 3PAR OS 3.1.1 or later. See VMware KB article 2014849 for the details on how to confirm to determine support for other arrays.

You will need to enable SSH on the ESXi 5.0 or 5.1 host.
To do so, go to the Host and Clusters view in vSphere Client, choose the host from the tree on the left side, choose the Configuration tab in the right side, choose Security Profile under the Software section, then click Properties… under the Services section. Scroll down to find SSH, double-click and then start the service. ESXi will display a warning that SSH is enabled. When you complete running your commands, don’t forget to disable SSH.
SSH into the host. Navigate to the VMFS datastore you would like to perform the UNMAP on.
Once you attach to the host using SSH, issue

[code] cd /vmfs/volumes/<datastore_name>[/code]

Now that you’re locate inside the LUN, you are ready to perform the UNMAP. The actual command is simple.

[code]vmkfstools -y 99[/code]

Using the -y parameter, you specify the integer percentage of free space to reclaim. If the LUN has 300GB of free space reported in vSphere, the command will create a balloon file that is 99% of 300GB in size (297GB). Once the file is created, it is removed zeroing the space. You should adjust your percentage as you see fit. Starting somewhere in the 80’s would be a good place to start.

The safest recommendation is to run the storage reclamation outside of business hours or at least outside of peak business hours. Impacts may vary, but in my experience the 3PAR handles the requests easily .

July 31, 2013 2 comments

Datacenter PowerCLI Software Storage Virtualization

Reload inaccessible or invalid VM’s with this PowerCLI one-liner

by Philip Sellers June 3, 2013

written by Philip Sellers

Can I let you in on a secret? Although I blog to help others through my own experiences and troubleshooting, one of the other things I do is use my blog as my own personal knowledge base. Once I’ve hit something, if I blog it, its there for me as much as you…

One of the problems I have run into several times are “inaccessible” or “invalid” VM in vCenter after a storage outage. The only solution that I had found was cumbersome. I had to unregister the VM from vCenter, browse to the appropriate datastore (which is more difficult today with storage pools obfuscating view), then locate the VMX file and register again. While it works, its neither elegant nor fast.

But, thanks to Matt Vogt’s and Mike Letschin’s research and blog post, there is a much more acceptable solution to recover from this. Matt’s post focuses on NFS storage, but from my personal experiences, its possible with a Fibre Channel outage and iSCSI, too. If you have a storage outage and then discover “inaccessible” or “invalid” VMs, you can try the reload function through PowerCLI. For the full details, check out Matt’s blog on the topic.

Matt links to a French language blog, hypervisor.fr, and I really like the PowerCLI one-liner that I found there. It was quick and effective to reload the VM’s.

[code lang=”powershell”](Get-View -ViewType VirtualMachine) |?{$_.Runtime.ConnectionState -eq "invalid" -or $_.Runtime.ConnectionState -eq "inaccessible"} |%{$_.reload()}[/code]

June 3, 2013 1 comment

Datacenter Software Storage Virtualization

3PAR StorServ 7000 series best practices for vSphere 5.1

by Philip Sellers April 22, 2013

written by Philip Sellers

This is part 4 in a multi-part series about 3PAR arrays from an EVA Administrator’s point of view – and specifically here, for the VMware Administrator, too. Running 3PAR StorServ with vSphere 5.1 comes with a short list of best practices from HP, which is great since VMware ESXi does most of the work for you. The best practices for EVA are the same as the best practices for 3PAR StorServ, except the IOPS count. In summary, the best practices are:

In 3PAR Management Console, set the host persona to host persona 11/VMware if running 3PAR OS 3.1.2 or later, which ships with the StorServ 7000 series arrays.
Set the default VMware Path Selection Policy (PSP) to VMW_PSP_RR.
Set the default VMware Storage Array Type Plugin (SATP) to VMW_SATP_ALUA.
Set the IOPS value for 3PAR data LUNs to a value of 100.

Create a Host Persona for the ESXi server

Using the 3PAR OS CLI, log into your 3PAR array and you will need to create a host where you can export your storage. Since this post focuses on the 3PAR StorServ 7000 series, these units ship with OS 3.1.2, so you should use host persona 11 – the VMware ALUA persona. 3PAR OS 3.1.2 includes support for asymmetric logical unit access (ALUA) and the host persona 11/VMware and new ESXi hosts should use this persona according to the HP 3PAR VMware ESX Implementation Guide, revision March 2013.

[sourcecode]createhost -persona 11 ESXiHostName WWN/iSCSI_Port_ID1 WWN/iSCSI_Port_ID2[/sourcecode]

With a host created in the 3PAR, you can now export Virtual Volumes (VV) to hosts. But before you export any LUNs from the 3PAR, you will want to configure your ESXi hosts with a default storage array type plugin and path selection policy.

Set the default SATP to VMW_SATP_ALUA and PSP to VMW_PSP_RR

With 3PAR OS 3.1.2 using host persona 11/VMware in your hosts objects on the 3PAR array, the array supports ALUA and we should set your storage type array plugin (SATP) to match by default in VMware – otherwise, you have to set this setting on each LUN presented/exported. In addition, the recommended path selection policy (PSP) is Round Robin. Most Recently Used (MRU) is also acceptable, but not the optimal path selection policy. To set this on your ESXi hosts, use the esxcli below. (If you haven’t used vSphere CLI/esxcli – see this blog post at VMware for some tips on how to provide credentials).

[sourcecode]esxcli storage nmp satp set -s VMW_SATP_ALUA -P VMW_PSP_RR[/sourcecode]

For anyone wanting a way to accomplish this in PowerCLI, you should check out Jonathan Medd’s post adapting the esxcli command to PowerCLI. If you have storage that was presented prior to setting the default, you will need to go through and set these LUNs to Round Robin from MRU, which is the out-of-box default.

[sourcecode]Get-VMHost | Get-ScsiLun -CanonicalName "naa.600*" | Set-ScsiLun -MultipathPolicy "roundrobin"[/sourcecode]

You will also need to create a custom rule to set the IOPS value on each LUN added.

[sourcecode]esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -P "VMW_PSP_RR" -O iops=100 -c "tpgs_on" -V "3PARdata" -M "VV" -e "HP 3PAR Custom iSCSI/FC/FCoE ALUA Rule"[/sourcecode]

Now for the same in PowerShell, here’s a short group of commands to accomplish the same. First, you have to connect directly to a ESX host and to a vCenter instance. Second, issue these:

[sourcecode lang=”powershell”]Connect-VIServer -Server [servername]
$esxcli = Get-EsxCli
$esxcli.storage.nmp.device.list() | where {$_.device -like "naa.600*"} | %{
$configBefore = $esxcli.storage.nmp.psp.roundrobin.deviceconfig.get($_.device)
$esxcli.storage.nmp.psp.roundrobin.deviceconfig.set(0, 1, $_.device, [long]100, "iops", $false)
$configAfter = $esxcli.storage.nmp.psp.roundrobin.deviceconfig.get($_.device)
# Uncomment the following lines if you want to report the settings
# $configBefore
# $configAfter
}[/sourcecode]

Lastly, in earlier versions of 3PAR OS, you needed to add a plugin for VAAI to ESXi in order to gain the benefits of VAAI – offloading processes directly to the array from the ESXi host. With 3PAR OS 3.1.1, it is no longer needed –

HP 3PAR VAAI Plug 2.2.0 should not be installed on the ESXi 5.x if connected to an HP 3PAR StoreServ Storage running HP 3PAR OS 3.1.1 or later, because the VAAI primitives are handled by the default T10 VMware plugin and do not require the HP 3PAR VAAI plugin.

Supporting best practices for multiple storage systems in a single VMware environment can be tough, but in my experience supporting both EVA and 3PAR in the same environment is very simple and straight forward if you’re using 3PAR OS 3.1.2. As another tip, Host Profiles work well for picking up SAN settings on a single host and applying them again the other nodes in the cluster. If you boot from SAN, you may have to go into a profile and disable some settings specific to boot LUNs on a host, but otherwise, the storage settings from the host profile should work well for replicating settings.

April 22, 2013 0 comments