Keeping VMFS datastores thin on 3PAR StoreServ

Earlier this year, I moved a large part of my infrastructure onto HP’s 3PAR StoreServ 7400.  As promised, our initial migration netted us over 50% savings through the thin technologies baked into the 3PAR StoreServ platform.  But one my biggest concerns was how well could 3PAR maintain its lean status over months of operation with my vSphere running there. (Looking for how to free space on 3PAR, jump down to 2 Commands to Reclaim section below.)

After several months of running on 3PAR, I can report that it does a noble job identifying and thinning the storage under my vSphere environment.  But, its not perfect.  3PAR can only identify what’s truly zeroed and on a VMFS5 datastore, that equates to space that has not been utilized.  vSphere doesn’t clean up its VMFS datastore in vSphere 5.1.  And this behavior has nothing to do with 3PAR.

History of Space Reclamation in vSphere

With vSphere 5, VMware introduced a new primitive using the SCSI UNMAP command to its VMware APIs for Array Integration (VAAI).  Billed as VMware’s Block Space Reclamation feature, it allowed vSphere to communicate with the storage array and identify which space was no longer occupied by a VM and could be reclaimed.  VMware shipped vSphere 5 with the EnableBlockDelete advanced  storage feature enabled by default.

Unfortunately, the Block Space Reclamation feature induced overhead on the arrays and caused varying performance impacts on systems, in some cases causing severe performance degradation.  Ultimately, VMware decided to disable the EnableBlockDelete advanced feature by default and disabled is the recommendation for production environments.  Fast forward to vSphere 5.1 and VMware has not reversed its recommendation.

So where does that leave customers with bloated filesystems?  The good news is that you can run a manual UNMAP using the vmkfstools command on a particular LUN.  Effectively, the manual UNMAP writes a large balloon file on the filesystem which is used to zero out the space that was previously occupied so that the storage array can identify it as free space.

On 3PAR, once the vmkfstools command is run, 3PAR then begins its discovery of freespace through the Zero Detect feature and over the next several hours will begin reporting true free space.

2 Commands to Reclaim

The command is easy for initiating an UNMAP in vSphere.  But, you have to do a bit of prep work. UNMAP is supported with 3PAR StoreServ arrays running HP 3PAR OS 3.1.1 or later.  See VMware KB article 2014849 for the details on how to confirm to determine support for other arrays.

1. You will need to enable SSH on the ESXi 5.0 or 5.1 host.  
   To do so, go to the Host and Clusters view in vSphere Client, choose the host from the tree on the left side, choose the Configuration tab in the right side, choose Security Profile under the Software section, then click Properties… under the Services section.  Scroll down to find SSH, double-click and then start the service.  ESXi will display a warning that SSH is enabled.  When you complete running your commands, don’t forget to disable SSH.  
2. SSH into the host.  Navigate to the VMFS datastore you would like to perform the UNMAP on.
   Once you attach to the host using SSH, issue

   [code] cd /vmfs/volumes/<datastore_name>[/code]

Now that you’re locate inside the LUN, you are ready to perform the UNMAP.  The actual command is simple.

[code]vmkfstools -y 99[/code]

Using the -y parameter, you specify the integer percentage of free space to reclaim.  If the LUN has 300GB of free space reported in vSphere, the command will create a balloon file that is 99% of 300GB in size (297GB).  Once the file is created, it is removed zeroing the space.  You should adjust your percentage as you see fit.  Starting somewhere in the 80’s would be a good place to start.

The safest recommendation is to run the storage reclamation outside of business hours or at least outside of peak business hours.  Impacts may vary, but in my experience the 3PAR handles the requests easily .