(For PowerCLI commands to accomplish the same best practices, see this post).
HP provided us with the best practices document for ESX4 connected to an HP EVA array. There is a major change in ESX4. For the first time, ESX is ALUA (Asymmetric Logical Unit Access) aware. (See this post on Yellow Bricks for more detail about ALUA.) ALUA allows the array and ESX to determine the optimal path — the path to the managing controller’s ports in the EVA’s case — and use those optimal paths until one isn’t available. This is important because it prevents flip-flopping on the ESX host.
In previous version of ESX, the desired storage setting was fixed path for the EVA. In our case, we simultaneously presented the ESX3.5 and ESX4 hosts to the same LUNs, meaning some were fixed and some were set to the ESX4 default, which was MRU. This caused problems. After initial issues, we backed away and presented one LUN at a time, performed our VMotions and then unpresented the LUN from the old cluster. This prevented any flapping issues between controllers.
The best practices document describes two settings for best practice with an EVA that is ALUA aware (EVA 4000/6000/8000 and EVA 4400/6400/8400):
- Set the default path selection policy to round robin (VMW_PSP_RR)
- Set the IOPS Limit to 1
The path selection policy is the bigger deal. The default path selection policy in ESX4 is most recently used (MRU). This is still a valid policy and works with the EVA, but it is not their best recommendation because Round Robin allows the ESX host to use two paths or four paths (depending on EVA model) to send traffic to the ESX host which can provide greater performance.
I was not sure what the IOPS setting actually does based on the documentation, but Duncan Epping has since posted about the IOPS=1 setting and about how in a real-world environment this setting may not make a difference. I agree with his conclusions.
Now, I should not and do not want to take credit for the instructions below… I did not come up with this solution — simply a user. The esxcli commands came directly from the HP Best Practices Guide, but the scripted IOPS setting changes for the LUNs came from another blog – Virtual IEF. This solution has worked brilliantly in our environment. Virutal IEF has a great, in-depth posting about how the EVA LUN ownership works and what makes an optimal path.
Setting Changes for Round Robin PSP
To change the default path selection policy (PSP) to Round Robin. If LUNs are already presented, you will need to reboot to allow the LUNs to rescan and set to Round Robin instead of the default most recently used (MRU) setting.
esxcli nmp satp setdefaultpsp --satp VMW_SATP_ALUA --psp VMW_PSP_RR
Change the IOPS limit value for the interactive session.
for i in `ls /vmfs/devices/disks/ | grep naa.600`;
do esxcli nmp roundrobin setconfig --type "iops" --iops=1 --device $i ;done
Verify the IOPS value is correct, run this script and check values.
for i in `ls /vmfs/devices/disks/ | grep naa.600`;
do esxcli nmp roundrobin getconfig --device $i ;done
Your output should look like:
Device: naa.600508b4001087e400009000012b0000 I/O Operation Limit: 1 Limit Type: Iops Use Active Unoptimized Paths: false Errors: Unable to find device with the name naa.600508b4001087e400009000012b0000:1 Byte Limit: 10485760
Unfortunately, the IOPS value will reset on reboot many times. To ensure that this value is correctly set, you are able to set a value in the post boot script. Add the same for statement to/etc/rc.local to set the IOPS value on each reboot.
for i in `ls /vmfs/devices/disks/ | grep naa.600`;
do esxcli nmp roundrobin setconfig --type "iops" --iops=1 --device $i ;done
