Storage

http://vmguy.com/wordpress/index.php posts his weekly Notable KB articles from the week.

“

Apparently the VMware support team got back to work full steam ahead this week.  81 articles were new or updated this week.  There are some really good troubleshooting ones this week.  Here’s this week’s selection:

• Converting a virtual disk into a Raw Device Mapping
• GHO image files cannot be converted
• Performing V2P (virtual to physical) migrations
• Determining if a virtual machine is orphaned
• Moving VirtualCenter to a different datastore
• Networking Error, IP Address Already Assigned to Another Adapter
• Verifying if software is causing your virtual machine to fail
• Sample configuration of virtual switch VLAN tagging (VST Mode) and ESX Server
• Identifying shared storage issues with ESX 3.x
• Investigating the health of a VirtualCenter database server
• Troubleshooting a virtual machine that has become unresponsive because of an ESX host
• ESX Server host or virtual machines have intermittent or no network connectivity
• iSCSI and Jumbo Frames configuration on ESX 3.5

“

Thanks to the VMguy, and VMware for Content.

A few of the articles that caught my eye.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=3443266

“

Converting a virtual disk into a Raw Device Mapping

KB Article
3443266

Updated
Jan. 12, 2009

Products

VMware ESX

Details

What is the command to convert a virtual disk into a Raw Device Mapping in ESX environment?

Solution

A virtual disk can be imported into a raw LUN for use with an RDM using following commands:

• For virtual compatibility mode use: vmkfstools –i <srcfile> -d rdm:/vmfs/devices/disks/vmhbac:t:l:p /vmfs/volumes/datastore/vmdir/vmname.vmdk

• For physical compatibility mode use: vmkfstools –i <srcfile> -d rdmp:/vmfs/devices/disks/vmhbac:t:l:p /vmfs/volumes/datastore/vmdir/vmname.vmdk

“

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003228

“

GHO image files cannot be converted

KB Article
1003228

Updated
Jan. 09, 2009

Products

VMware Converter

Details

• Cannot select Symantec Ghost/Norton Ghost image files as source for a virtual to virtual conversion
• .GHO files cannot be selected as a source

Solution

Image files created as .GHO format are not supported by Converter. You must use an image format with the file extension .sv2i.

The following Symantec formats are supported by Converter 3.x:

• Norton Ghost version 9.0, 10.0, and 12.0 (.sv2i files only)

• Symantec Backup Exec System Recovery (formerly LiveState Recovery) 6.5 and 7.0

• LiveState Recovery 3.0 and 6.0

A detailed list of supported formats for VMware Converter 3.x is included in the Converter 3.0.3 User’s Manual.

“

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007654

“

SCSI and Jumbo Frames configuration on ESX 3.5

KB Article
1007654

Updated
Jan. 05, 2009

Products

VMware ESX

Product Versions

VMware ESX 3.5.x

Symptoms

• Large queues are seen while monitoring Jumbo Frames

Purpose

This article provides information related to setting up an iSCSI environment with Jumbo Frames.

Resolution

Important Jumbo Frame Information

The following is important information to know when using Jumbo Frames:

• Any packet larger than 1500 MTU is a Jumbo Frame. ESX 3.5 supports frames up to 9kB (9000 Bytes). They are limited to data networking only (virtual machines and the VMotion network) on ESX 3i/3.5.

• Jumbo Frames are not supported for NAS and iSCSI traffic at this time.

• Jumbo Frames must be enabled for each vSwitch or VMkernel interface through the command‐line interface on your ESX Server 3 host.

• To allow ESX Server 3 to send larger frames out onto the physical network, the network must support Jumbo Frames end to end for Jumbo Frames to be effective.

• Before enabling jumbo frames, ensure the NIC or LOM supports jumbo frames.

• VMware supports jumbo frames with the following vendors:

  • Intel (82546, 82571),

  • Broadcom (5708, 5706, 5709),

  • Netxen (NXB-10GXxR,

  • NXB-10GCX4),

  • Neterion (Xframe, Xframe II, Xframe E).

Creating a Jumbo Frames-enabled vSwitch

1. Log in directly to your ESX Server 3 hostʹs console.

2. Run the esxcfg-vswitch -m <MTU> <vSwitch> command to set the MTU size for the vSwitch.
   This command sets the MTU for all uplinks on that vSwitch. Set the MTU size to the largest MTU size among all the virtual network adapters connected to the vSwitch.

3. Run the esxcfg-vswitch -l command to display a list of vSwitches on the host, and check that the configuration of the vSwitch is correct.

Creating a Jumbo Frames-enabled VMkernel interface

To create a Jumbo Frames-enabled VMkernel interface:

1. Log in directly to your ESX Server 3 host’s console.

2. Run the esxcfg-vmknic -a -i <ip address> -n <netmask> -m <MTU> <portgroup name> command to create a VMkernel connection with Jumbo Frame
   support.

3. Run the esxcfg-vmknic -l command to display a list of VMkernel interfaces, and check that the configuration of the Jumbo Frame‐enabled interface is correct.
   Note: ESX Server 3 supports a maximum MTU size of 9000.

“

I was reading http://blog.fosketts.net/ ‘s article on Iomega and how EMC has thrown them a life line. 

I saw them talk about Iomega’s ix2, and how it was HCL certified. ( http://www.vmware.com/resources/compatibility/search.php?action=search&deviceCategory=san&productId=1&keyBasic=iomega&maxDisplayRows=50&key=&release[]=-1&datePosted=-1 )

I thought this was interesting, as it indicates that you could get a very low cost VMware supported NAS device for under $500.

“

Partner Name Model Array Type Supported Releases

EMC      Iomega ix2      NAS            ESX 3.5 U3, ESX 3.5 U2, ESX 3.5 U1, ESX 3.5

From VMware’s HCL.

”

Here is EMC’s Press Release on the device.

I Would love to hear if anyone has tried this device on ESX or ESXi, and what type of results they have gotten.

[/fusion_builder_column][/fusion_builder_row][/fusion_builder_container]

I want to thank The VMguy for his weekly Notable KB articles for the week.

Here they are again, me just copying and placing it on the blog.

”

Sorry for the late post. The VMware support team never sleeps. Here’s a few articles new or updated this past week:

• Using Application Sync to rollback problematic updates <good to know for ThinApp
• Moving VirtualCenter to a different datastore
• Installing and Configuring NTP on VMware ESX Server
• Ensuring a virtual machine is not inaccessible due to a VMware VirtualCenter issue
• Troubleshooting a virtual machine that has stopped responding
• Troubleshooting unresponsive guest operating system issues
• Investigating disk space on an ESX host
• Troubleshooting connectivity issues between the agent, client, and connection server in VMware Virtual Desktop Manager
• Troubleshooting a “Desktop not available” message
• ESXi 3 Hosts without swap enabled cannot be added to a VMware High Availability Cluster

”

Full Post http://vmguy.com/wordpress/

The Investigating disk space on an ESX host caught my attention, and I will expand on what it says below.

“

Symptoms

• Virtual machine fails to power on with following error message:
  Could not power on VM: No space left on device. Failed to power on VM

• The management agent (hostd) cannot start because the root partition is full

• The VirtualCenter agent (vpxa) cannot start because the root partition is full

• The system cannot create any new files or directories on / or /tmp partition, with the error:
  no space left on device

• The /tmp directory is filled with cimclient_root* logs, potentially thousands (~600,000 or more)

• The vpxa log may contain these error:
  [fusion_builder_container hundred_percent=”yes” overflow=”visible”][fusion_builder_row][fusion_builder_column type=”1_1″ background_position=”left top” background_color=”” border_size=”” border_color=”” border_style=”solid” spacing=”yes” background_image=”” background_repeat=”no-repeat” padding=”” margin_top=”0px” margin_bottom=”0px” class=”” id=”” animation_type=”” animation_speed=”0.3″ animation_direction=”left” hide_on_mobile=”no” center_content=”no” min_height=”none”][2008-10-13 11:02:05.423 ‘Libs’ 3076454304 warning] Cannot make directory /var/run/vmware/root/27591: No space left on device
  [2008-10-13 11:02:05.423 ‘App’ 3076454304 error] Exception: Failed to initialize authd server
  [2008-10-13 11:02:05.423 ‘App’ 3076454304 error] Backtrace:
  [00] eip 0x909dd92
  [01] eip 0x9043444
  [02] eip 0x907f975
  [03] eip 0x908024c
  [04] eip 0x9033d74

• VMware VMotion failure at 10%

Purpose

For troubleshooting purposes, it may be necessary to check the available free disk space on your ESX host. This provides you with the steps to check the available disk space and steps to free up space if required.

Resolution

Checking disk space usage on the ESX Server service console partitions

To check the free space on an ESX Server service console partitions:

1. Log in to the ESX Server service console as root from either an SSH session or directly from the console of the server.

2. Type df -h.

3. When you have finished reviewing the output, type logout and press Enter to exit the system.

The output appears similar to:

Filesystem Size Used Avail Use% Mounted on
/dev/sda2 4.9G 3.0G 1.6G 66% /
/dev/sda1 99M 18M 77M 19% /boot
none 145M 0 145M 0% /dev/shm
/dev/sda7 2.0G 135M 1.7G 8% /var/log
[root@server]#

Note: The partitions shown are from a default installation of an ESX host. If you have modified the partition configuration the output may show differently.

Review the Use% for each of the listed items. If any of the volumes listed are 100% full they must be investigated to determine if space can be freed. The most important mount points to investigate on a default installation of ESX are the / and /var/log mounts because if they are full they can prevent proper operation of the ESX host.

Checking disk space usage on a VMFS volume of an ESX Server

To check the Free space on a VMFS volume of an ESX Server:

1. Log in to the ESX Server service console as root from either an SSH session or directly from the console of the server.

2. Type vdf -h.

3. When you have finished reviewing the output, type logout and press Enter to exit the system.

The output appears similar to:

Filesystem Size Used Avail Use% Mounted on
/dev/sda2 4.9G 3.0G 1.6G 66% /
/dev/sda1 99M 18M 77M 19% /boot
none 145M 0 145M 0% /dev/shm
/dev/sda7 2.0G 135M 1.7G 8% /var/log
/vmfs/devices 439G 0 439G 0% /vmfs/devices
/vmfs/volumes/458865ba-b31110fd-43d5-00127994e616
68G 47G 20G 69% /vmfs/volumes/San_Storage
/vmfs/volumes/45b5eb1a-808343db-ecab-00114335854b
26G 9.7G 16G 36% /vmfs/volumes/Local_Storage
[root@server]#

Note: The partitions shown are dependant on the VMFS volumes you have defined and presented to the ESX host.

Review the Use% for each of the listed items. If any of the volumes listed are 100% full they must be investigated to determine if space can be freed. If a VMFS volume is full you cannot create any new virtual machines and any virtual machines that are using snapshots may fail.

Freeing disk space on an ESX Server

Freeing disk space can become a large task if there has not been any recent changes to the ESX host. There are a few common reasons that partitions become full:

• Information, such as a virtual machine or an ISO, was copied to the system.

• Snapshots are being used on virtual machines.

• Log files have not yet been rotated after a substantial amount of information was written to them.

There are also several common places to start for determining what is taking up the space on the volume:

• The /vmimages folder. This folder is used to store operating system install files such as the VMware Tools or other ISO files.

• The /var/core folder. This folder is used to store crash files for processes on the service console.

• The /var/log folder. This folder stores the majority of the logs for the ESX host.

• Any VMFS volume. These are used to store the virtual machine data.

If you are unable to determine where the space is being used, you can run the following command:

[root@server]# find / -size +10240000c -exec du -h {} ; | less
This command provides a list of all files that are larger than 10MB in size. See the Additional Information section of this article for more options on tweaking this command.

After you have determined what is taking up the free space you can use the rm command to free up space by deleting files.

Warning: When you delete a file there is no way to recover it, so use caution when deleting anything.

The following are a list of files that are safe to delete:

• Old vm-support logs.

• Virtual machines that are not being used and are not needed

• ISO files that were copied to the system
  Note: Do not delete the VMware Tools ISO files.

• Old log files that are no longer needed.

• Virtual Machine log files can be removed if there are a lot (thousands) of vmware*.log files in the virtual machine’s folder. You may need to delete the vmware*.log files using “rm vmware*.log” or if there are too many arguments for this to complete successfully you can try using “rm vmware-1[1-5]*.log” followed by “rm vmware-1*.log”, “rm vmware-2[1-5]*.log” followed by “rm vmware-2*.log”, and then “rm vmware*.log”. This will delete all the log files for this virtual machine.

If you are unsure about deleting a specific file please contact VMware Support for assistance with this. If a system file is removed inadvertently this may cause damage to your ESX host that can require a re-installation of the software.

Additional Information

Note: Recommended partition sizes can be found in the “Datastore Partitioning” chapter in the VI3 Install & Upgrade guide which can be found at http://www.vmware.com/pdf/vi3_301_201_installation_guide.pdf

Note: Here are additional ways to use the find command:

• To look for files larger than 1MB, change the command so that-size is: +1024000c

• The find command listed above looks through the entire file system tree. The-mountswitch will stop find from crossing mount points. This is useful as you don’t want to look in the/vmfs part of the tree, since all the files there are big and not what you are looking for.

“

I know to some of you this is wasted space, but I wanted to remind both my self and others on how important disk space is when it comes to your VMware Host.

If time permits, I would like to dig into ways to prevent this from happening.

Thanks to VMware and the Author of the source blog post.

[/fusion_builder_column][/fusion_builder_row][/fusion_builder_container]

http://www.channelregister.co.uk has a article, with some metrial containing some information obtained from Bogomil Balkansky, senior director of product marketing at VMware.

Here are some quotes from the body of the article.

“
The VirtualCenter management tool for ESX Server hypervisors will become vCenter; software that rides atop the hypervisor will be referred to as vServices, and virtual file systems and other storage-related code will be called vStorage.
“

“
Balkansky says that in 2009, the hypervisor will double VirtualSMP capability to eight cores and with quadruple memory to 256 GB maximum per virtual machine.

Balkansky is expecting that virtualization-driven server consolidation will accelerate in 2009, and for an interesting reason: to boost performance on a single piece of iron. “A lot of applications running on servers today cannot take advantage of the extra cores chip makers are delivering,” Balkansky explains. “They end up wasting the extra horsepower.”

“
In the 2009 release, ESX Server will also get the capability to add virtual memory or CPUs to a VM instance without having to shut down and reboot the VM.
“

Next year’s hypervisor update from VMware will also include a less-virtual feature called VM Direct Path, which is an I/O passthrough that allows a virtual machine to be tired directly to a physical I/O device, such as a disk controller or network feature card.

VMware wants to do this for two reasons, according to Balkansky. For one thing, by going physical with the link to an I/O device, the performance gets closer to native speeds. Moreover, if you have a peripheral that is not supported by the virtual I/O created inside ESX Server, you can use VM Direct Path to support that peripheral. You don’t have to wait for a driver from VMware or the peripheral supplier to come along. The only issue with VM Direct Path is that supporting peripherals in this manner sacrifices their virtual mobility. So VMotion, which allows a VM to teleport around a network of servers using shared storage, and DRS, the distributed resource scheduling software in the VMware Infrastructure stack, won’t work on VMs that use the VM Direct Path feature.

Balkansky says that VMware is also cooking up its own fault tolerant VM capability, not code imported from partners Stratus Technologies or SteelEye Technology. While not divulging a lot of details, Balkansky says the code will keep two virtual machines running on two distinct physical servers in lock-step, with one VM active and the other operating in passive mode until a failure. This code was demonstrated at VMworld last summer, in fact.

On the storage front, the vNetwork Distributed Switch, a software switch that interfaces the network links to VMs that VMware created in conjunction with Cisco Systems, will be delivered as a virtual appliance running atop ESX Server (or whatever the future product is called). Cisco will also apparently sell a version of the product, too, called the Cisco Nexus 1000V.

The vStorage APIs are going to be opened up so storage administration tools, which think in terms of LUNs and arrays, can see where VMs and their affiliated VMDK files are located on the disk arrays. Right now, VirtualCenter can’t easily tell you where the VMDKs are physically located on the storage, which means storage admins can’t easily figure out what is running where before they tweak an array in some way.

VMware is also cooking up a variant of thin provisioning for VMDKs that will allow a virtual machine to overcommit its storage capacity, much as VMware has done for years with memory capacity. Such overcommitment may make some people jumpy, but allocating memory and disk to a VM and then only using a fraction of it is inefficient.

“

I thought this was a great read, and it seems that they have compiled a lot of information here, I can’t wait for 09.

I thought this was intresting Forum Discussion at the VMware Communities, Using multiple SAN solutions

“
Hi everyone,

I am reviewing our vmware environment and I have a question regarding multiple SAN implementations.
Currently we have diverse clusters and I am looking at combing these where possible.
In our environment we have two different SAN solution and this could be more in future.
We will be looking into storage virtualization but that is another story.
In the mean time I would like to know if it is ok to connect a host to more than one storage array?
Actually all hosts in cluster would probably have this configuration for obvious reasons.
Reasons for wanting to do this is obviously because we have them and also so I can mix workloads on the servers.
My fast or critical vmdks could be stored on the enterprise SAN while the entry level SAN can be used for less
critical VMs.
“

Check out the comments, if you don’t know the answer.

Full Post : http://communities.vmware.com/message/1130420#1130420

http://professionalvmware.com has a blog post titled : What RAID level do you use for your VMFS?

I saw this, ( and more at Professionalvmware.com ) and I didn’t get a chance to post it right away, and there was some good discussion on it as well.

“
Many of you will say “RAID 5, of course!” and for those of you that do I would like to ask why? Is it because you really understand the differences between the different RAID levels? Granted that you may but have you ever had to do a RAID rebuild of a RAID 5 volume? How long did it take? Was it actually successful?

What RAID level would I use for my VMFS? RAID 10, of course! 😉

Needless to say this post is not about asking questions, but instead give you insight into my years of experience with this very subject. RAID 5 was an acceptable choice when the cost of hard drives were so high as to be prohibitive. However, these days quite the opposite is true. The benefits for using RAID 10 (performance, recovery, etc) far outweigh the costs for the additional drives required.

Art S. Kagel has written an excellent article about the different RAID levels (including RAID 3 and 4) as well as why he also chooses RAID 10 over RAID 5:

http://www.miracleas.com/BAARF/RAID5_versus_RAID10.txt

If you are currently using RAID 5 for you critical uptime production VMFS then you may want to consider creating a new RAID group, carve out a LUN and Storage VMotion your VM’s.
“

Check out my comment, as well as others. I also have a simular question about raid levels on my forums. Here.

Full Post http://professionalvmware.com/2008/12/19/what-raid-level-do-you-use-for-your-vmfs

Leo’s Ramblings’s has a blog post on EMC ‘s best practices for Qlogic HBA ‘s and CX-series arrays titled : EMC storage arrays and QLogic HBAs

I have found that EMC documenation can be hard to find it when you need it, ( at least from a public side ) and therefor, I thought I’d post this.
“

Very few people know, but EMC put out a guide on best practice HBA settings for their CX-series of arrays.

Not only can adhering to their standards, improve peroformance, but it can also provide for a more stable and less problematic system.

“
Please Refer to the full post for the configuration recommendations.

Full Post : http://blog.core-it.com.au/?p=383

blog.scottlowe.org has a post that sum’s up the various network articles on the blog titled : VMware ESX Networking Articles

“
Over the past few years—almost four years now that I’ve been writing here on this site—I’ve written quite a few articles on VMware ESX networking. Here’s a collection of links to some of the more notable, and hopefully more useful, articles in that group.”

Here are the notable links, but please read the full blog post.
“
VMware ESX, NIC teaming, and VLAN trunking

VMware ESX, NIC teaming, and VLAN trunking with HP ProCurve

VMware ESX and the native VLAN

NIC utilization in VMware ESX, and then provided a follow up post

using the CLI to change the vSwitch load balancing policy

CLI guide to modify a port group

configuring VMware ESX, IP-based storage, and jumbo frames
“

What a great resource, thanks to the author.

Full Post : http://blog.scottlowe.org/2008/12/19/vmware-esx-networking-articles/