The VMguy’s Notable KB articles for the week

I want to thank The VMguy for his weekly Notable KB articles for the week.

Here they are again, me just copying and placing it on the blog.

”

Sorry for the late post. The VMware support team never sleeps. Here’s a few articles new or updated this past week:

”

Full Post http://vmguy.com/wordpress/

The Investigating disk space on an ESX host caught my attention, and I will expand on what it says below.

“

Symptoms

Virtual machine fails to power on with following error message:
Could not power on VM: No space left on device. Failed to power on VM

The management agent (hostd) cannot start because the root partition is full
The VirtualCenter agent (vpxa) cannot start because the root partition is full
The system cannot create any new files or directories on / or /tmp partition, with the error:
no space left on device

The /tmp directory is filled with cimclient_root* logs, potentially thousands (~600,000 or more)
The vpxa log may contain these error:
[fusion_builder_container hundred_percent=”yes” overflow=”visible”][fusion_builder_row][fusion_builder_column type=”1_1″ background_position=”left top” background_color=”” border_size=”” border_color=”” border_style=”solid” spacing=”yes” background_image=”” background_repeat=”no-repeat” padding=”” margin_top=”0px” margin_bottom=”0px” class=”” id=”” animation_type=”” animation_speed=”0.3″ animation_direction=”left” hide_on_mobile=”no” center_content=”no” min_height=”none”][2008-10-13 11:02:05.423 ‘Libs’ 3076454304 warning] Cannot make directory /var/run/vmware/root/27591: No space left on device
[2008-10-13 11:02:05.423 ‘App’ 3076454304 error] Exception: Failed to initialize authd server
[2008-10-13 11:02:05.423 ‘App’ 3076454304 error] Backtrace:
[00] eip 0x909dd92
[01] eip 0x9043444
[02] eip 0x907f975
[03] eip 0x908024c
[04] eip 0x9033d74
VMware VMotion failure at 10%

Purpose

For troubleshooting purposes, it may be necessary to check the available free disk space on your ESX host. This provides you with the steps to check the available disk space and steps to free up space if required.

Resolution

Checking disk space usage on the ESX Server service console partitions

To check the free space on an ESX Server service console partitions:

Log in to the ESX Server service console as root from either an SSH session or directly from the console of the server.
Type df -h.

When you have finished reviewing the output, type logout and press Enter to exit the system.

The output appears similar to:

Filesystem Size Used Avail Use% Mounted on
/dev/sda2 4.9G 3.0G 1.6G 66% /
/dev/sda1 99M 18M 77M 19% /boot
none 145M 0 145M 0% /dev/shm
/dev/sda7 2.0G 135M 1.7G 8% /var/log
[root@server]#

Note: The partitions shown are from a default installation of an ESX host. If you have modified the partition configuration the output may show differently.

Review the Use% for each of the listed items. If any of the volumes listed are 100% full they must be investigated to determine if space can be freed. The most important mount points to investigate on a default installation of ESX are the / and /var/log mounts because if they are full they can prevent proper operation of the ESX host.

Checking disk space usage on a VMFS volume of an ESX Server

To check the Free space on a VMFS volume of an ESX Server:

Log in to the ESX Server service console as root from either an SSH session or directly from the console of the server.

Type vdf -h.
When you have finished reviewing the output, type logout and press Enter to exit the system.

The output appears similar to:

Filesystem Size Used Avail Use% Mounted on
/dev/sda2 4.9G 3.0G 1.6G 66% /
/dev/sda1 99M 18M 77M 19% /boot
none 145M 0 145M 0% /dev/shm
/dev/sda7 2.0G 135M 1.7G 8% /var/log
/vmfs/devices 439G 0 439G 0% /vmfs/devices
/vmfs/volumes/458865ba-b31110fd-43d5-00127994e616
68G 47G 20G 69% /vmfs/volumes/San_Storage
/vmfs/volumes/45b5eb1a-808343db-ecab-00114335854b
26G 9.7G 16G 36% /vmfs/volumes/Local_Storage
[root@server]#

Note: The partitions shown are dependant on the VMFS volumes you have defined and presented to the ESX host.

Review the Use% for each of the listed items. If any of the volumes listed are 100% full they must be investigated to determine if space can be freed. If a VMFS volume is full you cannot create any new virtual machines and any virtual machines that are using snapshots may fail.

Freeing disk space on an ESX Server

Freeing disk space can become a large task if there has not been any recent changes to the ESX host. There are a few common reasons that partitions become full:

Information, such as a virtual machine or an ISO, was copied to the system.
Snapshots are being used on virtual machines.
Log files have not yet been rotated after a substantial amount of information was written to them.

There are also several common places to start for determining what is taking up the space on the volume:

The /vmimages folder. This folder is used to store operating system install files such as the VMware Tools or other ISO files.

The /var/core folder. This folder is used to store crash files for processes on the service console.
The /var/log folder. This folder stores the majority of the logs for the ESX host.
Any VMFS volume. These are used to store the virtual machine data.

If you are unable to determine where the space is being used, you can run the following command:

[root@server]# find / -size +10240000c -exec du -h {} ; | less
This command provides a list of all files that are larger than 10MB in size. See the Additional Information section of this article for more options on tweaking this command.

After you have determined what is taking up the free space you can use the rm command to free up space by deleting files.

Warning: When you delete a file there is no way to recover it, so use caution when deleting anything.

The following are a list of files that are safe to delete:

Old vm-support logs.
Virtual machines that are not being used and are not needed

ISO files that were copied to the system
Note: Do not delete the VMware Tools ISO files.
Old log files that are no longer needed.
Virtual Machine log files can be removed if there are a lot (thousands) of vmware*.log files in the virtual machine’s folder. You may need to delete the vmware*.log files using “rm vmware*.log” or if there are too many arguments for this to complete successfully you can try using “rm vmware-1[1-5]*.log” followed by “rm vmware-1*.log”, “rm vmware-2[1-5]*.log” followed by “rm vmware-2*.log”, and then “rm vmware*.log”. This will delete all the log files for this virtual machine.

If you are unsure about deleting a specific file please contact VMware Support for assistance with this. If a system file is removed inadvertently this may cause damage to your ESX host that can require a re-installation of the software.

Additional Information

Note: Recommended partition sizes can be found in the “Datastore Partitioning” chapter in the VI3 Install & Upgrade guide which can be found at http://www.vmware.com/pdf/vi3_301_201_installation_guide.pdf

Note: Here are additional ways to use the find command:

To look for files larger than 1MB, change the command so that-size is: +1024000c

The find command listed above looks through the entire file system tree. The-mountswitch will stop find from crossing mount points. This is useful as you don’t want to look in the/vmfs part of the tree, since all the files there are big and not what you are looking for.

“

I know to some of you this is wasted space, but I wanted to remind both my self and others on how important disk space is when it comes to your VMware Host.

If time permits, I would like to dig into ways to prevent this from happening.

Thanks to VMware and the Author of the source blog post.

[/fusion_builder_column][/fusion_builder_row][/fusion_builder_container]

Storage

The VMguy’s Notable KB articles for the week

Checking disk space usage on the ESX Server service console partitions

Checking disk space usage on a VMFS volume of an ESX Server

Freeing disk space on an ESX Server

Twitter via Power Shell

Denied after applying to Start a local VMUG Group.

You may also like