Category Archives: Uncategorized

Been Busy, New content soon.

We recently decided on a NetApp FAS 3140 at work, ( more on that later) so we have been busy implementing it into our environment.

Also recently, we purchased the VMware Foundation kit.

Once the FAS had the Fiber hooked up to the ESX server, and was setup ( more on this later ), I moved all of my existing VMware Guests from ESXi hosts to the ESX host.

Also I migrated my Onbase SQL, SQL temp, and Diskgroup local attached disk (SAS) to the San.

Then I configured our 3750G switch ( details later) for iSCSI, setting up three dual port nic teams, and enabling jumbo frames, and then configuring the ESX server (details later)

Then I have been working on configuring Cacti to monitor the FAS 3140, which only has SNMP V1, ( Ontap 7.2.X ), and playing with templates. ( details later )

So, as you can see I have some content I need to write up formally and publish, since I have not posted lately, I wanted to tell you all that I am alive, but just busy.

Roger L

One, Two, or Four vCPU’s in VMware?

 

 

Someone in Twitter today was asking about performance and how it relates to vCPU’s.

 

Personally I have found much better performance, and less CPU usage with 1 vCPU vs 2 vCPU’s and have never tried 4 vCPU’s

 

Let’s see if we can find some other information on the subject.

 

Co-scheduling SMP VMs in VMware ESX Server talks about the following

VMware ESX Server efficiently manages a mix of uniprocessor and multiprocessor VMs, providing a rich set of controls for specifing both absolute and relative VM execution rates. For general information on cpu scheduling controls and other resource management topics, please see the official VMware Resource Management Guide.
For a multiprocessor VM (also known as an "SMP VM"), it is important to present the guest OS and applications executing within the VM with the illusion that they are running on a dedicated physical multiprocessor. ESX Server faithfully implements this illusion by supporting near-synchronous coscheduling of the virtual CPUs within a single multiprocessor VM.
The term "coscheduling" refers to a technique used in concurrent systems for scheduling related processes to run on different processors at the same time. This approach, alternately referred to as "gang scheduling", had historically been applied to running high-performance parallel applications, such as scientific computations. VMware ESX Server pioneered a form of coscheduling that is optimized for running SMP VMs efficiently.

 

But the only document I can find that really addresses this is a older document titled:

 Best Practices Using VMware Virtual SMP

Virtual SMP can provide a significant advantage for multi-threaded applications and applications
using multiple processes in execution. However, since not all applications are able to take
advantage of a second or multiple CPUs, SMP virtual machines should not be provisioned by
default. For each virtual machine, system planners should carefully analyze the trade-off
between possible increases in performance and throughput gained through individual virtual
machines and the number of virtual machines supported by physical hardware.
The best practice for deploying Virtual SMP depends on a number of performance and
configuration factors, some of which are similar to physical SMP. You should analyze your
specific applications to understand if deploying them in SMP virtual machines will improve
throughput. In addition, you can achieve more efficient use of the underlying hardware by
deploying a mixture of single-CPU and SMP virtual machines to ESX Server platform systems.
Some key points regarding Virtual SMP deployment to take away from this white paper are:
• Virtual SMP allows a virtual machine to access two or more CPUs.
• Up to two physical processors can be consumed when you run a two-way virtual machine.
• Only allocate two or more virtual CPUs to a virtual machine if the operating system and the
application can truly take advantage of all the virtual CPUs. Otherwise, physical processor
resources may be consumed with no application performance benefit and, as a result,
other virtual machines on the same physical machine will be penalized.

I bolded and Underlined the key points I found in the conclusion of the document.

 

What do other blogger’s say?

The Lone Sysadmin has a posted titled : Why My Two vCPU VM is Slow

Sometimes computers are counterintuitive. One great case continues to be why a virtual machine with two vCPUs runs more slowly than a virtual machine with one vCPU.

Think of virtualization like a movie. A movie is a series of individual frames, but played back the motion looks continuous. It’s the same way with virtual machines. A physical CPU can only run one thing at a time, which means that only one virtual machine can run at a time. So the hypervisor “shares” a CPU by cutting up the CPU time into chunks. Each virtual machine gets a certain chunk to do its thing, and if it gets chunks of CPU often enough it’s like the movie: it seems like the virtual machine has been running continuously, even when it hasn’t. Modern CPUs are fast enough that they can pull this illusion off.

When one virtual machine stops running another virtual machine has an opportunity to run. If you have a virtual machine with one vCPU it needs a chunk of time from a single physical CPU. When a physical CPU has some free time that single vCPU virtual machine will run. No problem.

Similarly, in order for a virtual machine with two vCPUs to run it needs to have chunks of free time on two physical CPUs. When two physical CPUs are both available that virtual machine can run.

The trouble comes when folks mix and match single and dual-vCPU virtual machines in an environment that doesn’t have a lot of CPU resources available. A two-vCPU virtual machine has to wait for two physical processors to free up, but the hypervisor doesn’t like to have idle CPUs, so it runs a single vCPU virtual machine instead. It ends up being a long time before two physical CPUs free up simultaneously, and the two vCPU virtual machine seems really slow as a result. By “seems really slow” I mean it doesn’t perform very well, but none of the performance graphs show any problems at all.

To fix this you need to set the environment up so that two physical CPUs become free more often. First, you could add CPU resources so that the probability of two CPUs being idle at the same time is higher. Unfortunately this usually means buying stuff, which isn’t quick, easy, or even possible sometimes.

Second, you could set all your virtual machines to have one vCPU. That way they’ll run whenever a single physical CPU is free. This is usually a good stopgap until you can add CPU resources.

Last, you can group all your two vCPU machines together where those pesky single vCPU virtual machines won’t bother them. When a two vCPU virtual machine stops running it’ll always free up two physical CPUs. This usually means cutting up a cluster, though, so that will have also have drawbacks.

Virtualization can be awesome, but it can be pretty counterintuitive sometimes, too.

Very Interesting, take what you can from it. To my it says that you should keep like VM’s 1 vCPU’s and 2 vCPU’s on different hosts.

 

I fond this on the VMware Communities site.

 

Multiple vCPU’s (SMP); convincing the business

As most have come to understand, it is not a good practice to designate multiple vCPU’s for a VM unless the application running can actually use the 2nd processor effectively (see discussionhttp://communities.vmware.com/thread/131269). I am fairly new to the VM environment at my new job and noticed ~20% of the VM’s have either 2 or 4 (i assume very bad) vCPU’s assigned. We steadily get requests for VM’s with multiple CPU’s and it appears they are coming from requirements that assume physical servers are being used (ex:Two (2) VM Servers; Each with 2 Dual-Core/8GB RAM 108GB Usable disk space). The VMware support team understands the implications of using vSMP when not needed, but how do we convince the business that their application will work on a single CPU when their requirements call for 2 dual-core; 4 processor cores. Any documentation available that we can hand out or any suggestions is greatly appreciated. Thanks.

Dave

Responses that caught my eye are.

 

beagle_jbl

It’s not bad using vSMP – one just has to do it intelligently and sparingly. It’s only really bad if you have the same number of cores on the VM as the ESX host. Also, if you assign too many multi-core VMs you could run into really high CPU Ready times and therefore poor performance. Simply put, the ESX host needs to have 4 IDLE cores to satisfy a 4-core VM. On a fairly busy 8-core ESX host, 4 available cores may be hard to come by. In that instance, that 4-core VM could potentially run slower than a single-core VM (depending on the CPU shares assigned and such).
VMWare recommends having at least double the amount of cores on the ESX host as compared to any multicore VMs. I think that works OK with the odd 2-core on a 4-core ESX. However, I’ve insisted on using 16-core ESX hosts for any 4-core VMs. In those two scenarios… with a little monitoring and tweaking… I have seen no issues with CPU Ready counters.

 

I recommend reading the post, but the point is you need understand is that the Physical CPU to use 2 or 4 vCPU’s without a performance hit.

And has Texiwill points out in the Thread, most of the time you will see no increase by going to more vCPU’s, but you will see a performance hit.

 

Summary

To my understanding, the only time you would want to do multiple vCPU’s is when you have lots of Physical CPU’s / Cores, and you are running a Multiple CPU threaded Application, and then don’t mix vCPU’s on the same Host / Cluster.

 

 

 

Thoughts? Am I understanding this correctly? Post comments please

Idle Memory Tax

 

Jason Boche has taken the time to outline Idle Memory Tax in VMware ESX.

Jason, I wanted to thank you for taking the time on this, and I am hoping you do not mind me quoting your Article.

Memory over commit is a money/infrastructure saving feature that fits perfectly within the theme of two of virtualization’s core concepts: doing more with less hardware, and helping save the environment with greenness. While Microsoft Hyper-V offers no memory over commit or page sharing technologies, VMware has understood the value in these technologies long before VI3. I’ve mentioned this before – if you haven’t read it yet, take a look at Carl Waldspurger’s 2002 white paper on Memory Resource management in VMware ESX Server.

One of VMware’s memory over commit technologies is called Idle Memory Tax. IMT basically allows the VMKernel to reclaim unused guest VM memory by assigning a higher “cost value” to unused allocated shares. The last piece of that sentence is key – did you catch it? This mechanism is tied to shares. When do shares come into play? When there is contention for physical host RAM allocated to the VMs. Or in short, when physical RAM on the ESX host has been over committed – we’ve granted more RAM to guest VMs than we actually have on the ESX host to cover at one time. When this happens, there is contention or a battle for who actually gets the physical RAM. Share values are what determine this. I don’t want to get too far off track here as this discussion is specifically on Idle Memory Tax, but shares are the foundation so they are important to understand.

Back to Idle Memory Tax. Quite simply it’s a mechanism to take idle/unused memory from guest VMs that are hogging it in order to give that memory to another VM where it’s more badly needed. Sort of like Robin Hood for VI. By default this is performed using VMware’s balloon driver which is the more optimal of the two available methods. Out of the box, the amount of idle memory that will be reclaimed is 75% as configured by Mem.IdleTax under advanced host configuration. The VMKernel polls for idle memory in guest VMs every 60 seconds. This interval was doubled from ESX2.x where the polling period was every 30 seconds.

Here’s a working example of the scenario:

  • Two guest VMs live on an ESX/ESXi host with 8GB RAM
  • Each VM is assigned 8GB RAM and 8,192 shares. Discounting memory overhead, content based page sharing, and COS memory usage, we’ve effectively over committed our memory by 100%
  • VM1 is running IIS using only 1GB RAM
  • VM2 is running SQL and is request the use of all 8GB RAM
  • Idle Memory Tax allows the VMKernel to “borrow” 75% of the 7GB of allocated but unused RAM from VM1 and give it to VM2.  25% of the unused allocated RAM will be left for the VM as a cushion for requests for additional memory before other memory over commit technologies kick in

Here are the values under ESX host advanced configuration that we can tweak to modify the default behavior of Idle Memory Tax:

  • Mem.IdleTax – default: 75, range: 0 to 99, specifies the percent of idle memory that may be reclaimed by the tax
  • Mem.SamplePeriod – default: 60 in ESX3.x 30 in ESX2.x, range: 0 to 180, specifies the polling interval in seconds at which the VMKernel will scan for idle memory
  • Mem.IdleTaxType – default: 1 (variable), range: 0 (flat – use paging mechanism) to 1 (variable – use the balloon driver), specifies the method at which the VMKernel will reclaim idle memory. It is highly recommended to leave this at 1 to use the balloon driver as paging is more detrimental to the performance of the VM

VMware recommends that changes to Idle Memory Tax are not necessary, or even appropriate. If you get into the situation where Idle Memory Tax comes into play, you need to question the VMs that have large quantities of allocated but idle memory. Rather than allocating more memory to the VM than it needs, thus wrongly inflating its share value, consider reducing the allocated amounts of RAM to those VMs.

I thought this was very interesting, I my self have never played with this setting.

 

I did have some questions on performance on both the VMware Host side, and VMware Guest side. Do either have a performance hit when ESX is using Idle Memory Tax to move that memory.

I could see large uses for this, if you have a system that only uses large amounts of memory at certain times of the day.

In fact, would you want to design your systems to use this ( by forcing low ram ), if you have certain systems that run at night, when other systems are idle ( with more ram) that normally are only used during the day?

 

Source Post http://www.boche.net/blog/index.php/2009/01/29/idle-memory-tax/

Xiotech Market information and Points of Interest.

A local vendor sent me these links, and I thought I would share.

 

http://www.xiotech.com/About_Press_Release.aspx?ID=NR-09-17-08 Virtual View, recognized in Storage Software for Virtualization category, dramatically simplifies storage monitoring, management and provisioning tasks in VMWare virtual environments
http://www.drunkendata.com/?p=2100  “Xiotech gets my nod for best technology initiative of 2008”
http://www.byteandswitch.com/document.asp?doc_id=169956&WT.svl=news2_1  Article featuring Dan Lewis of USC Marshall School of Business. Lewis declares that his Emprise 7000 system provides an 8x better performance than his old HP EVA3000. He also praises Virtual View, self-healing capabilities, and reduced energy consumption
http://searchstorage.techtarget.com.au/articles/28223-Five-hot-storage-technologies-for-2-9-and-five-flops- ISE among the top 5 technologies for 2009
http://www.drunkendata.com/?p=2085 Jon Toigo comments on the most important IT-related things that happened in 2008. Xiotech holds both the #1 and #2 spots – with ISE and Web Services based management.
http://www.crn.com/storage/212501502;jsessionid=4JXY4Y42KKETWQSNDLRSKH0CJUNN2JVN?pgno=9 The 10 Biggest Storage Stories Of 2008
http://www.xiotech.com/About_Press_Release.aspx?ID=NR-12-15-08  Xiotech Selected as 2009 Hot Companies Finalist
http://www.xiotech.com/About_Press.aspx#  More, more, more
http://www.networkworld.com/supp/2009/ndc1/012609-storage-rethink.html?page=1 Article on Web Services and ISE

 

Source Xiotech and local Vendor.

Exchange 2003 Outgoing SMTP delivery issues

Exchange 2003 Outgoing SMTP delivery issues

We have had an Exchange 2003 mail server for a few years and have had great results without many big configuration changes, until now. Recently we added a new Qwest T-1 and changed our firewall to a Watchgaurd. After the change we have had a number of problems delivering mail to certain hosts. These hosts require that your mail server send mail with certain mail header requirements.

I am trying to get input on what’s required on all levels for mail delivery.

For outgoing mail to pass header requirements what should be in place at your DNS provider Ex: A record, your ISP Ex: Reverse DNS, your firewall and internal Windows DNS server, and your Exchange server?

Currently our mail server is sending an EHLO that won’t resolve. The format is internalhostDNSserver.internaldoamin.xxx.
Here is an example display of full headers received portion when I receive mail in my Yahoo account.

Received:

from 209.188.99.98 (EHLO appsrv10.mahowaldins.net) (209.188.99.98) by mta152.mail.re2.yahoo.com with SMTP; Wed, 28 Jan 2009 19:14:08 -0800

What should this be? If the internal mail server is called “exch” can you use mail.doamin.com then rely in internal DNS to do the rest?

How should this also correlate with your ISP reverse DNS, should they matchup some how?

Solution:
The solution was provided by Web: http://www.linkedin.com/in/astorrs Twitter: twitter.com/astorrs
http://www.msexchange.org/tutorials/Sender-Policy-Framework.html
SPF or Sender Policy Framework will help your Exchange Server delivery mail smoothly.

Going Tapeless in Enterprise

Dave Graham @ http://flickerdown.com wrote a blog about going tapeless.

I was asked the other day about the potential for finally going tapeless in Commercial and Enterprise spaces. Truth be told, this is becoming a more common occurence as those mechanical beasts hit the tail-end of their maintenance windows. With that in mind, what are some (not all) of the business drivers that move enterprises from tape to disk (or other mediums)?

“Tape is Dead!” has echoed through the halls of storage IT for quite some time. It’s been an ideological pursuit of the “next generation backup” mantra hawked by the Tier Ones in the storage space. What has been noted is the continued price depreciation of disk as capacity has increased, the tighter compliance metrics from government agencies, as well as the legacy of natural disasters that have arrived on the scene since 2001. Let’s unpack these a bit.

He discusses the following topic’s.

Price Depreciations and Increased Capacity

Tighter Compliance

Natural Disasters

Remediating the Problem

Price Depreciation and Capacity Increase – Tapeless

Tighter Compliance – Tapeless

Natural Disasters – Tapeless

Please check out the full post

http://flickerdown.com/2009/01/going-tapeless-in-enterprise/

Converting a Linux Virtual Machine With an LVM

http://technodrone.blogspot.com has a write up on Converting a VM from A older version of Xen to VMware using the VMware Converter 4.0 titled : Converting a Linux Virtual Machine With an LVM

He went through quite a process. Make sure to check it out.

I was entrusted with the task to try and migrate a VM from Xen 3.2 to ESX. Well you would think that is a trivial task – it should be. But let us not forget that we are talking here about version 3.2, which is two generations back from the current Citrix Xensource which is in use today.
After successfully migrating it I would like to share with you the procedure.
This was done on a Xen Guest VM runnning Red Hat Enterprise Linux ES release 3 (Taroon Update 7)

 

 

Full Source Post.

 

http://technodrone.blogspot.com/2009/01/converting-linux-virtual-machine-with.html

VMware KnowledgeBase Articles released to do with Time

I saw these come across the VMware RSS feed today, and I have personally had time problems in Linux VM’s, so I wanted to post these links.

 

Time in a Linux virtual machine jumps backward when using clock=pit

KB Article
1006086

Updated
Jan. 27, 2009

Time runs slower than real time due to lost timer interrupts

KB Article
1006088

Updated
Jan. 27, 2009

Time in a Linux 2.6 guest operating system runs faster than real time due to lost tick overcompensation

KB Article
1006113

Updated
Jan. 27, 2009

Time in virtual machine drifts due to hardware timer drift

KB Article
1006072

Updated
Jan. 27, 2009

Time runs too fast in a Windows virtual machine when the Multimedia Timer interface is used

KB Article
1005953

Updated
Jan. 27, 2009

Time falls behind in a virtual machine when the memory of the virtual machine is paged from disk by the VMkernel

KB Article
1005861

Updated
Jan. 27, 2009

Time drifts in virtual machines and the service console due to the HPET misreporting its frequency

KB Article
1006090

Updated
Jan. 27, 2009

 

I hope these help those of you that are having time issues in VM’s are able to

 

Thanks to VMware.

Notable KB Articles from the week from TheVMguy and me.

http://vmguy.com/wordpress/index.php posted his weekly Notable KB Articles from the week.

27 Articles new or updated this past week.  The first one is very relevant.

 

None of these hit home for me, ( just didn’t apply) so I went looking for more, and found the following.

 

How to Store Snapshot Files on an Alternate LUN

Datastores no longer available during upgrade

 

I’ll quote them below.

 

How to Store Snapshot Files on an Alternate LUN

KB Article
1003023

Updated
Jan. 26, 2009

Products

VMware ESX

VMware VirtualCenter

Details

Snapshot files are stored in the directory where the <VMname>.vmx file exists.

If you have limited space on the LUN where the <VMname>.vmx file exists, it may be necessary to specify an alternate location for the delta files that are created when you take snapshots of a VM.

Solution

This can be accomplised by defining an alternative working directory in the <VMname>.vmx   for the VM. The side effect of defining an alternative working directory is that the swap file will also be created in the alternative location.

The step to do this are:

1) Power off the VM

2) Edit the <VMname>.vmx file

      * add a line that looks like the following **

workingDir = "/vmfs/volumes/47384c5e-41fd4f1c-69c3-00163581bcf/directory_for_vmname_snapshotfile/"

3) Remove the VM from inventory in VirtualCenter

4) Add the VM to inventory in VirtuaCenter

5) Power on VM

6) Take snapshots as desired

Note: It may not be necessary to remove from inventory and add back into inventory, but in repro this produced consistent results.

You cannot use the advanced edit setting feature in VirtualCenter to add this line to the <VMname>.vmx file and have this take effect.

Note: This appears to only work as expected when you edit the .vmx file directly.

And

Datastores no longer available during upgrade

KB Article
1008031

Updated
Jan. 25, 2009

Products

VMware Lab Manager

Product Versions

VMware Lab Manager 3.0.x

Symptoms

  • Cannot complete an upgrade from Lab Manager 2.x to 3.x.

  • Required VMFS datastores no longer available.

  • An error is received while upgrading Lab Manager from version 2.x to 3.x after specifying VirtualCenter credentials in the configuration wizard.

  • The wizard does not proceed forward until each unavailable datastore is deleted from the configuration.

Resolution

This issue occurs when the credentials you gave for VirtualCenter do not have sufficient permissions to access the datastores and other required objects.

To verify the permissions, connect to VirtualCenter using the same credentials specified in the Lab Manager configuration wizard. If you are not able to see the configured hosts, clusters, or datastores, you must reconfigure the account privileges to make these objects accessible, or use a different account.

For more information on configuring VirtualCenter permissions, see the Basic System Administration guide, specifically the Managing Users, Groups, Permissions, and Roles section.

Those two caught my eye.

 

Thanks to The VMguy and VMware.