Roger Lund

I joined the VMworld Underground Podcast this morning, and was allowed to talk. If you didn’t know, I love to talk. We did end up having a nice conversation about VMworld 2009 Europe. We discussed the news of the day, and since neither of us were able to attend. I would love to have more enlightenment from some of the attendees. Please comment away. I wanted to thank vesanclark for hosting the podcast, and letting me chatter away!

Here is what post that http://www.vmworldunderground.com did on the podcast, titled : VMworld Europe 2009 – Day 1 Update Podcast available for download

“

Thanks to all that participated on the podcast this morning! VMworld Underground Podcast
Had some lively discussions about this morning’s announcements at VMworld Europe in Cannes. Below is the link to the recorded podcast out on Talkshoe. Don’t forget that we’ll be doing the same thing tomorrow morning at 10:30 AM Central Standard Time (roughly 5:30 PM CET in Cannes).

”

Click pay to listen, or check it out on the right side of my blog.

Someone in Twitter today was asking about performance and how it relates to vCPU’s.

Personally I have found much better performance, and less CPU usage with 1 vCPU vs 2 vCPU’s and have never tried 4 vCPU’s

Let’s see if we can find some other information on the subject.

Co-scheduling SMP VMs in VMware ESX Server talks about the following

“

VMware ESX Server efficiently manages a mix of uniprocessor and multiprocessor VMs, providing a rich set of controls for specifing both absolute and relative VM execution rates. For general information on cpu scheduling controls and other resource management topics, please see the official VMware Resource Management Guide.
For a multiprocessor VM (also known as an "SMP VM"), it is important to present the guest OS and applications executing within the VM with the illusion that they are running on a dedicated physical multiprocessor. ESX Server faithfully implements this illusion by supporting near-synchronous coscheduling of the virtual CPUs within a single multiprocessor VM.
The term "coscheduling" refers to a technique used in concurrent systems for scheduling related processes to run on different processors at the same time. This approach, alternately referred to as "gang scheduling", had historically been applied to running high-performance parallel applications, such as scientific computations. VMware ESX Server pioneered a form of coscheduling that is optimized for running SMP VMs efficiently.

“

But the only document I can find that really addresses this is a older document titled:

 Best Practices Using VMware Virtual SMP

“

Virtual SMP can provide a significant advantage for multi-threaded applications and applications
using multiple processes in execution. However, since not all applications are able to take
advantage of a second or multiple CPUs, SMP virtual machines should not be provisioned by
default. For each virtual machine, system planners should carefully analyze the trade-off
between possible increases in performance and throughput gained through individual virtual
machines and the number of virtual machines supported by physical hardware.
The best practice for deploying Virtual SMP depends on a number of performance and
configuration factors, some of which are similar to physical SMP. You should analyze your
specific applications to understand if deploying them in SMP virtual machines will improve
throughput. In addition, you can achieve more efficient use of the underlying hardware by
deploying a mixture of single-CPU and SMP virtual machines to ESX Server platform systems.
Some key points regarding Virtual SMP deployment to take away from this white paper are:
• Virtual SMP allows a virtual machine to access two or more CPUs.
• Up to two physical processors can be consumed when you run a two-way virtual machine.
• Only allocate two or more virtual CPUs to a virtual machine if the operating system and the
application can truly take advantage of all the virtual CPUs. Otherwise, physical processor
resources may be consumed with no application performance benefit and, as a result,
other virtual machines on the same physical machine will be penalized.

”

I bolded and Underlined the key points I found in the conclusion of the document.

What do other blogger’s say?

The Lone Sysadmin has a posted titled : Why My Two vCPU VM is Slow

“

Sometimes computers are counterintuitive. One great case continues to be why a virtual machine with two vCPUs runs more slowly than a virtual machine with one vCPU.

Think of virtualization like a movie. A movie is a series of individual frames, but played back the motion looks continuous. It’s the same way with virtual machines. A physical CPU can only run one thing at a time, which means that only one virtual machine can run at a time. So the hypervisor “shares” a CPU by cutting up the CPU time into chunks. Each virtual machine gets a certain chunk to do its thing, and if it gets chunks of CPU often enough it’s like the movie: it seems like the virtual machine has been running continuously, even when it hasn’t. Modern CPUs are fast enough that they can pull this illusion off.

When one virtual machine stops running another virtual machine has an opportunity to run. If you have a virtual machine with one vCPU it needs a chunk of time from a single physical CPU. When a physical CPU has some free time that single vCPU virtual machine will run. No problem.

Similarly, in order for a virtual machine with two vCPUs to run it needs to have chunks of free time on two physical CPUs. When two physical CPUs are both available that virtual machine can run.

The trouble comes when folks mix and match single and dual-vCPU virtual machines in an environment that doesn’t have a lot of CPU resources available. A two-vCPU virtual machine has to wait for two physical processors to free up, but the hypervisor doesn’t like to have idle CPUs, so it runs a single vCPU virtual machine instead. It ends up being a long time before two physical CPUs free up simultaneously, and the two vCPU virtual machine seems really slow as a result. By “seems really slow” I mean it doesn’t perform very well, but none of the performance graphs show any problems at all.

To fix this you need to set the environment up so that two physical CPUs become free more often. First, you could add CPU resources so that the probability of two CPUs being idle at the same time is higher. Unfortunately this usually means buying stuff, which isn’t quick, easy, or even possible sometimes.

Second, you could set all your virtual machines to have one vCPU. That way they’ll run whenever a single physical CPU is free. This is usually a good stopgap until you can add CPU resources.

Last, you can group all your two vCPU machines together where those pesky single vCPU virtual machines won’t bother them. When a two vCPU virtual machine stops running it’ll always free up two physical CPUs. This usually means cutting up a cluster, though, so that will have also have drawbacks.

Virtualization can be awesome, but it can be pretty counterintuitive sometimes, too.

”

Very Interesting, take what you can from it. To my it says that you should keep like VM’s 1 vCPU’s and 2 vCPU’s on different hosts.

I fond this on the VMware Communities site.

Multiple vCPU’s (SMP); convincing the business

“

As most have come to understand, it is not a good practice to designate multiple vCPU’s for a VM unless the application running can actually use the 2nd processor effectively (see discussionhttp://communities.vmware.com/thread/131269). I am fairly new to the VM environment at my new job and noticed ~20% of the VM’s have either 2 or 4 (i assume very bad) vCPU’s assigned. We steadily get requests for VM’s with multiple CPU’s and it appears they are coming from requirements that assume physical servers are being used (ex:Two (2) VM Servers; Each with 2 Dual-Core/8GB RAM 108GB Usable disk space). The VMware support team understands the implications of using vSMP when not needed, but how do we convince the business that their application will work on a single CPU when their requirements call for 2 dual-core; 4 processor cores. Any documentation available that we can hand out or any suggestions is greatly appreciated. Thanks.

Dave

“

Responses that caught my eye are.

“

beagle_jbl

It’s not bad using vSMP – one just has to do it intelligently and sparingly. It’s only really bad if you have the same number of cores on the VM as the ESX host. Also, if you assign too many multi-core VMs you could run into really high CPU Ready times and therefore poor performance. Simply put, the ESX host needs to have 4 IDLE cores to satisfy a 4-core VM. On a fairly busy 8-core ESX host, 4 available cores may be hard to come by. In that instance, that 4-core VM could potentially run slower than a single-core VM (depending on the CPU shares assigned and such).
VMWare recommends having at least double the amount of cores on the ESX host as compared to any multicore VMs. I think that works OK with the odd 2-core on a 4-core ESX. However, I’ve insisted on using 16-core ESX hosts for any 4-core VMs. In those two scenarios… with a little monitoring and tweaking… I have seen no issues with CPU Ready counters.

“

I recommend reading the post, but the point is you need understand is that the Physical CPU to use 2 or 4 vCPU’s without a performance hit.

And has Texiwill points out in the Thread, most of the time you will see no increase by going to more vCPU’s, but you will see a performance hit.

Summary

To my understanding, the only time you would want to do multiple vCPU’s is when you have lots of Physical CPU’s / Cores, and you are running a Multiple CPU threaded Application, and then don’t mix vCPU’s on the same Host / Cluster.

Thoughts? Am I understanding this correctly? Post comments please

I discussed VMware and Onbase 8.2 with Hyland software, they sent me a client guide with the following statement.

“

About Virtual Environments
OnBase is designed to work on the Microsoft Windows operating system. When OnBase is
operated in a virtual environment (such as Citrix, VMWare or Windows Terminal Server),
there are limitations imposed by the environment, in addition to potentially subtle
environmental differences. The Hyland Software QA Department runs testing on physical
hardware and virtual environments. Hyland Software depends upon the suppliers of these
virtual environments to validate that their software is equivalent to a physical machine. When
it appears that an OnBase performance-related issue is either caused by, or is unique to, the
virtual environment, organizations that have deployed an OnBase solution in a virtual
environment may be asked to validate that the performance issue occurs in a non-virtual
environment, to facilitate troubleshooting. Since virtual environments utilize memory and
other components of the operating system differently than non-virtual environments,
removing the virtual component can facilitate a more rapid resolution.
Hyland Software strongly recommends that any customer desiring to run an OnBase solution
in one of these virtual environments thoroughly test the solution prior to putting it into
production, as each site has unique environmental considerations.
For information about using OnBase in a Citrix and Terminal Server environment, see the
OnBase in a Citrix and Terminal Server Environment reference guide, available from your
service provider.

©2008 Hyland Software, Inc.
All rights reserved.

“

Thanks to Hyland for the information.

I plan to move forward with my Onbase VMware Migration.

We recently upgrade our Hyland Onbase server to a new version, 8.2.

This required a touch to be done to each device, as the following software needed to be loaded.

1. The Hyland Code Access Security
2. The Visual C ++ run-time components
3. And .net 2.0 sp1

To install these the user needed Admin rights, & I didn’t think it would be a good use of time to sit at each device, and wait for .net 2.0 sp1 to install. This and we had over 50 devices to hit.

Here was my solution

I copied all the files to a share, lets call the server test, and the share onbase

We run spybot corporate on all of our devices , therefore I had to shut that down first, and I introduced ping’s to pause the bat file, to allow the computers to load before starting the install.

I used taskkill to kill spybot more than once, for the same process, because of the variation of computer speeds in our environment.

I had to copy the Code Access Security to the desktop to get it to see the XML file correctly, from which in pulls the production server name.

Therefore, I developed a login script that did the following.

Here we go

“

net use Z: \testonbase$
Z:

taskkill /f /im Tea*
ping 127.0.0.1
ping 127.0.0.1
ping 127.0.0.1
taskkill /f /im Tea*
ping 127.0.0.1
ping 127.0.0.1
ping 127.0.0.1
vcredist_x86.exe
taskkill /f /im Tea*
NetFx20SP1_x86.exe /qb /norestart
ping 127.0.0.1
ping 127.0.0.1
ping 127.0.0.1
ping 127.0.0.1
ping 127.0.0.1
ping 127.0.0.1
mkdir "C:Documents and Settings%username%Desktoponbase"
copy "Hyland Code Access Security.msi" "C:Documents and Settings%username%DesktoponbaseHyland Code Access Security.msi"
copy settings.xml "C:Documents and Settings%username%Desktoponbasesettings.xml"
"C:Documents and Settings%username%DesktoponbaseHyland Code Access Security.msi" /passive
ping 127.0.0.1
ping 127.0.0.1
ping 127.0.0.1
rmdir /S /Q "C:Documents and Settings%username%Desktoponbase

net use Z: /delete  /Y
logoff

“

Therefore, with this script all you had to do was login to the device and walk away.

I hope someone can get some use from this.

If you didn’t get a chance to read this article in this weeks Information Week I’d say it’s a must read.

http://www.informationweek.com/news/software/operatingsystems/showArticle.jhtml?articleID=212903501

Jason Boche has taken the time to outline Idle Memory Tax in VMware ESX.

Jason, I wanted to thank you for taking the time on this, and I am hoping you do not mind me quoting your Article.

“

Memory over commit is a money/infrastructure saving feature that fits perfectly within the theme of two of virtualization’s core concepts: doing more with less hardware, and helping save the environment with greenness. While Microsoft Hyper-V offers no memory over commit or page sharing technologies, VMware has understood the value in these technologies long before VI3. I’ve mentioned this before – if you haven’t read it yet, take a look at Carl Waldspurger’s 2002 white paper on Memory Resource management in VMware ESX Server.

One of VMware’s memory over commit technologies is called Idle Memory Tax. IMT basically allows the VMKernel to reclaim unused guest VM memory by assigning a higher “cost value” to unused allocated shares. The last piece of that sentence is key – did you catch it? This mechanism is tied to shares. When do shares come into play? When there is contention for physical host RAM allocated to the VMs. Or in short, when physical RAM on the ESX host has been over committed – we’ve granted more RAM to guest VMs than we actually have on the ESX host to cover at one time. When this happens, there is contention or a battle for who actually gets the physical RAM. Share values are what determine this. I don’t want to get too far off track here as this discussion is specifically on Idle Memory Tax, but shares are the foundation so they are important to understand.

Back to Idle Memory Tax. Quite simply it’s a mechanism to take idle/unused memory from guest VMs that are hogging it in order to give that memory to another VM where it’s more badly needed. Sort of like Robin Hood for VI. By default this is performed using VMware’s balloon driver which is the more optimal of the two available methods. Out of the box, the amount of idle memory that will be reclaimed is 75% as configured by Mem.IdleTax under advanced host configuration. The VMKernel polls for idle memory in guest VMs every 60 seconds. This interval was doubled from ESX2.x where the polling period was every 30 seconds.

Here’s a working example of the scenario:

• Two guest VMs live on an ESX/ESXi host with 8GB RAM
• Each VM is assigned 8GB RAM and 8,192 shares. Discounting memory overhead, content based page sharing, and COS memory usage, we’ve effectively over committed our memory by 100%
• VM1 is running IIS using only 1GB RAM
• VM2 is running SQL and is request the use of all 8GB RAM
• Idle Memory Tax allows the VMKernel to “borrow” 75% of the 7GB of allocated but unused RAM from VM1 and give it to VM2.  25% of the unused allocated RAM will be left for the VM as a cushion for requests for additional memory before other memory over commit technologies kick in

Here are the values under ESX host advanced configuration that we can tweak to modify the default behavior of Idle Memory Tax:

• Mem.IdleTax – default: 75, range: 0 to 99, specifies the percent of idle memory that may be reclaimed by the tax
• Mem.SamplePeriod – default: 60 in ESX3.x 30 in ESX2.x, range: 0 to 180, specifies the polling interval in seconds at which the VMKernel will scan for idle memory
• Mem.IdleTaxType – default: 1 (variable), range: 0 (flat – use paging mechanism) to 1 (variable – use the balloon driver), specifies the method at which the VMKernel will reclaim idle memory. It is highly recommended to leave this at 1 to use the balloon driver as paging is more detrimental to the performance of the VM

VMware recommends that changes to Idle Memory Tax are not necessary, or even appropriate. If you get into the situation where Idle Memory Tax comes into play, you need to question the VMs that have large quantities of allocated but idle memory. Rather than allocating more memory to the VM than it needs, thus wrongly inflating its share value, consider reducing the allocated amounts of RAM to those VMs.

“

I thought this was very interesting, I my self have never played with this setting.

I did have some questions on performance on both the VMware Host side, and VMware Guest side. Do either have a performance hit when ESX is using Idle Memory Tax to move that memory.

I could see large uses for this, if you have a system that only uses large amounts of memory at certain times of the day.

In fact, would you want to design your systems to use this ( by forcing low ram ), if you have certain systems that run at night, when other systems are idle ( with more ram) that normally are only used during the day?

Source Post http://www.boche.net/blog/index.php/2009/01/29/idle-memory-tax/