I ran into an issue earlier this week where customer had a 2 node ESXi 4.1 deployment using NFS shared storage. We were running into performance issues on the SQL and CRYSTAL VM's in particular. After tracking down the issue, I noted that the freezing correlated to the same time the Veeam backup job was running.

<!--more-->

I instantly pulled the VMs out of the backup jobs and the problems quieted down.

There was an open snapshot on the SQL VM, so shutting down the VM and "Deleting All" snapshots resolved that issue.

Roping in Veeam support led me down the path to a VMware KB Article which shed some light on the subject and introduced me to a couple settings under the VMware hood that I had not been introduced too.

The VMware KB Article in question is <strong><a href="http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&amp;docType=kc&amp;externalId=1031106&amp;sliceId=1&amp;docTypeID=DT_KB_1_1&amp;dialogID=152827602&amp;stateId=0%200%20152829571">HERE</a></strong>. And it is properly titled: <strong>"Virtual machine freezes temporarily during snapshot removal on an NFS datastore in a ESX/ESXi 4.1 host"</strong>, and it is highly recommended reading.

<strong>Moral of the story if you are backing up your VMs to an NFS Datastore and ctkEnabled on VM (via Change Block Tracking)  then you could very well experience some pain and suffering.</strong>

To resolve the issue, I had to perform 2 Steps:

1. Go into the Veeam backup job and turn off a "recommended" advanced setting (see below) effectively <strong>Turning off Change Block Tracking:</strong>

<a href="http://blogs.eGroup-us.com/wp-content/uploads/2011/02/veeam1.jpg"><img class="aligncenter size-full wp-image-3374" src="http://blogs.eGroup-us.com/wp-content/uploads/2011/02/veeam1.jpg" alt="" width="607" height="525" /></a>
<p style="text-align: center"><strong><a href="http://blogs.eGroup-us.com/wp-content/uploads/2011/02/veeam1.jpg"></a></strong></p>
 

2. Power down the VM experiencing the performance hit, in vCenter go to Edit Settings, Options, General, Configuration Parameters, and scroll down until you see ctkEnabled (see below). This setting needs to be changed back over to <strong>FALSE</strong>. (Note the Veeam  backup job with the Change Block Tracking enabled) will set this on the VM to <strong>TRUE</strong>.

<a href="http://blogs.eGroup-us.com/wp-content/uploads/2011/02/ctkEnabled.jpg"><img class="aligncenter size-full wp-image-3375" src="http://blogs.eGroup-us.com/wp-content/uploads/2011/02/ctkEnabled.jpg" alt="" width="603" height="575" /></a>
<p style="text-align: center"><a href="http://blogs.eGroup-us.com/wp-content/uploads/2011/02/ctkEnabled.jpg"></a></p>
Note: This can only be changed while the VM is powered down. Now that the setting has been changed on the Veeam backup job and on the VM itself. The VM can be powered backup, and the VM can be added back into the Veeam backup job.

As soon as the Veeam 5 Best Practices are released, I will certainly follow backup and link them to this post.