We recently were faced with troubleshooting an issue in a customer's environment where we were intermittently losing SharePoint 2010 and Exchange 2007 services (on separate servers). There was no pattern and the only commonality discovered was a .Net runtime error in the Windows Event Log. This was very painful for the customer as they would lose one or both of these systems during critical business hours. Read more to learn what the root cause was and how it was resolved.

<!--more--><strong>The Windows Event Log errors we were seeing followed one of 2 patterns.</strong>

<strong>We saw errors similar to:</strong>

<tt>.NET Runtime
.NET Runtime version * Fatal Execution Engine Error *
Application Error:
Faulting application name: w3wp.exe *
Faulting module name: mscorwks.dll *
Exception code: 0xc00000005</tt>

<strong>as well as:</strong>

.NET Runtime version 2.0.50727.4952 - Fatal Execution Engine Error (000007FEF896FA42) (80131506)

It took us a lot of digging to get to a point where we were reliably correlating the application crash times with DRS moves. This particular customer uses Full Automation and the servers were being moved based on a setting of Aggressive. This realization led us to the following VMware KB:

<a href="http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&amp;docType=kc&amp;externalId=1029398&amp;sliceId=1&amp;docTypeID=DT_KB_1_1&amp;dialogID=153668109&amp;stateId=0">VMware ESXi 4.1 Patch ESXi410-201011401-BG: Updates Firmware</a>

<strong>The relevant excerpt from the KB article:</strong>

If a 64-bit Windows virtual machine that uses software techniques  for instruction set virtualization or Binary Translation (BT) is resumed  from the suspended state or is migrated to an ESX 4.1 host, the virtual  machine might stop responding, and the Microsoft Windows Event Logs  might display error messages similar to the following:
<tt>.NET Runtime
.NET Runtime version * Fatal Execution Engine Error *
Application Error:
Faulting application name: oobe.exe *
Faulting module name: mscorwks.dll *
Exception code: 0xc00000005</tt>

In this particular case our initial validation of this solution was to set DRS to disabled for the 5 VMs that could potentially be affected (WIN2K8 R2 64). We allowed this change to bake for approximately 7 days before declaring this a sound solution to the problem. The recommended patch was not an option for us as the customer was running hardware that prevented this. Given that Disabling DRS was on the recommended solutions list we reviewed we elected to leave things as they are and wait for the next hardware and ESXi upgrade by the customer to re-enable DRS on these machines.