About Me

My photo
This blog has been created to share technical information, interesting innovations that I notice on daily basis and Architectural/Consulting overview of various technologies. My areas of interest, on which I would be blogging, are VMware, Microsoft and Citrix Technologies. I hope you will enjoy this blog and share your experience with me.

Create a Memory Dump from a Suspended Virtual Machine - VMware vSAN

If you have a VMware VSAN environment and you wanted to capture a memory dump of the Virtual Machine for debugging or want to provide memory.dmp to VMware GSS or R&D for further analysis go ahead and read further!

Use Case – In our scenario had a few VDI Desktops running Windows 10 1607 + Horizon 7.3.1 + App Volumes Writable Volumes 2.13.1 + UEM 9.2.1 that were getting into unresponsive state. As a last resort we wanted to capture the memory dump to find out more what is causing the VM to get unresponsive.

Step by Step Instructions:

-- Using the vCenter console select the Virtual Machine VM – Power – Suspend

-- This will create the *.vmss and *.vmem file for Debugging. (Note the *.vmem file is applicable for ESXi 6.0 onwards)
VM Directory

--
Make a note of the ESXi host Name/IP for the VM is in Suspend state

-- SSH to the ESXi Host and browser to the VM Directory location:

# cd /vmfs/volumes/vsanDatastore/od-av-troub-1 (Where “od-av-troub-1” is the VM name)


-- Now lets open the *.vmem file using “cat” command to retrieve the Object ID information. Make a note of the ObjectID

# cat od-av-trou-1-7622414e.vmem

Object ID

--
In my scenario the Object ID was properly pre-created I didn’t have to use the objtool to find out the Object opened. However, in some cases you might have to run the following command

    # /usr/lib/vmware/osfs/bin/objtool open -u <object ID> (Where Object ID is displayed using the ‘cat’ command

    This screenshot below is an example of situations where in the Object ID is not properly pre-created. It will provide you with an output Object Opened at path:
    Object opened at

-- Now using WINSCP login to the same ESXi Host and go the path:
Object ID – /vmfs/device/vsan/2c86055a-573b-d20a-5cdf-ecf4bbea1e48 (my scenario)
Or/else Object opened at path and download the file “2c86055a-573b-d20a-5cdf-ecf4bbea1e48” which is your ”*.vmem file and move the files to local or remote location that you are using the WINSCP tool.

--
Rename the Object ID to a friendly name shown in the VM Directory Folder. I renamed it (od-av-trou-1-7622414e.vmem)

-- For the *.vmss (od-av-trou-1-7622414e.vms) you can directly WINSCP to the ESXi Host and go to the location in the table and move the files to your local or remote location

-- Once you have both the files *.vmem and *.vmss you can use a VMware Vmss2core Fling and convert it to a dump. Please make sure you meet the requirements and use the appropriate switches to your environment

# vmss2core -W8 od-av-trou-1-7622414e.vmss od-av-trou-1-7622414e.vmem 

-- The above command will generate a memory.dmp file which can used in WINDBG for further analysis. If you are sending the dump file to someone make sure use *.zip and compress it before sending.

I hope you will find these steps useful and save a lot of time during daunting unresponsive VM issues. A big thanks to Frank EscarosBuechsel to helping with the entire procedure.

Thanks,
Aresh Sarkari

No comments:

My Blog List