About Me

My photo
This blog has been created to share technical information, interesting innovations that I notice on daily basis and Architectural/Consulting overview of various technologies. My areas of interest, on which I would be blogging, are VMware, Microsoft and Citrix Technologies. I hope you will enjoy this blog and share your experience with me.

Export VMware App Volumes - Writable Volumes from VSAN Datastore

If you have a VMware VSAN environment and you wanted to export a App Volumes - Writable Volumes from the vsanDatastore to another datastore, storage or for VMware GSS/R&D for further analysis go ahead and read further!

Background – The traditional way of exporting the Writable Volumes from the source vsanDatastore was to attach the *.vmdk to a dummy VM as a “Existing Hard Disk” and export the dummy VM using the “Export OVF Template” option from the vCenter. Repeat all the steps on target datastore where-in it needs to be imported. However, if you want an alternate and easy method than the dummy VM then follow the below steps.

Step by Step Instructions:

-- SSH to any ESXi Host Resource Cluster where the WV is stored and browser to the cloudvolumes/writable directory location:

# cd /vmfs/volumes/vsanDatastore/cloudvolumes/writable (This is the location where all end-user writable volumes are stored)

-- Now search for the end-user (E.g twood) for which you want to export the Writable Volumes.

# ls -lh | grep twood

User to export WV

-- Now lets open the *.vmdk file using “cat” command to retrieve the Object ID information. Make a note of the ObjectID

# cat DOMAIN!5C!twood.vmdk

Cat to find ObjectID

--
In my scenario the Object ID was properly pre-created I didn’t have to use the objtool to find out the Object opened. However, in some cases you might have to run the following command

# /usr/lib/vmware/osfs/bin/objtool open -u <object ID> (Where Object ID is displayed using the ‘cat’ command

This screenshot below is an example of situations where in the Object ID is not properly pre-created. It will provide you with an output Object Opened at path:
Objecttool Output

-- Now using WINSCP login to the same ESXi Host and go to the path:
Object ID – /vmfs/device/vsan/d17efe58-5610-4dd2-0d9e-ecf4bbea2830 (my scenario)
Or/else Object opened at path in the screenshot above.

Download the file “d17efe58-5610-4dd2-0d9e-ecf4bbea2830” which is Writable Volumes (*.vmdk) file and move the files to local or remote location that you are using the WINSCP tool.

-- Rename the Object ID to a friendly name shown in the cloudvolumes/writable Directory Folder. I renamed it (DOMAIN!5C!twood.vmdk)

-- You don’t need the *.vmdk.metedata file

-- The Writable Volumes is now exported out of the VSAN environment you can attach the *.vmdk to a non App Volumes Agent machine to look at the contents inside the Writable Volumes. If you are going to send the vmdk to VMware GSS/R&D make sure to zip it before uploading

I hope you will find these steps useful and help you export a Writable Volume from your vsanDatastore. I haven’t been able to try AppStacks with this method its on my to-do list.

Thanks,
Aresh Sarkari

McAfee Exclusion for VMware App Volumes 2.x – 100% CPU Issues

In your Virtual Desktop Infrastructure with the following configurations:

If you start noticing 100 % - CPU Usage for prolonged period of time and the Horizon Session getting disconnected from time to time after launch then you might need to include the following exclusion within your Writable Volumes (UIA+Profile) snapvol.cfg file:

#McAfeeExclusion
exclude_process_path=\Program Files\Common Files\McAfee\SystemCore

My colleague Daniel Bakshi has written an extensive blogpost on how to modify the snapvol.cfg for individual or group of end-users please reference it to make the necessary changes - Using the VMware App Volumes snapvol.cfg File to Customize Writable Volumes

I hope you will find these exclusion useful and will help you resolve a similar issue a lot quicker. A big thanks to Art Rothstein in helping to troubleshoot and resolve the issue.

Thanks,
Aresh Sarkari

Create a Memory Dump from a Suspended Virtual Machine - VMware vSAN

If you have a VMware VSAN environment and you wanted to capture a memory dump of the Virtual Machine for debugging or want to provide memory.dmp to VMware GSS or R&D for further analysis go ahead and read further!

Use Case – In our scenario had a few VDI Desktops running Windows 10 1607 + Horizon 7.3.1 + App Volumes Writable Volumes 2.13.1 + UEM 9.2.1 that were getting into unresponsive state. As a last resort we wanted to capture the memory dump to find out more what is causing the VM to get unresponsive.

Step by Step Instructions:

-- Using the vCenter console select the Virtual Machine VM – Power – Suspend

-- This will create the *.vmss and *.vmem file for Debugging. (Note the *.vmem file is applicable for ESXi 6.0 onwards)
VM Directory

--
Make a note of the ESXi host Name/IP for the VM is in Suspend state

-- SSH to the ESXi Host and browser to the VM Directory location:

# cd /vmfs/volumes/vsanDatastore/od-av-troub-1 (Where “od-av-troub-1” is the VM name)


-- Now lets open the *.vmem file using “cat” command to retrieve the Object ID information. Make a note of the ObjectID

# cat od-av-trou-1-7622414e.vmem

Object ID

--
In my scenario the Object ID was properly pre-created I didn’t have to use the objtool to find out the Object opened. However, in some cases you might have to run the following command

    # /usr/lib/vmware/osfs/bin/objtool open -u <object ID> (Where Object ID is displayed using the ‘cat’ command

    This screenshot below is an example of situations where in the Object ID is not properly pre-created. It will provide you with an output Object Opened at path:
    Object opened at

-- Now using WINSCP login to the same ESXi Host and go the path:
Object ID – /vmfs/device/vsan/2c86055a-573b-d20a-5cdf-ecf4bbea1e48 (my scenario)
Or/else Object opened at path and download the file “2c86055a-573b-d20a-5cdf-ecf4bbea1e48” which is your ”*.vmem file and move the files to local or remote location that you are using the WINSCP tool.

--
Rename the Object ID to a friendly name shown in the VM Directory Folder. I renamed it (od-av-trou-1-7622414e.vmem)

-- For the *.vmss (od-av-trou-1-7622414e.vms) you can directly WINSCP to the ESXi Host and go to the location in the table and move the files to your local or remote location

-- Once you have both the files *.vmem and *.vmss you can use a VMware Vmss2core Fling and convert it to a dump. Please make sure you meet the requirements and use the appropriate switches to your environment

# vmss2core -W8 od-av-trou-1-7622414e.vmss od-av-trou-1-7622414e.vmem 

-- The above command will generate a memory.dmp file which can used in WINDBG for further analysis. If you are sending the dump file to someone make sure use *.zip and compress it before sending.

I hope you will find these steps useful and save a lot of time during daunting unresponsive VM issues. A big thanks to Frank EscarosBuechsel to helping with the entire procedure.

Thanks,
Aresh Sarkari

Persistence Profile - F5 LTM Load Balancing for VMware Unified Access Gateway Appliance

If you are using F5 LTM in the DMZ to load balance (LB) the VMware Unified Access Gateway (UAG) appliance, it is very important to use the iAPP or the F5 Deployment guide to set the Persistence Profile options properly or/else you might end up with issues.

Background:

The F5 LTM VIP for UAG Appliance was created manually without using the f5_vmware_view iApp and the Persistence Profile settings were manually configured. (I highly recommend to use the iApp and go through the F5 deployment guides)

Issue1:

The BLAST connection fails in the backend. The original SessionID request was going to UAG1 and due to the LB in the front the next request for the same SessionID was going to UAG2.

Log Snippet UAG1:
[2017-XX-XX 12:50:33.428] [INFO]    2289 [absg-master] - Added route 810DF5FF-*** to target 10.x.x.x|22443

Log Snippet UAG2:
[2017-XX-XX 12:50:35.589] [ERROR]    2723 [absg-worker] - Failed to resolve proxying route: 810DF5FF-***

As noted above the SessionID is the same but the initial BLAST connection request is going to different UAG appliance instead of going to the same appliance which it originally initiated.

Issue2:
You might time to time receive an Error Message “Your session has expired. Please re-connect the server” while entering the username, password and 2-factor authentication details on UAG landing page. It has to do with the timeout value on the F5 persistence profile – Source IP Address

Session has expired

Solution:
Whenever you have F5 LTM as the Load Balancer in front of UAG make sure you handle these three settings carefully to not run into the above described issue:

Timeout Value: Specifies the duration of the persistence entries.
This value should match the Horizon Administrator(Global Settings - View Administrator session timeout) time out value. The default value set on the F5 LTM is 180 seconds = 3 mins

Example - If the View Administrator session timeout is 480 mins

View Admin Session Timeout
Then we should set the same value under the F5 Timeout value in seconds

F5 Timeout Value

Mirror Persistence: If the active unit goes into the standby mode, the system mirrors any persistence records to its peer.

Mirror Persistence

We had this option un-check as it was a manually configured persistence profile

Match Across Services: All persistent connections from a client IP address that go to the same virtual IP address also go to the same node. The default is disabled

Match Across Services

We had this option un-check as it was a manually configured persistence profile

How does the overall Persistence of the profile look:
Persistence Profile f5

If you are using the F5 Horizon iApp for the configuration of the UAG VIP then you might not end-up with the above issue.

I hope you find these tips useful during the F5 LTM VIP creation for VMware Unified Access Gateway Appliance.

Thanks,
Aresh Sarkari


Troubleshooting Horizon TrueSSO aka Horizon Enrollment Server like a Ninja!

If you have deployed Horizon TrueSSO feature within your environment. Then the most obvious question is how do you troubleshoot during issues? Let me give you some tips and tricks around troubleshooting TrueSSO aka Enrollment Server feature:

  • If you have two teams split one team managing the Active Directory/Certificate Services and other team managing Horizon infrastructure. Then following are the tips for the Horizon Admins. Install the Microsoft RSAT tools on your domain joined machine or Enrollment Servers and install the AD Certificate Services Tools. This will provide you the ability to see the following snap-ins in read-only mode:
    • Enterprise PKI – Allows you to check the CDP and CRL and Issuing CA Status
    • Certificate Templates – TrueSSO, Enrollment Agent (Computer) Templates etc.

SNAGHTML6730c9ff

  • Make sure to Enable the Trace logging on the Enrollment Servers and Horizon Agent (within master image) during troubleshooting. It will provide additional details on the error message
    • [HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VDM]
      "debugEnabled"="true"
      "traceEnabled"="true"
    • How to know whether the end-users logged in via TrueSSO – Interactive_SmartCard_Logon will be visible in the Horizon Agent (if Trace Log is enable)image
    • If TrueSSO is not used and SAML - CLEAR(Text)_PASSWORD is used you will receive the following in Horizon Agent logs (if trace is enable
      image
  • If you have two Issuing CA’s for High Availability and redundancy then make sure you import the TrueSSO template by Clicking Certificate Templates > New > Certificate Template to Issue. Select “TrueSsoTemplate” from the “Enable Certificate Templates” dialog and press “OK.” on the other Issuing CA. If you skip this step it will complain in Horizon Administrator dashboard - The primary and secondary enrollment server is not connected to the certificate servers “XXXXXX
  • Read and learn to use the VMWare Fling es_diag.exe it will provide a lot of information from the Horizon Enrollment Server stand point and equip you to troubleshoot issues with Certificate Servers.
    • /ListConfigs
    • /ListEnvironment
    • /EnrollmentTest

My colleague Tarique Chowdhury has posted few troubleshooting steps in the following post under Section – Testing it will provide more details as to what to look in the logs.

Log Entries 1

Log Entries 2

I hope you find this post useful during the Horizon TrueSSO aka Enrollment Server troubleshooting.

Thanks,
Aresh Sarkari

Top 10 lessons during Horizon TrueSSO deployment aka Horizon Enrollment Servers

Recently got an opportunity to deploy the VMware Horizon TrueSSO within our environment. TrueSSO provides user with the True SSO (single sign-on) feature, after users log in to VMware Identity Manager (WorkSpaceOne) using a RSA SecurID authentication(optional), users are not required to enter Active Directory credentials in order to use virtual desktop or hosted application.

Let me share my top 10 lessons learnt from the deployment:

  1. In the production deployment recommend to size the Enrollment Server Windows VM as same as the Connection Server(ES role is not very resource intensive)
    • CPU - 4 vCPU
    • Memory – 10 GB RAM
    • HDD – 80 GB
  2. Make sure the “Group Scope” is selected as “Universal” for the  Active Directory Group in which the Enrollment Server - Computer Account is added
  3. On the newly created TrueSSO template (SmartCard Login and Client Authentication) make sure under the Security Tab “Authenticated Users” group has Read permissions and The Active Directory group for the Enrollment Servers (Computer Account) has Read and Enroll
  4. If you are deploying more than one Enrollment Server go in the Horizon ADAM database and add the following value to load balance between two Enrollment Servers:
    cs-view-certsso-enable-es-loadbalance=true
  5. For Large scale AD deployments, it is recommend to add the registry for “ConnectToDomains”=domainname.com
    HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VDM\Enrollment Service

    ConnectToDomain
  6. Make Sure the template to be used for TrueSSO, you have selected the check box “Do not store certificate and request in the CA database” and run the following command on the CA server. (without quotes)
    “certutil –setreg DBFlags +DBFLAGS_ENABLEVOLATILEREQUESTS”

    TrueSSO Template Properties
  7. To support Smartcard Logon the following Requirements must be met by the Domain Controller or Kerberos Authentication Certificate:
    • Template name should be Domain Controller or Kerberos Authentication Certificate
      Kerberos Template Properties
    • DNS Name should be selected under Subject Name
      Subject Name Properties
    • Key Usage Extension should be “Digital Signature” and “Key Enciphement
      Key Usage Extension
  8. Make sure the the CA issuing Domain Controller Certificates has the following requirements met (Use GPO’s to deploy the below)
    • Add the Root Certificate to the Enterprise NTAuth Store
    • Add the Root Certificate to Trusted Root Certification Authorities
    • Add an Intermediate Certificate to Intermediate Certification Authorities
  9. Use the True SSO Diagnostic Utility Fling to troubleshoot Enrollment Server, Active Directory PKI Settings and Enterprise CA
  10. On the Domain Controllers under the registry location HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\EnterpriseCertificates\NTAuth\Certificates
    A key with the “Issuing CA Certificate” thumbprint needs to be created on all the domain controllers participating in the TrueSSO. Ideally if the Step 7&8 are done correctly you should not run into this problem. (In our case we had to open-up a Microsoft Case to get this resolved as we were receiving KDC errors.)

My colleague Tarique Chowdhury has written three awesome blog post on the TrueSSO feature make sure to check them out:

Introduction - https://blogs.vmware.com/euc/2016/03/true-sso-single-sign-on-view-identity-manager-authenticate.html

Advance - https://blogs.vmware.com/euc/2017/02/horizon-7-sso-advanced-features.html

Setting up in Lab - https://blogs.vmware.com/euc/2016/04/true-sso-setting-up-in-a-lab.html

I hope you find this post useful during the Horizon TrueSSO deployment

Thanks,
Aresh Sarkari

Vulnerability Scanner for WannaCry and NoPetya – VDI environments

With a lot of enterprises in the middle of the WannaCry and NoPetya vulnerability. If you are running a enterprise VDI environment the fix is pretty simple. Just target your Master VM or Golden Master images and run the Windows Update. Once you have updated the image simply Recompose or Push-Image the desktops pools with the latest updates. Your environment is quickly secured! These vulnerability reiterate the importance of regular patching within the production environments for your Core infrastructure + Master Images.

WannaCry Patch for All Windows versions - https://technet.microsoft.com/en-us/library/security/ms17-010.aspx

Vulnerability Scanner

A quick and easy way to scan your environment is using a free EternalBlue vulnerability scanner. - http://omerez.com/eternalblues/

image

Simply download the scanner and launch it on a Windows VM of your choice on Windows 7/8.1/10.

IP Range:
The tool by default tends to select the /24 subnet. However, if you have a bigger subnet like a /19 to scan simply enter the Start and End of the entire subnet range. In this example its a 192.168.0.0/19. It will scan for 8190 IP addresses.

image

I hope you scan your environment ASAP! Get rid of the vulnerability ASAP!

Thanks,
Aresh

Horizon 7.2 – RDS Farm with View Composer fails on “Customizing”

While creating a RDSH Farm in Horizon 7.2 using View Composer – Linked Clones and Custom Specification Manager the creation would fail on “Customization” within the View Administrator console. Upon investigation within the vCenter the Windows Servers 2012 R2 RDS Session host VM’s where not getting a valid IP and receiving the169.x.x.x APIPA addresses.

After researching quite a bit the most common solution to the problem was:

  • Un-install and re-install vmwaretools
  • Un-install and re-install Horizon Agent 7.2 on RDS Master Image
  •  

After performing the above two steps the issue completely changed from getting 169.x.x.x APIPA address to a proper DHCP server routable address. However, we are getting a different error this time:

Windows could not finish configuring the system after a generalized sysprep”.

windows error-sysprep

Final Solution

Within the master image we were using the MacAfee VSE Agent Patch 7 as the antivirus protection. This particular version was causing the issue with the sysprep to fail during customization.

After following the below MacAfee KB and installing VSE Patch 9 the error was resolved and customizing of the RDS VM as per the Custom Specification Manager was successful.

Reference Link:
Windows could not finish configuring the system (Sysprep fails when VirusScan Enterprise Patch 7/8 is included in a Windows installation image)

I hope this solution will save time to get the Horizon 7.2 RDSH Farm created quickly.

Thanks,
Aresh

Error 1303 The installer has insufficient privileges to access this directory – Upgrade from App Volumes 2.12 to 2.12.1

With the latest version of App Volumes 2.12.1, you don’t have to uninstall the older version of App Volumes Manager. The latest App Volumes Manager 2.12.1 installer takes care of uninstalling, fresh-install and retain all the configuration details and settings automatically for you.

During the upgrade I encountered the following error:

“Error 1303. The installer has insufficient privileges to access this directory: C:\Program Files(x86)\CloudVolumes\Manager\log. The installation cannot continue. Log on as an administrator or contact your system administrator.”

App Volumes Upgrade Error

Resolution:
In our scenario we have VMware vRealize Log Insight Agent installed on the App Volumes Manager VM’s which is doing Syslog. The Log Insight agent captures the logs(production.log) inside the folder “C:\Program Files(x86)\CloudVolumes\Manager\log”. As the service is in the running state, it didn't allow the folder to delete and left a ghost folder on the filesystem.

Log Insight Agent Service

After going into the services.msc and stopping the VMware vRealize Log Insight Agent service and click Retry, the setup manages to complete the upgrade successfully.

I hope this workaround helps you during your upgrade if you encounter a similar error message.

Thanks,
Aresh

EUC Session for VMworld 2017

Folks, I have submitted a session for the VMworld 2017. If you would like to see them go on stage then please vote!

My Session:
The secret sauce behind VMware’s internal Horizon desktop deployments [1255]
Ever asked yourself “How does VMware architect their own global Horizon desktop environment?”, “Have they encountered the same obstacles we are facing?” Over the past two years VMware has been re-architecting and re-deploying their virtual desktop infrastructure with Horizon, App Volumes and User Environment Manager (UEM) running on top of the full VMware SDDC stack (vSphere, VSAN, NSX) and integrating with vRealize Operations Manager and Log Insight. In this session the lead architects will reveal all.

Direct Link to my session VOTE HERE: https://my.vmworld.com/scripts/catalog/uscatalog.jsp?search=1255

How to Vote?
Create a new account if you don’t have a existing one -
https://www.vmworld.com/myvmworld.jspa and click on “Create Account”

VMworld 2017 Catalogue
Search in VMworld 2017 Catalogue -
https://my.vmworld.com/scripts/catalog/uscatalog.jsp. Search here for other interesting sessions.

I highly recommend voting on other great sessions submitted by my colleagues.

Please Vote!
Aresh Sarkari

My Blog List