Both of those are great features (as well as the many others I didn’t mention), but one I think that flew a little under the radar was the support for multi NIC vMotion. How much time as a VMware Administrator have we waited for VM’s to be evacuated from a host going into maintenance mode? If you were like me and only had 1GB uplinks on your hosts you were limited (IE supported by VMware) to four concurrent vMotions. Set Maintenance Mode and go to Starbucks. If you are one of the cool kids and have 10GB uplinks that number gets doubled to 8 concurrent vMotions. Set Maintenance Mode and run to the break room for a coffee refill.

If these numbers look familiar they should, they are the same as vSphere 4.1. And this is were the joy of Multi NIC vMotion comes in! Since the amount of concurrent vMotions hasn’t changed, how about changing the amount of NIC’s that can be used? You can still only move four VM’s at once, but instead of using a single NIC how about two or four? Same amount VM’s being moved but just more bandwidth to move them. If you are utilizing 1GB NICs the maximum you can combine is 16. For 10GB that number is decreased to four.

If you are not familiar with setting up Multi NIC vMotion both VMware and Duncan Epping have fantastic posts on how to do so. The VMware KB article (KB2007467) is located HERE and the blog post from Duncan is located HERE.

So after following either of the above guides lets do some checking to see if it truly is working. I find the easiest to see this in action is via ESXTOP. Log into one of your hosts (or vMA) launch ESXTOP and press “n” to view the networking screen. Identify your vmk/vmnics that you configured and verify that traffic is crossing each.

To demonstrate this I setup a vSwitch with two uplinks and configured two vMotion portgroups. In the screen shot below you can see traffic crossing over each interface.

A more in depth way to see this as well is to view the /var/log/vmkernel.log. In the screenshot below you will see the kernel binding to the two vmk ports:

For overall performance testing in my lab environment I tested moving four VM’s with only a single vMotion interface and the same four VM’s with multiple vMotion interfaces:

As you can see from the chart there was almost a 50 percent reduction in time when comparing the single interface to the dual interface. If you have upgraded your environment to vSphere 5 be sure to revisit your network design to make sure you are make use of this great new feature.

-Jason

I received the drives on Thursday and brought down one of my hosts and swapped out the existing internal RAID 5 I was running for the SSD’s. I installed ESXi on one SSD and used the second as a dedicated datastore (this config is just for performance testing, each host will end up with a single SSD) to start some testing.

My ESXi hosts are whitebox machines made up of an ASUS P7F-E motherboard, Intel Xeon X3430 Lynnfield 2.4GHZ QC, and 16GBs of RAM. One draw back of my hosts is that the onboard SATA controller is only 3GBs where as the drives are 6GBs capable which will have in effect on my performance numbers.

For testing I created a new VM based on Windows Server 2008 R2 with 2GB of memory and a single vCPU. IOmeter was installed on the VM and I used the configuration provided from the “Unofficial Storage Performance” thread located HERE on the VMware Communities. Each test was performed three times and used the averages to populate the charts below. The tests where performed with the VM running on the same SSD as ESXi was installed (Shared Datastore) and the VM running on a SSD that was configured as a standalone datastore (Dedicated Datastore).

And now the results:

As you can see from the charts above the overall IOPS and MBps are about the same whether the VM was running on the shared or dedicated datastore. However, I was surprised by the changes in the Max IO Response Time numbers.

All in all very happy with the performance numbers I am seeing, and as the main goal was to replace the internal RAID5 on each host (mostly from a power consumption perspective) I am glad I made the purchase.

-Jason

I would have liked to say that branching out and moving my career another step forward was a new years resolution, but in fact it was something that started back last summer. At that time my goal was to learn more and dig deeper into VMware, go beyond my VMware VCP certification and obtain my VCAP-DCA certification. I scheduled the exam for August and set off on my studying. Long story short I had successfully passed the exam and was feeling good about the additional knowledge I had learned. But that was nothing compared to what happened later that month.

Even though I have been working with VMware for the last 4 years or so, I had never had the ability/option of attending VMworld. There had always been talk amongst managers about sending someone, but it never really seemed to pan out. This year that was all going to change, my boss had notified me that I would be attending VMworld 2011! For me this had to be the biggest eye opening event in my career. I had heard on podcasts, and read on blogs about experiencing it, but none of what I read or heard prepared me for what I was about to see. Between all of the sessions, the vendor displays, or just talking to and meeting new folks it was amazing! After soaking up as much as I could, it was at this point I knew I wanted to, better yet needed to take that next step.

So here I am, waiting for that new challenge, that new adventure to start. I got my bags packed and am ready to go!

For this objective I used the following resources:

• vSphere Monitoring and Performance
• vSphere Troubleshooting
• VMware White Paper “The Architecture of VMware ESXi”
• VMware Support Insider Blog
• VMware KB Article 2000988 (Troubleshooting vSphere Auto Deploy)
• VMware KB Article 1011641 (Collecting Diagnostic Information for VMware vCenter Server)
• VMware KB Article 653 (Collecting Diagnostic Information for VMware ESX/ESXi Using the vSphere Client)
• VMware KB Article 1010705 (Collecting Diagnostic Information for VMware ESX/ESXi Using the vm-support Command)
• VMware KB Article 1027932 (Collecting Diagnostic Information for VMware vCenter Server and ESX/ESXi Using the vSphere PowerCLI)

Knowledge

Identify General ESXi Host Troubleshooting Guidelines

The vSphere Troubleshooting guide is the one stop shop for this section

Troubleshoot Common Installation Issues

Refer to Objective 1.3 and make sure your hosts meet the hardware requirements as well as the VMware HCL. If using AutoDeploy refer to pages 20 thru 26 of the vSphere Troubleshooting guide and also VMware KB 2000988 (Troubleshooting vSphere Auto Deploy).

Monitor ESXi System Health

With the release of ESXi back in the VI 3.5 days it provided a new way to manage your hosts, the Common Information Model (CIM). CIM allows for a standard framework to manage computing resources and presents this information via the vSphere Client. For further information read the VMware White Paper “The Architecture of VMware ESXi” as well as this VMware Support Insider blog post.

To actually see how ESXi and vSphere Client leverages CIM read pages 25 thru 28 of the vSphere Monitoring and Performance documentation.

Export Diagnostic Information

There are multiple ways to get at this information, but I will assume the exam is going to be geared more towards using the vSphere Client for this task. For completeness however I have included links that cover using a console session as well as PowerCLI.

Gathering vCenter Server Log Bundles (VMware KB 1011641, Collecting Diagnostic Information for VMware vCenter Server)

To generate a vCenter Server log bundle, select Start > All Programs > Vmware and select either “Generate vCenter Server Log Bundle – Extended” or “Generate vCenter Server Log Bundle” (Be sure to “Run as Administrator”).

Gathering vCenter Server and ESXi Log Bundles (VMware KB 653, Collecting Diagnostic Information for Vmware ESX/ESXi Using the vSphere Client)

From within the vSphere Client connected to vCenter  click Administration from the menu bar and select Export System Logs. This will allow you to export either vCenter logs, ESX/ESXi logs, are all of the above in a single zip file.

To use the vm-support command or PowerCLI refer to VMware KB 1010705 (Collecting diagnostic information for VMware ESX/ESXi using the vm-support command) and VMware KB 1027932 (Collecting diagnostic information for VMware vCenter Server and ESX/ESXi using the vSphere PowerCLI)

Tools

• vCenter Server and Host Management Guide
• vSphere Monitoring and Performance Guide
• vSphere Troubleshooting Guide

For this objective I used the following resources:

• VCP 5, Section 2 – Plan and Configure vSphere Networking
• VMware Information Guide “VMware Virtual Networking Concepts”
• VMware KB Article 1003893 (Troubleshooting Virtual Machine Network Connection Issues)
• VMware KB Article 1001938 (ESX/ESXi Host Requirements for Link Aggregation)
• VMware KB Article 1004048 (Sample Configuration of EtherChannel/Link Aggregation with ESX/ESXi and Cisco/HP Switches)
• VMware KB Article 1005577 (What is beacon probing)

Knowledge

Verify Network Configuration

Refer to each objective under Section Two. Focus on the core concepts and configuration of both vNetwork Standard Switches and vNetwork Distributed Switches:

• Port/dvPort Groups
• Load Balancing and Failover Policies
• VLAN Settings
• Security Policies
• Traffic Shaping Policies

For additional information read the VMware Information Guide “VMware Virtual Networking Concepts”. This document is based on VI3 but still does a good job with the core functions of a vStandard Switch.

Verify a Given Virtual Machine is Configured with the Correct Network Resources

Instead of duplicating work, refer to VMware KB 1003893, “Troubleshooting Virtual Machine Network Connection Issues”. More then enough information listed there.

Troubleshoot Virtual Switch and Port Group Configuration Issues

One key aspect to remember is when setting up Port Groups or dvPort Groups, spelling counts (as well as upper/lower case)! If a Port Group is spelled Test on one host and is spelled test on a second host vMotion will fail. Same holds true with Security Policies, if one vSwitch on a host is set to accept Promiscuous Mode and it is set to Reject on the other host, again vMotion will fail. Also, refer to the objectives under Section Two to be sure your switches are configured correctly.

Troubleshoot Physical Network Adapter Configuration Issues

This is pretty straight forward as there is not a lot of configuration done at the physical network layer. Be sure that your physical nics that are assigned to a virtual switch (vSwitch or dvSwitch) are configured the same (speed, vlans, etc) on the physical switch. If using IP Hash as your load balancing method make sure on the physical switch side link aggregation has been enabled. Refer to VMware KB 1001938  and VMware KB 1004048 for further details as well as examples. If using beacon probing for network failover detection it standard practice to use a minimum of three (or more) uplink adapters. See VMware KB 1005577 for further details.

Identify the Root Cause of a Network Issue Based on Troubleshooting Information

Using the above notes as well as the linked VMware KB articles one should be able to isolate issue to one of four areas:

• Virtual Machine
• ESX/ESXi Host Networking (uplinks)
• vSwitch or dvSwitch Configuration
• Physical Switch Configuration

Tools

• vSphere Troubleshooting guide
• vSphere Networking guide

For this objective I used the following resources:

• vSphere Monitoring and Performance
• vSphere Storage
• vSphere Troubleshooting
• VMware KB Article 1008205 (Using ESXTOP to identify storage performance issues)
• VMware KB Article 1003728 (Testing VMkernel network connectivity with the vmkping command)
• VMware KB Article 1020692 (Frequent path state changes indicate storage problem)
• VMware KB Article 1005009 (Analyzing SCSI Reservation conflicts)
• VMware SAN System Design and Deployment Guide
• Duncan Eppings Yellow-Bricks.com Blog

Knowledge

Verify storage configuration

Refer to the vSphere Storage and the SAN System Design and Deployment Guide (not specific to vSphere 5, but worth a read) by VMware. This will cover a lot of areas needed for working with a FC/iSCSI SAN environment with vSphere. Also a good understanding of the hardware you are using on the backend (storage arrays, FC switches, networking, etc) and there “vSphere Best Practices” documents will assist in the proper configuration.

Troubleshoot storage contention issues

When they mention “storage contention” I am taking this as I/O throughput or I/O latency issues. I find the quickest and easiest way of measuring/checking this is via esxtop/resxtop. VMware KB 1008205 and Duncan Eppings esxtop blog post covers this is in more detail.

Metrics to be aware of:

Also see pages 47 thru 50 of the vSphere Troubleshooting documentation for further information.

Troubleshoot storage over-commitment issues

Storage over-commitment can occur in several ways:

• As mentioned above with I/O latency
• Datastore usage (Think Thin Provisioning and over subscription)
• SCSI reservation issues (VMware KB 1005009)
• Path thrashing (VMware KB 1020692)
• LUN queue depth

Troubleshoot iSCSI software initiator configuration issues

Things to look out for:

• Setting up a proper iSCSI Network (pg 74 thru 79 of the vSphere Storage document)
• Configuration of Jumbo Frames (pg 80 of the vSphere Storage document)
• Network connectivity using vmkping (See VMware KB 1003728)

For further information refer to pages 107 thru 112 of the vSphere Storage documentation as well as page 51 of the vSphere Troubleshooting documentation.

Troubleshoot Storage Reports and Storage Maps

See pages 29 thru 32 of the vSphere Monitoring and Performance documentation.

Identify the root cause of a storage issue based on troubleshooting information

The vSphere Troubleshooting document covers several issues that you may run into. See Pages 45 thru 51.

Tools

• vSphere Storage guide
• vSphere Troubleshooting guide

Well I can say they definitely ratcheted up the degree of difficulty. It appears gone are the days of simply memorizing Min/Max configurations (past experience with VCP 3 and 4) and not having any real in-depth hands on experience.  Yes the exam is still made up of multiple choice and  pick the best two or three type questions. However the questions are more situational based, you are having X type of issue or how would you configure Y. Another new twist (and something MS exams have done for years) is present you with a screen capture and base the question/scenario around the information provided. With all that being said I felt like the content in the exam lined up with the VCP 5 Blue Print that VMware provides and that the questions didn’t stray away from it.

Now for what you are probably looking for, the resources I used for studying:

First and foremost is the excellent VMware vSphere 5 training course offered by TrainSignal and put together by David M Davis (blog/twitter) and Elias Khnaser (blog/twitter). The course provides over 30 lessons and totals over 17 hours of material covering all aspects of what you need to know (not just for the exam) on how to setup and configure a vSphere 5 environment. I believe this is a must have.

Secondly Scott Lowes (blog/twitter) newest book Mastering VMware vSphere 5. I wore out my copy of his previous book, Mastering VMware vSphere 4, and will probably do the same with this one. Get it, read it, know it, love it.

Third, and sorry for the self promotion, but my VCP 5 Blue Print study notes. Josh Coen (blog/twitter) and I have been working through each section of the blue print and posting them on our respective blogs. This is covering all the bases on what material the exam is going to test you on. Just click the VCP-5 tab at the top of my blog page to get to the goodness.

And finally, lab time. I am fortunate enough to have a lab at home that I have built with white box ESX hosts, a cisco switch, and two Iomega IX4-200d NAS/iSCSI devices. With that and evaluation versions of vCenter and ESX I was able to get in and play with the new features introduced with vSphere 5 (Storage DRS, Storage Profiles, Web Client, vCSA) as well Distributed Virtual Switches, Storage vMotion, Fault Tolerance, etc. If you can’t dedicate hardware for your lab pickup a copy of VMware Workstation and run virtual ESX hosts and use one of the free open source NAS/iSCSI solutions out there (FreeNAS or Openfiler) to setup a sandbox environment (If you get the TrainSignal video they even cover setting up Openfiler!). It might not be speedy, but it will at least allow you to get your hands dirty.

Hopefully these tips will come in handy for you as they did me and good luck.

Just for some quick background, VAAI or vStorage APIs for Array Integration was first introduced with ESX/ESXi 4.1. It allows for offloading of certain VM and storage functions that would typically take place on the ESX host to the storage array. The benefit being faster completion of these tasks as well as saving compute resources on your ESX hosts. The following list are the offloading features VAAI provides:

Atomic Test & Set – Also known as Hardware Assisted Locking – Allows for granular locking of files on a VMFS volume

Cloning Blocks – Also known as  Full Copy/Extended Copy – Allows the array to make a copy of data (Think cloning and Storage vMotion)

Zero Blocks – Also know as Block Zeroing – Allows the array to zero out blocks (Think creating new VMDKs)

After confirming on the VMware HCL the array matched all requirements it was time to get down to business. After presenting the LUNs to a test host I quickly provisioned a Windows VM on one LUN to be used for the Storage vMotion as well as the clone testing. Also while in vCenter I confirmed with the NetApp Virtual Storage Console that it detected the array and confirmed that it was VAAI capable:

Next step was to fire up ESXTOP, go to the devices screen (u) and then deselect/select appropriate fields (removed B, F, G, I, and added O):

This will present the VAAI Stats counters in the ESXTOP Session. For a full run down of each counter refer to VMware Document 11812, Interpreting esxtop 4.1 statistics. For my testing I was only concerned with CLONE_RD, CLONE_WR, and CLONE_F.

After setting up ESXTOP I switched back over to my VI Client and issued a storage vMotion of my test VM between the new LUNs. I then went back to ESXTOP expecting see the counters climbing as the VM’s storage was moved. However, I got nothing:

Hmm, what’s up with that? So I did some searching and came up with two very helpful items. First a NetApp Technical Report (TR-3886 Understanding and Using vStorage APIs for Array Integration and NetApp Storage) and second VMware KB article 1021976 (vStorage APIs for Array Integration FAQ). Both documents provided excellent information on VAAI as well as few things to check to make sure you have a supported configuration.

I also searched the NetApp forums to see if anyone else had seen a similar issue. Came up empty, so I decided to put a post out there and also notified the twitter-verse looking for additional help. Received a reply from a NetApp employee named Rodrigo Nascimento and below is a culmination of steps outlined in the above documentation as well as things to look for Rodrigo had posted.

First off was to check to see if the new SCSI devices are showing as support for VAAI. This is accomplished either by checking the device in the Storage view of your ESX host or running an esxcli command “esxcli storage core device list | egrep “Display Name:|VAAI Status:”"

As you can see from the above screen shots all seems to be good from that standpoint. I jumped back to the VMware HCL again to double check the requirements. Paying a little closer attention I saw the following:

This got me thinking so I double checked the loaded modules on the ESX host with the following command looking for the vmw_vaaip_netapp module, “esxcli system module list| grep vmw_vaaip_netapp”. The results came up empty.

Not surprising after I look back on it as I just recently presented these LUNs to the host and had not being using NetApp storage before hand, so I quickly put the host in maintenance mode and rebooted.

After rebooting I connected back into the system and re-ran the “esxcli system module list| grep vmw_vaaip_netapp” command:

Bingo! Module is loaded and should be good to go. Fired up ESXTOP again and started my Storage vMotion test. Again nothing.

I went back to the previous mentioned documents as well as my post on the NetApp forum. At this time Rodrigo mentioned to verify if VAAI is enabled on the ESX host. This is quickly done by checking the following Advanced Settings are set to 1 (default) under Configuration -> Software:

DataMover.HardwareAccelerateMove
DataMover.HardwareAcceleratedInit
VMFS3.HardwareAcceleratedLocking

And from the console this completed by verifying that Int Value is set to 1:

esxcli system settings advanced list -o /DataMover/HardwareAcceleratedMove
esxcli system settings advanced list -o /DataMover/HardwareAcceleratedInit
esxcli system settings advanced list -o /VMFS3/HardwareAcceleratedLocking

Low and behold on our test ESX system these were set to 0, thus disabling VAAI. I reset them back to 1 to enable VAAI at the host level and rebooted the system (not sure if it was needed, but why not). After the reboot I again connected to the console and fired up ESXTOP and restarted the Storage vMotion test:

As you can see from the screen shot, finally success! The CLONE_RD and CLONE_WR as well as the MBC_RD/s and MBC_WR/s metrics are displaying activity.

Even though VAAI is  and should be enabled by default on an ESX host researching the issue and reading these documents as well as other posts on the web definitely gave me a better understanding of how it works.

For this objective I used the following resources:

• vCenter Server and Host Management
• vSphere Availability
• vSphere Resource Management
• vSphere Troubleshooting
• Duncan Eppings Yellow-Bricks.com Blog

Knowledge

Identify HA/DRS and vMotion requirements

HA Requirements

• All hosts must be licensed for vSphere HA
• You need at least two hosts in the cluster
• All hosts need to be configured with static IP addresses. If you are using DHCP, you must ensure that the address for each hosts persists across reboots
• There should be at least on management network in common among all hosts and best practices is to have at least two. Management networks differ depending on the version of host you are using.
  • ESX hosts – service console network
  • ESXi hosts earlier than version 4.0 – VMkernel network
  • ESXi hosts version 4.0 and later – VMkernel network with the Management Network checkbox enabled
• To ensure that any virtual machine can run on any host in the cluster, all hosts should have access to the same virtual machine networks and datastores
• For VM Monitoring to work, VMware tools must be installed
• Host certificate checking should be enabled
• vSphere HA supports both IPv4 and IPv6. A cluster that mixes the use of both of the protocol versions, however is more likely to result in a network partition

For further information see page 22 of the vSphere Availability documentation

DRS Requirements

• Shared Storage
  • Storage can be either SAN or NAS
• Shared VMFS volumes
  • Place the disks of all virtual machines on VMFS volumes that are accessible by all hosts
  • Set access mode for the shared VMFS to public
  • Ensure the VMFS volumes on source and destination host use volume names, and all virtual machines use those volume names for specifying the virtual disks
• Processor Compatibility – Processors of both the source and destination host must be of the same vendor (AMD or Intel) and be of the same processor family. This requirement is more for the use of vMotion and allowing a VM to execute its processes from one host to the other. vCenter provides advanced features to make sure that processor compatibility requirements are met:
  • Enhanced vMotion Compatibility (EVC) – You can use EVC to help ensure vMotion compatibility for the hosts in a cluster. EVC ensures that all hosts in a cluster present the same CPU feature set to virtual machines, even if the actual CPUs on the hosts differ. This prevents migration with vMotion from failing due to incompatible CPUs.
  • CPU Compatibility Masks – vCenter Server compares the CPU features available to a virtual machine with the CPU features of the destination host to determine whether to allow or disallow migrations with vMotion. By applying CPU compatibility mask to individual virtual machines, you can hide certain CPU features from the virtual machine and potentially prevent migrations with vMotion from failing due to incompatible CPUs.

For further information see pages 55 thru 56 of the vSphere Resource Management documentation

vMotion Requirements

• The virtual machine configuration file for ESXi hosts must reside on a VMware Virtual Machine File System (VMFS)
• vMotion does not support raw disks or migration of applications clustered using Microsoft Cluster Service (MSCS)
• vMotion requires a private Gigabit Ethernet (minimum) migration network between all of the vMotion enabled managed hosts. When vMotion is enabled on a managed host, configure a unique network identity object for the managed host and connect it to the private migration network
• You cannot use migration with vMotion to migrate a virtual machine that uses a virtual device backed by a device that is not accessible on the destination host
• You cannot use migration with vMotion to migrate a virtual machine that uses a virtual device backed by a device on the client computer

For further information see page 56 of the vSphere Resource Management documentation and pages 119 thru 120 of the vCenter Server and Host Management documentation

Verify vMotion/Storage vMotion configuration

See above sections for DRS and vMotion requirements. Key areas of focus will be proper networking (VMKernel interface for vMotion), CPU compatibility and shared storage access across all hosts.

Verify HA network configuration

• On legcacy ESX hosts in the cluster, vSphere HA communications travel over all networks that are designated as service console networks. VMkernel networks are not used by these hosts for vSphere HA communications
• On ESXi hosts in the cluster, vSphere HA communications, by default, travel over VMkernel networks, except those marked for use with vMotion. If there is only one VMkernel network, vSphere HA shares it with vMotion, if necessary. With ESXi 4.x and ESXi, you must also explicitly enable the Management Network checkbox for vSphere HA to use this network

For further information see page 32 of the vSphere Availability documentation

Verify HA/DRS cluster configuration

Configuration issues and other errors can occur for your cluster or its hosts that adversely affect the proper operation of vSphere HA. You can monitor these errors by looking at the Cluster Operational Status and Configuration Issues screens, which are accessible in the vSphere Client from the vSphere HA section of the cluster’s Summary tab.

For further information see page 31 of the vSphere Availability documentation

Troubleshoot HA capacity issues

To troubleshoot HA capacity issues first be familiar with the three Admission Control Policies:

• Host failures the cluster tolerates (default) – You can configure vSphere HA to tolerate a specified number of host failures. Uses a “slot” size to display cluster capacity
• Percentage of cluster resources reserved as failover spare capacity – You can configure vSphere HA to perform admission control by reserving a specific percentage of cluster CPU and memory resources for recovery from host failure
• Specify failover hosts – You can configure vSphere HA to designate specific hosts as the failover hosts

Things to look out for when troubleshooting HA issues:
• Failed or disconnected hosts
• Over sized VM’s with high CPU/memory reservations. This will affect slot sizes
• Lack of capacity/resources if you using “Specify Failover Hosts”, IE not enough hosts set as failovers

See pages 31 thru 33 of the vSphere Troubleshooting documentation that outlines common failover scenarios for each of the three Admission Control Policies. For further reading on the three admission control policies see page 16 thru 21 of the vSphere Availability documentation.

Troubleshoot HA redundancy issues

Like all other components in a vSphere design, you want design redundancy for a clusters HA network traffic.  You can go about this one of two ways or both. The use of NIC of teaming (two physical NICs preferably connected to separate physical switches) is the most common method used. This will allow either of the two links to fail and still be able to communicate on the the network. The second option is the setup and creation of a secondary management network. This second interface will need to be attached to a different virtual switch as well as a different subnet as the primary network. This will allow for HA traffic to be communicated over both networks.

See pages 33 thru 34 of the vSphere Availability documentation for further reading

Interpret the DRS Resource Distribution Graph and Target/Current Host Load Deviation

The DRS Resource Distribution Chart is used to display both memory and CPU metrics for each host in the cluster. Each resource can be displayed in either a percentage or as a size in mega bytes for memory or mega hertz for CPU. In the chart display each box/section represents a VM running on that host and the resources it is currently consuming. The chart is accessed from the Summary tab at the cluster level under the section for VMware DRS. Click the hyperlink for View Resource Distribution Chart.

The target/current host load deviation is a representation of the balance of resources across the hosts in your cluster. The DRS process runs every 5 minutes and analyzes resource metrics on each host across the cluster. Those metrics are plugged in an equeation:

(VM entitlements)/(Host Capacity)

This value returned is what determines the “Current host load standard deviation”. If this number is higher then the “Target host load standard deviation” your cluster is imbalanced and DRS will make recommendations on which VM’s to migrate to re-balance the cluster.

This is just my basic understanding of how DRS works. For complete down in the weeds explanations I would recommend reading this post as well as this one from Duncan Epping @ Yellow-Bricks.com.

Troubleshoot DRS load imbalance issues

DRS clusters become imbalanced/overcomitted for several reasons:

• A cluster might become overcommitted if a host fails
• A cluster becomes invalid if vCenter Server is unavailable and you power on virtual machines using a vSphere Client connected directly to a host
• A cluster becomes invalid if the user reduces the reservation on a parent resource pool while a virtual machine is in the process of failing over
• If changes are made to hosts or virtual machines using a vSphere Client connected to a host while vCenter Server is unavailable, those changes take effect. When vCenter Server becomes available again, you might find that clusters have turned red or yellow because cluster requirements are not longer met.

See pages 62 thru 66 of the vSphere Resource Management documentation for further information

Troubleshoot vMotion/Storage vMotion migration issues

For vMotion refer to section above for DRS and vMotion requirements. Make sure all requirements are being met.

For Storage vMotion be aware of the following requirements and limitations

• Virtual machine disks must be in persistent mode or be raw device mappings (RDMs). For virtual compatibility mode RDMs, you can migrate the mapping file or convert to thick-provisioned or thin-provisioned disks during migration as long as the destination is not an NFS datastore. If you convert the mapping file, a new virtual disk is created and the contents of the mapped LUN are copied to this disk. For physical compatibility mode RDMs, you can migrate the mapping file only.
• Migration of virtual machines during VMware Tool installation is not supported
• The host on which the virtual machine is running must have a license that includes Storage vMotion
• ESX/ESXi 3.5 hosts must be licensed and configured for vMotion. ESX/ESXi 4.0 and later hosts do not require vMotion configuration in order to perform migration with Storage vMotion
• The host on which the virtual machines is running must have access to both the source and target datastore

For further information read pages 119 thru 122 of the vCenter Server and Host Management documentation

Interpret vMotion Resource Maps

vMotion resource maps provide a visual representation of hosts, datastores, and networks associated with the selected virtual machine.

vMotion resource maps also indicate which hosts in the virtual machine’s cluster or datacenter are compatible, it must meet the following criteria:

• Connect to all the same datastores as the virtual machine
• Connect to all the same networks as the virtual machine
• Have compatible software with the virtual machine
• Have a compatible CPU with the virtual machine

For further information see page 144 of the vCenter Server and Host Management documentation

Identify the root cause of a DRS/HA cluster or migration issues based on troubleshooting information

Use information from above topics to help isolate the issue based on HA/DRS requirements as well pages from the reference documents listed.

Tools

• vSphere Availability guide
• vSphere Resource Management guide
• vSphere Monitoring and Performance guide
• vSPhere Troubleshooting guide

For this objective I used the following resources

• vSphere Resource Management guide
• VMware vSphere Examples and Scenarios guide

Knowledge

List vCenter default utilization alarms

List courtesy of the Alarms tab at the root of vCenter

List vCenter default connectivity alarms

List courtesy of the Alarms tab at the root of vCenter

List possible actions for utilization and connectivity alarms

Host related actions

• Send a notification email
• Send a notification trap
• Run a command
• Enter maintenance mode
• Exit maintenance mode
• Enter standby
• Exit standby
• Reboot host
• Shutdown host

VM related actions

• Send a notification email
• Send a notification trap
• Run a command
• Power on VM
• Power off VM
• Suspend VM
• Reset VM
• Migrate VM
• Reboot guest on VM
• Shutdown guest on VM

Create a vCenter utilization alarm

For this excercise we are going to create a host alarm based on network utilization

1. From the root object in vCenter select the Alarms tab
2. Select the Definitions tab next to View
3. In the right hand, right click in open a space and select New Alarm
4. In the General tab give the alarm a Name, Description, and select Hosts from the Alarm Type drop down.
5. Select the Triggers tab
6. Right click in the open space and select Add Trigger
7. Click the trigger type to display the list of available triggers
8. From the list select Host Network Usage (kbps)
9. For the Warning and Alert values select what is appropriate for your environment
10. Click the Actions tab
11. Right click in the open space and select Add Action
12. Select Send a notification email
13. Under the Configuration column input your email address
14. Leave the default setting to email when going from a warning state to an alert state
15. Click OK

Create a vCenter connectivity alarm

For this exercise we are going to create a host alarm based on lost storage path redundancy

1. From the root object in vCenter select the Alarms tab
2. Select the Definitions tab next to View
3. In the right hand, right click in open a space and select New Alarm
4. In the General tab give the alarm a Name, Description, and select Hosts from the Alarm Type drop down.
5. Select Monitor for specific events occuring on this object, for example, VM powered on
6. Select the Triggers tab
7. Right click in the open space and select Add Trigger
8. Click the trigger type to display the list of available triggers
9. From the list select Lost Storage Path Redundancy
10. Click the Actions tab
11. Right click in the open space and select Add Action
12. Select Send a notification email
13. Under the Configuration column input your email address
14. Leave the default setting to email when going from a warning state to an alert state
15. Click OK

Configure alarm triggers

See sections above Create a vCenter Connectivity Alarm and Create a vCenter Utilization Alarm. Also review pages 33 thru 39 of the vSphere Monitoring and Performance documentation.

Configure alarm actions

See sections above Create a vCenter Connectivity Alarm and Create a vCenter Utilization Alarm. Also review pages 33 thru 39 of the vSphere Monitoring and Performance documentation.

For a give alarm, identify the affected resource in vSphere implementation

If using any of the default vCenter alarms the alarm name as well as the alarm description should identify which resource is being affected.

Tools

• vCenter Server and Host Management guide
• vSphere Resource Management guide
• vSphere Monitoring and Performance guide
• VMware vSphere Examples and Scenarios guide