An uncommon EVC mode situation?

This post about setting EVC mode on cluster-level. It is not about per-VM EVC mode that was introduced in vSphere 6.7.

When you try to select a suitable EVC mode on cluster, it could happen that no mode can be set. The wizard shows a compatibility-error on every mode that because of running VMs the mode cannot be enabled or changed.

[In Text:
Compatibility
The host cannot be admitted to the cluster’s current Enhanced vMotion Compatibility mode. Powered-on or suspended virtual machines on the host may be using CPU features hidden by that mode.]

Sometimes it is not obvious why. In this post I outline a possible situation for this error.

Some facts about CPU features a VM gets

• It depends on your vCenter and host version, which EVC mode can be selected. More details about which mode can be set in which version, read KB 1003212
• When a VM is started on a host in a cluster with EVC mode enabled, VM gets CPU features represented by EVC mode.
• When a VM is started on a single host or on a host in a cluster with EVC mode disabled, VM gets CPU features of host’s CPUs at start-time. This is not limited by version of host in that way that a VM gets just features of max-EVC mode of host. So a VM can get for example Skylake-features when it is started on a 6.5-host with Skylake CPUs.
• Changing EVC mode is not allowed, when it would remove CPU features from a running VM.

Some useful PowerCLI commands

To show support EVC modes on connected vCenter, run:

($global:DefaultVIServer).ExtensionData.Capability.SupportedEVCMode.label

To show max. support EVC modes on ESXi hosts, run:

get-vmhost | select name, @{N="maxEVCmode"; e={$_.ExtensionData.summary.MaxEVCModeKey}}

Important: Shown mode is also limited by vCenter version.

To show min. required EVC mode per VM, run:

Get-VM | select name, @{N="minEVCMode"; e={$_.ExtensionData.summary.runtime.MinRequiredEVCModeKey}}

Important: When no mode is shown, generation of physical CPU of host can be a more current than vCenter or host version knows about. This would prevent setting any EVC mode.

Example

• vCenter and hosts run vSphere version 6.5 – Max. Intel-EVC mode is “Broadwell”.
• Hosts run Intel-Skylake CPUs.
• VMs are started on hosts when they were not part of a cluster or were part of a cluster with EVC disabled. These VMs get Skylake-features.
• When these hosts become a cluster on which you try to enable Broadwell-EVC mode, you will get the mentioned error.
• Commands above will show: no EVC mode for these VMs and “Broadwell” as max. EVC mode for these hosts.

Avoidance respectively solution

To avoid such a situation, always start a cluster with (max.) EVC mode enabled. But even this can be tricky. Image you have a host/version situation like in the example above. Even if your vCenter is your first VM that runs in the new cluster, you cannot enable EVC mode. Consider to deploy vCenter on a host in another cluster or on VMware workstation and move it powered-off into the cluster after setting EVC mode.

Solution for a uniform cluster

In a uniform cluster, all hosts are build of the same hardware. To solve the situation here, every VM must have a min. required EVC mode lower-equal than what is possible with vCenter and ESXi hosts. This could mean you have to upgrade vCenter and hosts. If this could be done, you can set EVC mode.

Solution for a non-uniform cluster

In this cluster different host-hardware is running. This could include different CPU-generations. Without EVC mode enabled, each VM gets its CPU features from its host CPUs. So, running VMs can be migrated between hosts that are capable of at least the same features. When a VM is started for example on a host with Skylake CPUs it cannot be online migrated to host with Broadwell CPUs. Furthermore, max. EVC in this example (Cluster consists of Skylake and Broadwell CPUs) will be Broadwell. Its obvious, EVC is not set-able without shutting down some VMs.

Per-VM EVC mode could defuse the situation. Mode per VM can be set even higher that mode on cluster level. But this is feature was just introduced with 6.7 and VMs must be powered off for configuration. So it cannot solve the problem but can help to avoid it – in a non-uniform cluster.

Performance

But what about decreased performance when operating host CPUs on a lower generation-level? There is a VMware paper that shows there is no large penalty when doing so.

Notes

Another reason for compatibility error can be found here.