Mitigate Spectre and Meltdown impact with vSphere ESXi

As most of the folks in the IT, I’ve spent last couple of days researching about famous Spectre and Meltdown attacks and their possible impact to our infrastructure.

These security flaws are especially bad, not only because they’ve been here for more than 2 decades, but mitigation comes with significant performance impacts.

Where to start?

As these attacks have to be executed locally, I would recommend anyone to focus on systems which have internet access and systems which are shared among multiple users, especially:

• Multi tenant environments – if you are a hosting company
• Terminal servers
• Workstations/Desktops (Both physical and virtual)
• Mobile devices – Phone/tablets

Although most of the CPU vendors and their CPUs are affected to some extend, I will be focusing on Intel CPUs here, as those are most vulnerable.

OS patching is not enough!

Almost every OS vendor came out with their patches, however what may not be clear for an average user is, If you want to be fully protected against known exploits you have to also update your CPU microcode which is usually coming together with uEFI/BIOS updates from your hardware vendors.

If you have slow HW vendor or they do not support your motherboard anymore, it doesn’t mean you are entirely doomed yet. There is a way how to sideload CPU microcode during the OS boot. For Linux you can find instructions and microcode at Intel’s website directly. In case of Microsoft, occasionally in case of critical issues they do provide CPU microcode updates via Windows Update, although I don’t know for sure when or if they will decide to release it this time, anyway in such issue I would bet yes.

There are 3 types of attack:

• CVE-2017-5753 (variant #1/Spectre) Bounds-checking exploit during branching
• CVE-2017-5715 (variant #2/Spectre) Indirect branching poisoning attack
• CVE-2017-5754 (variant #3/Meltdown) Speculative cache loading

OS(kernel) patches alone will protect you against Variant #1 and Variant #3, however protection against Variant #2 requires CPU microcode update as well.

Protection in Virtualized environments

When it comes to virtualization, it is a bit trickier as there is an additional layer included.

By updating CPU microcode to fix Spectre you are presenting additional CPU features to your operating system, however this is not the case if you are running in virtual machine as those features may be masked by hypervisor, especially in this case as it is very fresh. You also need to patch your hypervisor, so it will present those features to the OS in virtual machines as well.

VMware released new security advisory VMSA-2018-0004 together with patches and  kb52085 yesterday regarding enabling this capability.

What I like especially, that they included also CPU microcode with the new patches, therefore you don’t have to wait for HW vendor to release uEFI updates. Although updating uEFI is still strongly recommended!

How to protect against Spectre #2  with minimizing performance hit

1. update your vCenter to following versions
   1. 6.5 U1e
   2. 6.0 U3d
   3. 5.5 U3g
2. update ESXi and CPU microcode
   1. ESXi650-201801401-BG hypervisor
   2. ESXi600-201801401-BG hypervisor
   3. ESXi550-201801401-BG hypervisor and microcode
   4. ESXi650-201801402-BG microcode
   5. ESXi600-201801402-BG microcode
3. Update VM Hardware version to at least version 9 (to be protected), ideally version 11 to minimize performance impact in case your CPU supports PCID and INVPCID (Process-Context Identifiers).
4. Remember you still have to patch your guest operating system

vCenter patches will, make sure EVC version in cluster will be updated to the latest version only after you update all hosts within the cluster

Note: ESXi will never override a newer version of the microcode patch provided by BIOS nor will the BIOS with an older version prevent ESXi from applying the newer version.

I recommend you following articles to read more about handling the issue in Red Hat and Microsoft’s Operating systems:

Controlling the Performance Impact of Microcode and Security Patches for CVE-2017-5754 CVE-2017-5715 and CVE-2017-5753 using Red Hat Enterprise Linux Tunables

Speculative Execution Exploit Performance Impacts 

Windows Client Guidance for IT Pros to protect against speculative execution side-channel vulnerabilities

Windows Server guidance to protect against speculative execution side-channel vulnerabilities

Understanding the performance impact of Spectre and Meltdown mitigations on Windows Systems

The following two tabs change content below.

• Bio
• Latest Posts

Dusan Tekeljak

Dusan has over 8 years experience in the Virtualization field. Currently working as Senior VMware plarform Architect at one of the biggest retail bank in Slovakia. He has background in closely related technologies including server operating systems, networking and storage. Used to be a member of VMware Center of Excellence at IBM, co-author of several Redpapers. His main scope of work consists from designing and performance optimization of business critical virtualized solutions on vSphere, including, but not limited to Oracle WebLogic, MSSQL and others. He holds several IT industry leading certifications like VCAP-DCD, VCAP-DCA, MCITP and the others. Honored with #vExpert2015-2018 awards by VMware for his contribution to the community. Opinions are my own!

Latest posts by Dusan Tekeljak (see all)

• ESXi 6.7 U1 fixes: APD and VMCP is not triggered even when no paths can service I/Os - November 30, 2018
• Update manager error: hosts could not enter maintenance mode - November 19, 2018
• VMware fixes 2 data corruption bugs and VM to Host escape vulnerability! - November 19, 2018