A couple of months ago, after a migration from our legacy vSphere environment into our brand new vSphere based on IBM/Lenovo Flex Systems environment. Some of the our VMs stopped or were randomly responding on the network. My networking team discovered those VMs have duplicate MAC addressees with another VMs.
You would assume that is the case, however all of those systems were in different VLANs, therefore there should be no problem based on the basic VLAN isolation concepts(although you don’t want to have duplicate MAC addresses in your environment anyway – maybe except load balancing).
Root cause of this were our IBM SI4093 System Interconnects modules (Similar to Cisco FEXes). Those “switches” actually does not have Easy Connect mode. What is worse in their case it is their default mode of operation or so called Transparent (or VLAN-agnostic) mode.
Transparent (or VLAN-agnostic) mode.
In VLAN-agnostic mode (default configuration), the SI4093 transparently forwards VLAN tagged frames without filtering on the customer VLAN tag, providing an end host view to the upstream network. The interconnect module provides traffic consolidation in the chassis to minimize TOR port utilization, and it also enables compute node to compute node communication for optimum performance (for example, vMotion). It can be connected to the FCoE transit switch or FCoE gateway (FC Forwarder) device. https://lenovopress.com/tips1045
Easy Connect mode
provides transparent PureSystems connectivity to your existing Cisco, Juniper, or other vendor network. With Easy Connect enabled on the EN4093/R, CN4093, or G8264 switches, the core network sees a “big pipe” for compute traffic coming to and from the PureSystems chassis. The switch becomes a simple I/O module that connects servers and storage with the core network. It aggregates compute node ports, and behaves similarly to Cisco Fabric Extension (FEX) by appearing as a “dumb” device to the upstream network, with the main difference being that intra-chassis switching is supported. Unlike Cisco FEX, traffic does not have to be sent upstream if the network destination is housed in the same physical chassis. https://lenovopress.com/sg248089
It is not necessary a problem that switch forwards all frames regardless of VLAN, as you can handle it at the OS – Hypervisor level easily.
So why there is an issue with Easy Connect and Transparent modes?
In these modes switches MAC address table consist only from MAC address and Port where to send traffic to, but no VLAN information. Therefore if you have same MAC address in different VLANs, switch will learn its port based on transmission of the packets coming from it and will sent traffic to one port only – the one which is currently learned, not to all of them, as mentioned I wouldn’t mind, if it would be sending it to all ports with MAC address present and let the hypervisor handle it.
I’m also wondering if Microsoft Network Load Balancing in Unicast mode will work with this too…?
I contacted support believing it is a bug and hoping they will fix it, however they claim it is actually by design and recommended us to switch our SI4093 to VLAN aware mode (What we eventually did).
Unfortunately SI4093 has limitation for 250 active VLANs only and we are pretty close to reach it. Guess there might be an opportunity to leverage VMware NSX in our environment or buying the new switches into our Flex Chassis.
So what is the security issue here?
You can quite easily create DoS attack to a system in different VLAN. The only thing which you need to do is to change your MAC address and start transmitting so the switch will learn your port and start forwarding traffic to you, in case of the ESXi, it will most probably simple drop it at vSwitch, but I do believe that is still called Denial-of-Service.
To me fix by the vendor looks is pretty, just to add another column into they MAC address table with VLAN information also for Easy Connect or if you want Transparent VLAN mode. Doesn’t matter it forwards all frames, we already now we should handle that at the OS level. But current situation is, it actually not forwarding all of them.
What can I say? Pure ignorance by the vendor selling this to banks, hospitals, telecommunications, public sectors. I really doubt it can pass any security audit, which indeed you need in those….
Which devices are affected?
I believe all IBM/Lenovo Interconnect modules in default transparent mode and switches in Easy Connect mode:
SI4091 10Gb System Interconnect Module – Transparent Mode
SI4093 System Interconnect Module – Transparent Mode
EN4091 10Gb Ethernet Pass-thru – Easy Connect mode
EN4093 10Gb Scalable Switch – Easy Connect mode
EN4093R 10Gb Scalable Switch – Easy Connect mode
CN4093 10Gb Converged Scalable Switch – Easy Connect mode
IBM System Networking RackSwitch G8264CS – Easy Connect mode
IBM RackSwitch G8264 or G8124E – Easy Connect mode
IBM RackSwitch G8214 – Easy Connect mode
How can you fight?
- Optimally switch to VLAN aware mode whenever possible.
- To prevent users or attackers to change their MAC address in VMware environment always set MAC Address Changes and Forget Transmits to Reject in your virtual switches security settings
- Make sure you don’t have duplicate MAC addresses in your environment. One real life scenario when it can occur without any attack is if you have two vCenters in your environment with same ID. We had different, but it happened anyway, because there were previous vCenter upgrades and migrations which I was not aware off. By the way you can find a very nice article regarding considerations when migrating VMs between vCenter Servers by William Lam. vSphere 6.0 cross vCenter vMotion will take care of MAC address blacklisting for you 😉
Latest posts by Dusan Tekeljak (see all)
- ESXi 6.7 U1 fixes: APD and VMCP is not triggered even when no paths can service I/Os - November 30, 2018
- Update manager error: hosts could not enter maintenance mode - November 19, 2018
- VMware fixes 2 data corruption bugs and VM to Host escape vulnerability! - November 19, 2018
You are right when working in Easy Connect -Transparent Mode, then it works like an intelligent pass-thru device, with the exception of local switching between internal ports.
This is one “configuration” you can use with SI4093 and the other IBM/Lenovo switches. I consider it as an additional option between the all the other “mode of working” of this switches, not the only one or “always” the preferred.
In secure environment I also discourage the implementation of Easy Connect -Transparent Mode, I apply the more “declared” configuration(I mean, I configure all the parameters).