Possibility of cross vCenter vMotion is quite popular among customers. But the thing is that most of the times the vCenters, or the environments managed by those vCenters, are in different sites or locations. This simply means they do not share same vMotion network. Which means vMotion traffic needs to be routed, which means vMotion enabled vmkernel interface should reach some gateway.
With latest trend of Hybrid and Private on Public clouds, I started to get more and more question on how to solve this situation. Should customers run vMotion on Management Network, so that it can reach default gateway? Or a static route is required on each host which will send vMotion traffic to certain gateway?
Although both the above mentioned methods may work, answer to the questions is ‘there is better way to do it’.
Surprisingly not a lot of people know that starting from vSphere 5.5 there is a feature called TCP/IP stacks in vSphere. Yes, you can configure several different TCP/IP stack on your ESXi, and each of those will have its own set of Gateway and DNS configuration. After that, during vmkernel interface creation, you can chose which TCP/IP stack to use. This is quite well documented in VMware’s official documentation, but I will still show a brief how-to.
In vSphere 6 there are several TCPIP stack created by default. And you can create custom ones using command line.
Default ones are:
- Default – default TCP/IP stack used my management network
- Provisioning – used for Provisioning traffic
- vMotion – used for vMotion traffic
If you are an NSX user you will notice that a custom stack named vxlan is created during vTEP creation.
Here is where you can find those:
In Web Client, click on your host and navigate to Manage>Network>TCP/IP configuration. There you will see list of TCP/IP stacks. Select the one you need to edit and click the small Pencil icon.
In edit mode you can configure Gateway and DNS servers for the given TCP/IP stack.
Once you have it ready, select proper TCP IP stack during vmkernel port creations like shown on screenshot.
Note: Creating vmkernel interface with vMotion TCP/IP stack will disable vMotion on all other vmkernel ports which are using Default TCP/IP stack.
Nice and elegant solution, isn’t it?
Now, first time I saw this, my first question was: “Can I create my own TCP/IP stack, for example for iSCSI traffic, or secondary Management interface?” The answer is Yes and No. Yes, because you can create custom TCP/IP stack, but No, you cannot create TCP/IP stack for management network. Actually you cannot set any specific feature when creating TCP/IP stack, you cannot even create second TCP/IP stack for vMotion.
But for iSCSI yes, you can. This can be especially useful for Softlayer users, for those who don’t want their iSCSI traffic to use same VLAN with management network. UPDATE: 23.02.2017 – There are limitations though, and, based on comments from readers bellow, it looks like custom TCP/IP stacks for iSCSi are not supported by VMware. So you may try it, but I would not recommend using this for production.
To create Custom TCP/IP stack, get to the command line interface of your ESXi, either by SSH or direct console. Execute the following command:
esxcli network ip netstack add -N="NameOfStack"
Then return to your web client and edit the setting of the TCP/IP stack as it was shown above.
That’s it, as simple as that. Hope it helps. Let me know if you have any questions or comments.
Latest posts by Aram Avetisyan (see all)
- Configure Proxy settings for VSAN Health-Check Plugin(CLI) - August 11, 2016
- Replacing driver for MegaRAID SAS 9361-8i on ESXi 6 - July 12, 2016
- Remove and Re-Create Management Network (vmk0) VMkernel interface using ESXi Command line - July 6, 2016