Updated: Building vSphere 6 with VSAN on IBM Softlayer: Part 4 – Configuring VSAN on Softlayer

UPDATED(24.02.2016): Driver version changed, additional details added.

UPDATED(12.07.2016): Driver changed. VMware KB added

Updated(28.07.2016): Softlayer automated Firmware Update method added.

Update(03.02.2017): It looks like new servers ordered from Softlayer come with proper megaraid driver preinstalled. Please check if that’s the case for you, before making changes.

Update(26.04.2017): If you are an IBMer building a VSAN on IBM Cloud for your customer, you can find official GTS VSAN Cookbook using this IBM internal link. You can also reach me or Jack Cherkas over Sametime for more details.

Welcome back. In this part we will discuss specific steps to have VSAN configured on servers we’ve ordered in previous articles.

What we have is:

  1. 3 ESXi hosts
  2. 1 vCenter server
  3. (optionally) a dedicated VLAN with portable subnet for VSAN

The approach I will be describing is the one which was tested by me and some of my colleges and proved to be working.

Here is the summary of what we will do:

  1. Update driver of Storage controller to the latest version. (On all hosts one by one.),
  2. Check and update Firmware of storage controller to the latest version.(On all hosts one by one.),
  3. Reconfigure Disk subsystem to suit our needs.
  4. Create networking on vSphere level for VSAN,
  5. Configure VSAN.

Note: the RAID controller used in this example is MegaRAID SAS 9361-8i (aka LSI LSI00451 on Softlayer portal). As far as I was told, this is the default controller in Softlayer and the one which is supported by VMware.

Updating driver of MegaRAID SAS 9361-8i on ESXi 6

First we need to download the driver. I will include link to the latest supported(according to my info from VMware) driver, for today, but please make sure you check which one is latest and supported on the day you are performing configuration.

So the driver i was recommended to use can be downloaded from this link. Use this one unless you have more up-to-date info from VMware.

Just in case you need, here is how you can find the latest driver for vSphere 6 is to first go to the official website of the raid controller here. There, under Downloads, download what you find as a latest driver for vSphere 6, but it is actually text file which contains link to VMware website with latest supported driver.

Download the ZIP from here and unpack it. We will need the .vib file form inside. In my case its called scsi-megaraid-sas_6.610.15.00-1OEM.600.0.0.2494585.vib

Thing is that the driver bundled with ESXi is the so called Native driver Invader only one, and we need to install the so called Legacy driver. This will require an additional step after the installation of the .vib.

The process is pretty straight forward, like any other VIB:
1. Upload the file to the local datastore of ESXI host,
2. Put host to Maintenance mode,
3. Login to the ESXi host using SSH,
4. Execute the following command.

5. After you have done this. you need to disable the old lsi_mr3 driver. To do that, run:

6. Now reboot your host.

After reboot you can check if lsi_mr3 driver was replaced by megaraid driver by running:

Installing storcli command line tool on ESXi 6

As next step we will install storcli tool from Avago, this tool will help to configure disks on the controller. It can also be used to update firmware of the controller.

First we need to download it. You can find it on the same Download page on Avago website under Latest MegaRAID StorCLI.

Download the ZIP, unpack it and open the next ZIP inside. What we need is the VIB file from Vmware-NDS folder. In my case its called vmware-esx-storcli-1.19.04.vib.

Now, lets install it:
1. Upload the file to the local datastore of ESXI host,
2. Make sure ESXi is still in Maintenance mode,
3. Login to the ESXi host using SSH,
4. Execute the following command.

where DATASTORENAME is the name of your datastore.

Now you need to reboot your host to proceed with next steps:

 

Updating firmware of MegaRAID SAS 9361-8i on ESXi 6

There are two methods you can use to upgrade your firmware. You can either ask Softlayer to do it for you, or you can do it yourself.

Method 1

To request Softlayer to perform a firmware upgrade you will need to login to SoftLayer Portal, Navigate to server you need Firmware updated on. On the right side of the page you will see an Actions dropdown menu, click it and select Update Firmware.

UpdateFW1

 

You will be presented with some options of what to Update.

UpdateFW2

Select Raid Controller, or if you want you can select all to make sure all your FW is up to date.

This will send a request to Softlayer to perform firmware update, it can take up to one hour, and of course your server will be restarted.

Method 2

 

Second method is manual. First of all lets download the firmware file first. Once again visit Download page on Avago website in the Firmware section. We need the Latest MegaRAID Firmware. Check the ReadMe file first, it contains the actual version information. Looks for word Firmware in the file. Here is what I have there :

As a next step, we need to check the Firmware version of the controller installed on the server. to do that we will use storcli utility we just installed. Here is the command you can use to do it fast.

As you can see, in my case Firmware is not up to date, and we need update it.

Download the Firmware Zip file from Avago website, unpack it and upload to .rom file from inside to the datastore of your ESXi host. In my case the file is called mr3108fw.rom.

Once done, connect to your ESXi host with SSH. Execute the following command to flash the firmware:

where DATASTORENAME is the name of your datastore.

and reboot the host once again:

 

Recreating RAID groups using storcli

Now, as I mentioned in Part 1 of these series, the way Softlayer creates RAID groups is not best for using with VSAN. I am sure this issue will be fixed with time, but for now, lets recreate those manually.

First of all we need to check what we have. we can do that using storcli.

Command will show a lot of output, here is part of output i got on my system:

As you can see in VD LIST, Softlayer created RAID0 per disk and the cache mode which is used is RWBD (read ahead always, write back, direct). we need to recreate those VDs with RWTD (read ahead always, write through, direct) cache mode.

A colleague of mine created a simple script to do this. You can use it, or you can run each command one by one:

Note: Make sure you use correct drive number for disks. you can see those in PD LIST part of the output we saw before.

Once done run the /opt/lsi/storcli/storcli /c0 show to compare the output. Here is how it looks for me after executing the script:

Now we are ready to proceed with the remaining of the configuration.

Configuring Multicast for VSAN on Softlayer

Note: the following config should be done after VMkernel ports  for VSAN are already created.

We all know that VSAN requires multicast and there is a default configuration set by VMware. The t Default configuration are not supported in Softlayer. Softlayer supports the following multicast range 224.0.0.115-224.0.0.250 in their private network. So to work with this we need to change the multicast IPs used by VSAN.

Note: Softlayer multicast range 224.0.0.115-224.0.0.250 is per VLAN, which allows using of same address range in different VLANs. This also means that if you will configure another VSAN cluster in same Private VLAN, multicast addresses will need to be changed.

The procedure  is quite simple and is described in VMware KB2075451

But to make it faster here is the command which you can run:

where VMK is the name of vmkernel port used by VSAN( for example vmk1 or vmk2). Run it on each ESXi.

To check if it’s OK, run:

 

That’s it for the controller and multicast part, now we are ready to login to vCenter and proceed with remaining configuration.

Configuring VSAN on vSphere 6 running on Softlayer.

On vSphere level, there is not much difference between configuring VSAN on Softlayer and configuring VSAN on whatever else.

Here are some Key point you need to remeber:
1. Create Separate Vmkernel portgroup for VSAN, make sure Virtual SAN traffic is selected.
VSAN_TRAF
2. In case your SSD is not identified by SSD in vSphere, select it in Storage Devices view and click the magic F icon.
VSANSSD

The remaining configuration is as usual.

Small Update:

VMware recommends implementing the following KB, so take a look

That’s it for this series, I hope you enjoyed it, feel free to leave any comments or questions.

The following two tabs change content below.
Aram Avetisyan is an IT specialist with more than 15 years experience. He has rich background in various IT related fields like Cloud, Virtualization, SDN, Disaster Recovery, and so on. He holds several industry level certifications including but not limited to VCIX6-DCV, VCIX6-NV, VCAP-CIA. As VCI Aram is delivering VMware authorised courses. For his contribution to the community, he was recognised as vEXPERT in years 2014, 2015 and 2016.

About Aram Avetisyan

Aram Avetisyan is an IT specialist with more than 15 years experience. He has rich background in various IT related fields like Cloud, Virtualization, SDN, Disaster Recovery, and so on. He holds several industry level certifications including but not limited to VCIX6-DCV, VCIX6-NV, VCAP-CIA. As VCI Aram is delivering VMware authorised courses. For his contribution to the community, he was recognised as vEXPERT in years 2014, 2015 and 2016.
Bookmark the permalink.

12 Comments

  1. Pingback: Building vSphere 6 with VSAN on IBM Softlayer: Part 3 – Configuring Networking on Softlayer - The Virtualist

  2. Hello

    There is a new version of MegaRAID SAS 9361-8i driver. New name is identical with embedded driver:
    lsi-mr3-6.610.17.00-1OEM.600.0.0.2768847.x86_64.vib

    Instead of installing new driver you need to update it:
    “esxcli software vib update -v /vmfs/volumes/datastore1/lsi-mr3-6.610.17.00-1OEM.600.0.0.2768847.x86_64.vib”

    And now do NOT disable lsi_mr3, otherwise you loose disks after reboot.

  3. Pingback: Building vSphere 6 with VSAN on IBM Softlayer: Part 1 - Ordering ESXi hosts on Softlayer. - The Virtualist

  4. Pingback: Replacing driver for MegaRAID SAS 9361-8i on ESXi 6 - The Virtualist

  5. Hey Aram, You need the “.vib” on the command “esxcli software vib install -v=”/vmfs/volumes/DATASTORENAME/vmware-esx-storcli-1.19.04″ –no-sig-check”

  6. Hi Aram,

    Will there be an update for VSAN 6.6 anytime soon?

    • Hi Niels,
      I assume you mean update to the blog post.
      Yes, I am planning to update the whole series of articles, as soon as vSphere 6.5 appears on Softlayer catalogue. That will, of course, include latest version of VSAN, and will also cover All-Flash deployment.

  7. Hi Aram,

    Didn’t notice your reply, currently the 6.5 ESXi build is available on Softlayer, I have my doubts about driver versions and firmware version, would be great if you can take a look.

Comments are closed