Installing and Configuring the NX-OSv 9000 in ESXi

A site I work with uses Nexus 9000 series switches in their datacentre. I don’t really have much operational experience with NX-OS so was very happy to recently discover that Cisco has a freely downloadable virtual version of the Nexus 9000 software. With Cisco’s long term strategy seeming to be pushing the 9K’s into the access layer I feel it is wise to get up to speed on the capabilities and nuances of this OS.

Though it does not implement the full set of features, the interface throughputs are rate limited to 4Mb, and there is a limit of 64 interfaces, it has enough to make it very useful for modelling and testing various configurations before they are pushed into a production environment (’cause who has a couple of 9K’s just sitting around to play with?).

Though the setup process was fairly straight forward there were a few gotchas so this post will detail the process I used to deploy the NX-OSv 9000 into my home ESXi based lab.

First step is to obtain the software from the Cisco site. You need a Cisco login to access the download however there are no additional entitlements required.

There are two options for ESXi, an OVA and a VMDK.

The OVA option is the simplest so that is what this post will be based upon.

I am going to use the vSphere thick client for the various ESXi operations, this is very naughty as we have been told many times by VMWare that this will be removed in a future release and many new features have not been implemented in the thick client.

Meh!

When the HTML5 client is as fast and reliable as the thick client I will use it exclusively, however until that day I only use it when access to a particular feature requires it.

Deploy the OVA as you normally would, at the Network Mapping screen you will see a warning regarding having multiple source networks mapped to the same host network.

The first network adapter defined in the VM will be mapped to the NX-9000v management interface mgmt0. Subsequent VM network adapters will be sequentially mapped to NX-9000v data interface ports Ethernet1/1, Ethernet1/2, Ethernet1/3 etc. The management interface MUST be kept separated from the data interface ports and you must be very careful mapping NX-9000v data ports to vSwitches that house other hosts or connect to physical switching infrastructure as they will participate in STP, generating BPDU packets and forwarding broadcast and multicast traffic. Could get messy…..

At this stage just make sure you change the mapping for mgmt0 to an appropriate Destination Network that will be accessible via SSH later on for configuration.

 

Make sure the “Power on after deployment” option is NOT ticked at the last page and kick off the deployment.

When the deployment is completed, go into the configuration tab of the ESXi host and add a new vSwitch that is not connected to any physical adapters. Make sure you set Promiscuous mode to Accept and set the network to accept all VLANS.

Now edit the properties of our newly deployed NX-9000v and configure Network Adapters 2-6 to connect to the new vSwitch we have just created. This will ensure we don’t cause any problems upstream when we power on. Down the track I will create a vSwitch for each NX-9000v port that I wish to use and they will essentially become virtual cables, if I want to connect a virtual host or another instance of NX-9000v I just connect them to the appropriate vSwitch.

Now for the first gotcha.

The Cisco literature details creating a serial port that is mapped to an ESXi host network address and port pair to allow connection to the console port of the NX-9000v for initial configuration. Problem is that this feature only works if you have ESXi Enterprise licenses. Don’t know too many people who have Enterprise licenses in their home lab, and to be honest don’t know of too many small to medium sized businesses that have Enterprise licenses either!

To work around this you will need a Windows VM running on the same ESXi host as your new NX-9000v. What we will be doing is mapping the serial port to a named pipe that will allow us to get to the console port of the NX-9000v using a terminal emulator on the Windows VM.

Add a Serial Port to the NX-9000v VM and set the media to named pipe.

Configure with the following settings:

  • Pipe Name: \\.\pipe\VCom1
  • Near End: Server
  • Far End: A virtual machine
  • Connect at power on: Ticked
  • Yield CPU on poll: Ticked

Now we need to add a named pipe Serial port to the Windows VM.

This time we configure with the following settings:

  • Pipe Name: \\.\pipe\VCom1
  • Near End: Client
  • Far End: A virtual machine
  • Connect at power on: Ticked
  • Yield CPU on poll: Ticked

Power on the NX-9000v and Windows VM’s.

Log into the Windows box and fire up your terminal emulator. I like to use Putty.

Connect to COM1, hit Enter a few times and you should see the console of the NX-9000V complaining about a POAP failure.

Hit Enter and answer Y to abort POAP and start the setup process.

The default options should be fine, make sure you enter an IP address that is appropriate for the network that you have mapped the management port mgmt0 to.

The other gotcha.

When you have finished the initial setup process, you need to configure the boot image. If you don’t do this, when you restart the NX-9000V it will stop at the loader> prompt and will not boot.

Log into the switch and enter the following command:

dir bootflash:

In the output, look for a file called nxos.<something>.bin, this will vary depending on which version of NX-OSv 9000 you downloaded.

Mine is nxos.7.0.3.I6.1.bin

Enter the following commands, substituting nxos.<something>.bin with your version of NX-OSv 9000

configure terminal

boot nxos nxos.<something>.bin

end

copy run start

You should now be able to access the NX-9000V via SSH from the comfort of your management PC.

Job done!

10 Comments on "Installing and Configuring the NX-OSv 9000 in ESXi"


    1. I haven’t ever done it and there would be a few ways to do it (depending on your definition of connect) but at the simplest level I guess you could dedicate a NIC on the ESXi host to a vSwitch, connect that NIC to your 5K and then connect a port on the 9000v to the vSwitch. It would be as if you had connected them with a “dumb” switch in between. You may have trouble with some layer 2 things like LACP and BFD, I would be curious to hear your results!

      Reply

  1. I’ve been battling with this for days now.. I can’t even get the 9K to boot. I have the essentials licence on my esxi lab and running vcenter to deploy the OVA. This is being deployed onto a licenced 6.5 host. I don’t have much experience with esxi, however im pretty good a networking (well I think so!)…Anyway when I boot the 9K by using the OVA or VMDK file (converted) the VM just tries to DHCP boot, this generally means I’ve screwed the virtual drive up but I’ve gone back over this multiple times.

    What would you try next? Any idea where i’m going wrong? Even though the Serial port isn’t mapped the box will still boot from everything i’ve read, just just wont get to the loader prompt. Any help would be appreciated!!!!

    Reply

    1. Hi Rich

      I haven’t run a 9000v up in an ESXi 6.5 environment before but I would check that the download hasn’t been corrupted by confirming the checksum of the OVA and also check that the Firmware Boot Option on the VM is set to EFI.

      Regards

      Kirin

      Reply

    2. Hi Rich,

      I was able to get this loaded up onto my standalone evaluation mode ESXi 6.5 box today using the vSphere client version 6 (not vCenter). The VM Hardware version I used was 8, not 13 which is default for 6.5. Maybe choose different hardware version?

      Also, the ova I deployed was “nxosv-final.7.0.3.I7.2.ova”. The console messages I saw on load were a bunch of “intird” loading statements. It ended that process with “Leaving grub land” and then settling on an “image hash:” prompt.

      I failed the first time because I was impatient and did not allow the image to load completely before I was trying to hit enter in the console window and get to a prompt of some kind. It took a good 10 minutes to load (or more) on my box (Xeon E3-1225 v3 w/32GB RAM) but once done, I was able to power down, set up the serial connections and use that to complete the config after powering it on again.

      Hope this helps.

      Fritz

      Reply

    1. Hi Jose

      I was never alerted to any license violations during install or enabling of various features.
      Don’t quote me on this but I don’t believe licenses are required as the NX-OSv 9000’s are not designed for production workloads.
      The data plane through put is capped at 4Mbps and there is a limit of 64 interfaces. The intended use is for proof of concept modelling, testing and education.

      Regards

      Kirin

      Reply

  2. Hello Kirin, found your article when battling the network serial console thing. You state:

    > Problem is that this feature only works if you have ESXi Enterprise licenses.

    and it seems to be so, but what’s your source? I can’t find anything anywhere, only you are telling it and the feature’s not working for me. I am asking to get better at vmware…
    I worked around the probably same as you, named pipe to a mgmt linux VM and minicom from there.

    Also for future readers, ESXI 6.7 and nxos-9.3.1 have some problem and won’t start.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *