Network Virtualization Fundamentals

Introduction

I started developing this material in 2019. I was working at a VMWare cloud service provider delivering as a senior engineer in service delivery building private cloud environments on NSX (at the time, the -V flavor.) We were about 2 years into the NSX-backed platform, and I saw this fusion of networking and virtualization throw both our network engineers and system engineers for a loop as the streams had been irreversibly crossed. We were building on this platform that no-one fully understood. I wanted to put together something to help both our network and systems folks understand the platform and how the stuff they were building worked. The end goal was that our entire team would be able to build and troubleshoot more efficiently. But where do we begin, and where do we go?

I determined that, if after learning the material, someone could answer the question “how do two VMs in different networks on our cloud platforms communicate?”, they probably had a good grasp on things. Working backwards from there, I tried to work my way to a foundation of what you would need to know to answer that question, while ensuring that whether your starting strength was networking or systems/virtualization, we’d get you where you need to go. I had to do a lot of research along the way to fill gaps in my own knowledge, and learned a great deal. The end result was something like 12,000 words of material delivered in a series of well-received classes. I later changed jobs to another provider that provides both VMWare and Nutanix platforms, and thought, “well, gee, maybe I should dust off the training and update it!” NSX was now a very different animal (having undergone a codebase re-write and becoming NSX-T which is now just NSX), and Nutanix handled things its own way via Flow. So, I started researching, learning, and writing. A lot of writing. And when I was done (or the current version of done, I imagine I’ll be revising this continually as NSX and Flow change), someone suggested I should turn it into something I can share with the public as a learning tool (after removing any proprietary secret-sauce stuff.) So I updated it to NSX-T, and added information on Nutanix.

Today, I actually work for Nutanix as a Senior Staff Consultant in their Professional Services team, so I’m continuing to refine the Nutanix information, and that brings us to today! Network Virtualization Fundamentals, everything you need to know to get started with NSX and Flow.

How Things Work

As I said, I figured that if you could answer that question “how do two VMs in different networks on our cloud platforms communicate?”, you were doing alright. So let’s get there:

How do VMs communicate across networks in VMWare NSX and Nutanix Flow?

We’ll learn about differences between the two, but more important, the similarities. But, how do we get there? What else do we need to know? When we’re done, we’ll know enough to understand how VM communication works in both VMware NSX and Nutanix Flow. So, as Sam Reich says on Game Changer, The only way to learn is by playing, the only way to win is by learning, and the only way to begin is by beginning. So let’s begin. The questions we’re going to need to answer to get where we want to go are:

When we’re done, we’ll look at some example architectures for each. These are in progress right now, and are Coming Soon(TM).

Works Cited

Naturally, much of this information was sourced from places far and wide on the internet. As much as possible, I’ll include specific sourcing. For instance, I relied very heavily on Wikipedia while writing this, including but not limited to, the articles on Ethernet, Ethernet Framing, IPv4 Packets, OSI Model (and the individual pages for the layers), Geneve, VXLAN, and more.

I also sourced much of the NSX-T specific material from the NSX Reference Design Guide. This DID have an online version, but Broadcom has very recently locked that behind a Broadcom account. The PDF version can be found here: https://www.vmware.com/docs/nsx-t-reference-design-guide-3-2-v1.1-1

Nutanix information was sourced from the Nutanix Cloud Bible as well as the official Nutanix documentation for Flow Virtual Networking. Also shoutouts to some folks in the r/Nutanix Discord who patiently answered my questions and provided a lot of good information and insight when I was a client.

More will be added here as they come up while posting the material.