When building out a rocking cloud over this past weekend, the topic of networking specific to NFS on vSphere came up. One of the questions that was asked was if the "Management traffic" checkbox on a vmkernel needed to be ticked in order of that specifc vmkernel to communicate to the NFS export. I didn't believe this to be the case, and after some quick testing, we confirmed its as not needed.

The next question that came up is an important one to consider in your implementations of NFS on vSphere-- and it was the question of "which vmkernel will be used to communicate to the NFS export?". It's important to keep in mind that once the connection to an NFS mount is made, it will continue over the same vmkernel AND NIC that was used to establish the connection, so if you get it wrong the first time, it'll stay that way until you break the connection and re-establish it (which in our case, we tested by unmounting the export, making our changes, and then remounting).

What I had learned a ways back that I believed to still be true was that the the IP of the NFS server would determine which vmkernel would be used by using the vmkernel's route table, and if no specific route (or same subnet vmkernel interface), the default route/gateway would be used.

What we tested first was simply mounting the NFS export with a relatively "out of the box" ESXi host networking configuration. To the experienced vSphere administrator or architect this is obviously not something you'd do yourselves, but you'd be surprised at how far people take the "NFS is so easy" thought to their implementations, and if they can simply mount it without doing "anything special", a lot of the time they do.

In our configuration, the NFS server was on a different subnet, and so nothing in the route table matched the desired destination network, and the default gateway was used. This was also the vmkernel that was being used for Management traffic, and made for a non-optimal flow of traffic, especially for storage. To confirm our suspicions, we created a new vmkernel interface on the same subnet as the NFS server, and unmounted the NFS export. Then we mounted it again, and as expected, it used the new vmkernel on the same subnet (via a lookup to the route table) for the communication, and operated as desired.

A lot of reference architectures are being made around the use of NFS, and making sure your network is designed properly is critical to the success of whatever you stack on that file system!