Receive Packet Errors with VDS Health Check and Jumbo Frames

By | May 22, 2015

My team setup a new VDS in a greenfield site with 9000 MTU (at the VDS level) since the new location was jumbo frame enabled. Management, VMotion, NFS, and FT vmkernels were set to 9000 MTU and there were some guest distributed port groups. We ended up dropping the MTU for the Management vmkernel to 1500 when we found some routers in that location that did not support 9000. We enabled VDS health check for VLAN/MTU to verify that we were getting the appropriate MTU and VLANs to our hosts.

The linux team had a vlan that was being used by physical servers as well as VMs. On the physical servers they had configured their nic for 1500 MTU.

For some reason, they started to see excessive Rx Errors
RX packets:74975060 errors:254276 dropped:0 overruns:0 frame:254276

This was setting off their monitoring and giving them some grief.
Suspecting something with jumbo frames, they increased the MTU on a physical server to 9000 and ran wireshark.
They identified broadcast or multicast packets that were larger than 1500 and all started with “00:50:56” which is the manufacturer code for VMware.

We attempted to identify if there was a VM or a VMkernel sending these broadcasts, but we could not find a MAC address that matched the ones from the Wireshark.

The Linux team looked at the capture again and noticed that the packet type was “unknown (0x8922)” which lead to the forums.

Apparently 0x8922 broadcasts and the random MACs are used by VDS Health Check to validate VLAN and MTUs.

Remember earlier when I said we setup the VDS at 9000 MTU? Apparently that includes all of the distributed port groups as well. The way that the VLAN and MTUs are verified is that a broadcast packet is sent with the MTU that the VDS is set to (in this case 9000) across that VLAN/DPG. If the host on the other end receives it without fragmenting, then VMware knows that the VLAN is good and the MTU is good. If the packet is received but it had to be fragmented, then the VLAN is good but the MTU is not. If there is no packet received, then the VLAN is most likely not trunked in.

We don’t see this issue on the VMs probably because VMware either filters those out or has some sort of smarts to know that these shouldn’t be counted as errors for the VMs.

The problem is that the physical servers don’t know what that 0x8922 packet is, and since they are set to 1500 MTU, end up with Rx errors.

So the solution?

  • The obvious solution (for me) would be to change the VDS global setting to 1500 MTU so that the port groups are checked at 1500, the VMkernels can remain at 9000 while the default for the switch is 1500 (see kb). You would have to be careful to create the VMkernels at 9000 though since this is not the default (not a problem for us since we use host profiles for this location).
    • Unfortunately I did try this in non-prod and I saw hosts flicker on and off (I believe setting the MTU will cause the hosts to disconnect the networks and reconnect to renegotiate at the new MTU) and some hosts didn’t take the change at all. This could be due to the use of LACP in this environment, or it could just be something specific to MTU changes
  • A solution from the linux side would be to separate the physical servers onto a dedicated subnet. Re-ip ing the hosts and moving to nics to a new VLAN isn’t the easiest of tasks though.
  • The last solution (more of a stopgap) would be to just turn off VDS Health Check for VLAN/MTU. I did open a SR with VMware to see if I could only do a VLAN check at a lower MTU or tweak the MTU checks per portgroup (blog post courtesy of virtuallyghetto William Lam. Unfortunately the only thing that can be done is to change the frequency of the VLAN/MTU check (response from Vmware below):
    • 1) Is there a way to do ONLY VLAN checking and not MTU?

      No. Even at the api level you enable them together.
      You can modify the interval of the check, for instance you can set it to check once per hour, so there wouldn’t be so many errors on the physical hosts.
      Please review http://blogs.vmware.com/vsphere/2012/12/configuring-vds-network-health-check-interval-using-vsphere-api.html for a description on a perl script that can do it
      Also, please note that at GSS we don’t provide or test script as a rule.

      2) Is there a way to change the MTU check per portgroup?
      No. The MTU is set at the dvSwitch level, and all portgroups share the same MTU.
      You could create a new vDS switch with the portgroups you want to have 1500 MTU, but you would need extra uplinks for this.

At the end of the day I ended up turning off VDS Health Check for VLAN/MTU since it was the easiest thing to do. It’s really not a long term solution though since ensuring VLAN (and MTU) consistency is really important (we’ve been bitten in the past). I am hoping that I can bring up a new VDS and try to migrate hosts between the two, though I realize that this may be difficult with LACP.

 

3 thoughts on “Receive Packet Errors with VDS Health Check and Jumbo Frames

Leave a Reply

Your email address will not be published. Required fields are marked *