Recovering from a VSAN Array Controller Failure

I was taking a look at the new VSAN that I build (3-node of HP DL380 Gen 9), and I noticed an error message on the host/cluster. This was after I resolved the driver issue, so I knew it was something new.

From this screenshot, you will see that at the cluster level you see VIrtual SAN Health errors.

The same errors/alarms are visible in Monitor -> Issues -> Triggered Alarms

Going to the Monitor -> Virtual SAN -> Physical Disks, you will see a red error icon on the host and there are NO disks shown

At the Virtual Disk Level, the compliance for all of the disks is Noncompliant and there are missing components (data or witness). Since I only have three nodes, the failure in the one node means that there are not enough nodes to rebuild the components.

The Health section shows failed Data Health/Object Health

The host was still up, but all of the storage was gone. At this point I went into the ILO

 

The ILO Overview page had a critical alert

 

 

 

Summary showed a critical alert in storage

 

Lastly the HP Smart Array P440ar Controller itself showed as failed.

I was interested to see what this looked like in Log Insight as well:

 

I used a pre-canned query from the VSAN Content Pack for Log Insight and then went to the event trends.

There wasn’t anything specific to the hardware, but you could definitely see all of the VSAN errors.

At this point I opened a ticket with HP and collected the AHS logs. When they reviewed the logs, they told me that I didn’t have a failed controller, I hit a known issue. Apparently there was an updated array controller firmware that was released but was not part of the SPP disk (released after the SPP) but it WAS part of the HP Recipe. There is a footnote in the recipe that the firmware had to be downloaded separately.

I started reviewing the recipe and was set to download the firmware that was extra, but got distracted with this VSAN setup and just used the SPP as is.

If your controller has this same issue, you will see this error on boot-up

 

After using the VMware online components to apply the firmware, the array controller was back in action, all green in the Health Checks.

This reminded me of a conversation I had on twitter the other day:

@ikiris how do you mean? it checks against the HCL as it stands today?

— Duncan Epping (@DuncanYB) December 5, 2015

@DuncanYB @ikiris HCL and vendor recipes are a bit different though, while the HCL has everything other companies propagate a single combi

— Frank Büchsel (@fbuechsel) December 5, 2015

@fbuechsel @DuncanYB would be cool to pick that combination. Oneview may do this but usually only prior to upgrades, not as a daily check

— Chris Chua (@ikiris) December 5, 2015

VSAN Health Check worked great to notify me that I was using the wrong array driver (as determined from the HCL for VSAN), but there wasn’t a lot of checks for other firmware or drivers. I was trying to follow the November 2015 HP recipe guide, but I was missing the correct array firmware. What  would be really awesome is if Health Check could be loaded with HP’s recipes and tell me that I was using the wrong firmware/driver and if there were any related advisories to the firmware/driver that I was using. I haven’t used HP OneView for vCenter, but from what I can tell it displays software/firmware but it doesn’t match it against a preferred configuration. Maybe I’m looking for some sort of mash-up of VSAN Health Check and HP OneView.