Behind the NAT: A Greenfield VSAN Deployment

I have been consulting with a group in my company to setup a new VMware environment that would consist of a central component in the datacenter and remote components in each of the remote locations. There would be firewalls and NAT’ing setup to isolate these environments from the rest of the company intranet. This post will cover how I stumbled through a VSAN deployment in a greenfield environment all while dealing with a NAT.

I found out that there was no AD or DNS in this zone initially (first VMs to be stood up) and some services would be served from the regular corporate sourced (NTP for example).

This was my general plan:

1. Get access to ilo and ensure console and remote media would work
2. Build hosts (bios settings, firmware, ilo licensing, ESXi deployment)
3. Bootstrap VSAN on single node (William Lam’s Post)
4. Deploy VCSA 6 OVA (see last post for bootstrap, this post for extracting OVA)
5. Add hosts to vCenter, create VSAN cluster
6. Turnover to other team

Everything worked well until I got to step 4. My original plan of digging up the OVA from the Iso and deploying as-is was a bad idea. Unlike VCSA 5, there is no wizard once the VM is fired up. Nothing was up and running.

I deleted the VCSA and re-deployed using ovftool (see this post).  I had to create a batch file equivalent of William’s bash script (see below). Note that there weren’t any DNS services behind the firewall/NAT so I was deploying using IP address. In hindsight I could have used the GUI from the ISO to deploy the VCSA, but I think I was caught up in following William’s posts.

I could connect to the VCSA using the thick client via the external NAT address. The odd thing was that once I added the hosts to the VCSA (using internal IP), I could no longer access the console of the VMs. I also noticed that I could not download/upload using the datastore browser.

If I used the web client, it would redirect me to the sso page’s internal IP address (remember everything was setup using the internal IP). I needed the Web client to configure VSAN so I was stuck. My thoughts were to bring up a jump box in the environment (Windows) so that I could connect to the Web Client that way and it would be all local traffic.

So I stood up a Windows 2008 R2 box and installed the Google chrome msi (needed the pre-bundled flash) in order to access the web client. At this point I could have turned it over, but it still bothered me that I couldn’t use the web client externally or most of the functions successfully from external.
At that point I went to Twitter:

@ikiris Would DNS or reverse proxy help? Your vCenter doesn't know what its external IP is.

— Justin Warren (@jpwarren) December 1, 2015

Justin’s response gave me an idea.

I setup DNS records on the external side to have the same fqdn as the internal side and point to the external ips. For example: vcsa.domain.com -> 10.0.55.8
On the internal side, use hosts file to create name resolution of fqdn to internal ip. For example: vcsa.domain.com -> 192.168.55.8
In the above examples, 10.0.55.8 is the external ip and 192.168.55.8 is the internal ip. The firewall handles NAT’ing 192.168.55.8 -> 10.0.55.8

Justin calls this a split DNS:

@ikiris You'll need a split DNS: Inside knows the NATed inside IP of the host, outside knows the external address.

— Justin Warren (@jpwarren) December 1, 2015

I then needed to tell the VCSA to use the FQDN instead of the IP, but I could not figure out how to re-point to the embedded PSC using FQDN. I followed this  kb , but it would complain that the PSC was not a replication partner with the current PSC.

 

At this point I decided to ditch the VCSA. I renamed and re-ip’d my windows box, added hosts entries into C:\WINDOWS\system32\drivers\etc and installed vCenter 6. All of the ESXi host entries were added as well. The web client started to work externally, upload and download from the datastore browser worked  and once I bounced all of the hosts, console access worked as well.

Could I have used the VCSA in the same manner as the Windows vCenter? Doubtful. I needed to put the local hosts entries on the Windows VM prior to the vCenter install. With the VCSA deployment, there is no access to the OS to put those entries prior to installing the app. As mentioned earlier, I tried deploying without ovftool to just get the appliance up, but the appliance was in a bad state when it was up. If an external DNS server is reachable by the VCSA, you could possibly setup DNS entries on there using the internal IPs until the VCSA is up and running. Then update /etc/hosts with those entries and remove that DNS server from the VCSA. Last, update the external DNS entries. That seems like a lot more work than deploying Windows.

Justin also had a few more ideas:

 

TL;DR So at the end of the day, if I had to do it again I would follow these steps:

1. Get access to ilo and ensure console and remote media would work
2. Build hosts (bios settings, firmware, ilo licensing, ESXi deployment)
3. Create external DNS records for hosts,Ilos, vcenter, etc
4. Bootstrap VSAN on single node (link)
5. Upload Windows ISO to VSAN Datastore and deploy Windows VM
6. Add local hosts entries on Windows VM to match fqdn with internal IPs
7.  Install vCenter 6 with FQDN on Windows VM
8. Add hosts to vCenter using FQDN, create VSAN cluster
9. Turnover to other team to finish building out infrastructure