I spent quite a bit of time researching solutions for virtual routers. I’m definitely not a r/s expert, so I wanted something that was easy to setup, and still provided lots of tools for troubleshooting. Initially I setup the lab using vyos router, and it met my needs. But I since switched to pfsense, and it’s absolutely amazing. Plus it comes with a pretty sweet firewall. So my guide here, will be a tutorial on how to deploy pfsense and configure it for home lab purposes. If you already have a router in mind, or a physical router, feel free to skip these steps.
Load the UI of your baremetal esxi7 and under Virtual Machines select “Create / Register VM”. Then select Next.Name the vm “pfsense-router” and select the options as listed above.Select the SSD1 CPU should be fine, but feel free to make it 2 if you want. 2GB memory and 8GB HD is fine (Thin provisioned)Select the CD/DVD Drive 1 and the dropdown to Datastore ISO fileNavigate the datastore to find the pfsense ISO file. If you haven’t uploaded it yet, then upload it to a directory (I created a directory called ISOs)Setup should look like this. Click Next.Summary, click Finish.Power on the VM
Installing pfsense
Power on the VM and select the black GUI window, it will open the console window for this vmAccept the agreement.InstallConfigure with default keymapAuto (UFS) BIOS – Guided Disk Setup using BIOS boot methodNow let the installer run, should be fast, 1-2min tops.Select NoReboot, then exit the console. We need to add the networks to the vm.
Adding networks to pfsense vm
Click the vm and select Actions in the menu above, then “Edit settings”Add 5 more network adapters and configure them as follows. Click Save and power on the VM
Initial config and disable firewall in pfsense cli
Back in the console, and you can see it is still rebooting…After it has booted you will see this first time setup wizard. Select no for vlans.Enter em0 for the WAN (If you don’t see this screen, don’t worry)Leave this blank (If you don’t see this screen, don’t worry)Ok, setup is complete and now we need to disable the firewall so we can continue setup in the GUIType 8 and press enterRun the command: pfctl -d By default the firewall is very restrictive so you won’t be able to access the GUI. We will reenable it later, after we’ve opened up the correct firewall rules 🙂
GUI configuration of pfsense – Initial config of firewall
At this point you can access the GUI of the pfsense firewall. login with admin:pfsenseIt’s a good idea to go ahead and reset the admin account password, click the link in the red banner at the top of the page.Navigate over to the firewall tab, and select Rules. We need to configure some basic firewall rules to allow us access to the firewall. (Which is blocked by default)Select the Green Add button and let’s add the first rule. As you can see in this image, I modified the destination port to SSH (22) and the Destination “This firewall (self)”I also added a description. Click save.DO NOT APPLY THE CHANGES YET! Your screen should look like this..Go ahead and configure 2 more rules, in the same manner. for ICMP and port 443 (HTTPS)Lastly we need to remove these 2 rules, by editing the WAN interface. Click on the gear icon on the right side.Scroll all the way down on the page and you will see an option to uncheck these boxes. Uncheck “Block private networks and loopback addresses” and “Block bogon networks”Go ahead and apply these changes.Navigate back to the Firewall rules section and you’ll see those 2 rules are gone. All that is left is the 3 rules that you created. Go ahead and apply changes. It will take a minute, but if everything worked, you should still have access to the GUI.
Setting up the Interfaces
Next up, let’s rename and configure the interfaces for our WAN, then our 4 Internal vlans. Navigate to Interfaces -> AssignmentsClick Add a few times until you add all the available interfaces. Should look like this (Don’t worry if your interface names are slightly different)Now we’re going to go 1 by 1 through each of the interfaces and do the following: 1.Enable 2. Set Description 3. Static IPv4 4. IPv6 None 5. IPv4 Address and Mask (Select a static IP in your home network) 6. IPv4 Upstream gateway (Choose your gateway, ONLY SET THIS ON THE WAN)And lastly, make sure these are unchecked. Save.We’re going to repeat this process for each of the other interfaces. Here’s an example of the first vlan interface.Same thing, uncheck these boxes. Save.In the end, your interfaces should be setup like this. Check the network diagram if you are unsure of what IPs to use.
Firewall rules for each of the interfaces
I’m not going to go through each of the edit screens on how to add firewall rules, since I have shown examples in the above screenshots. Instead i’m just going to post screenshots of the firewall rules that I have added for each interface. In some instances I have just added a blanket allow all from any to any because its my home lab and pfsense is not my actual internet firewall 🙂 If you want to keep it more secure, go for it.
Nothing to configure on the Floating interface.Internet has the 3 rules we added initially. And my blanket allow everything rule 🙂 (which essentially makes the first 3 rules useless)The Anti-Lockout rule is set by default on the LAN interface, it’s not hurting anything so you can leave it. Add 2 more rules.I started getting lazy and only adding ipv4. Honestly, I don’t use ipv6..Same..Same..Last one.. And once they’re all set, you can Apply the changes.
Configure DHCP – Optional, but recommended
I would recommend setting up 50 or so IPs to be used for DHCP. It makes spinning up new vms in vcenter a little easier since it will auto select and IP, get the correct dns server, domain name, etc. Configure these on all the interfaces EXCEPT The WAN (Internet) interface.
Click Enable, and setup using similar settings as above. I chose the range .100 – .150 in all my vlans. You can choose any range in the /24.Set the DNS serverSet the domain name, and search listNothing here. Just hit save, then configure the rest.
Optional: Parse the xml and skip the step by step guide
This is my backup config. It probably won’t load for you if you try and do a system restore since I’ve removed cert info and passwords, etc. But if you prefer to parse this and setup pfsense, i’m making it available.
We now have our SuperMicro server setup, and the esxi7 network settings are configured. It’s time to deploy our 3 nested ESXi7 vms. In later steps, we will configure them to use vSAN and be managed with vcenter.
Deploy 3 ESXi7 VMs
Navigate to the virtual machine tab. It’s time to setup our nested ESXi7 vms.Click “Create / Register VM” and you’ll see the popup above. You can deploy esxi in 2 ways. We’re doing the ova method, so select the second option.
Methods to install nested ESXi in your lab:
The method we used for the baremetal install. Basically mount the ISO as a (Datastore ISO) to a newly created VM, and go through the install process. This takes time to do for 3 vms, and i’m lazy 🙂
Use a preconfigured ova, and just modify it to our needs. We will be using this method. Head on over to William Lam’s website to download the ova. Filename: ESXi 7.0 Update 2a Virtual Appliance.
Specify the name and download the ova from the above link (William Lam’s website) and select that ova image here.Select your storage deviceThe ova will take a few minutes to extract, be patient..Once it’s ready, click I agree to accept the EUL and click Next.Select VM Network, and Disk Provisioning Thin. Deselect Power on automatically. Next.Leave all these default, and click Next.Here’s the summary page, select Finish.
Configure the 3 nested ESXi7 VMs
Your screen should look like this. the vm is deployed, but it is powered down. If it’s not powered down, please do so now.Click on the VM and goto the Actions menu, select Edit Settings.We’re going to overprovision our environment pretty heavily here. My vsphere license allowed me to have 32 cpus, so I provisioned 3 ESXi hosts with 10, 10, and 12 CPUs. Depending on what your license is, you might change this number up or down slightly. Make sure “Expose hardware assisted virtualization to the guest OS” is checked!yada yada, Warning saying be careful about over provisioning lol. Ignore it 🙂 CPU is 10, memory is 128GB, everything else is default so far. Don’t worry about memory reservations under Memory tab, I left all that default.Create 3 new disks, these will be used for vSAN. (You will create 3 disks on each of the nested esxi vms)Pay attention to this step!!!!!!!! Set the hard disks to the GB as I listed above (Most importantly that the new disks are 200gb or more AND THAT THEY ARE ALL SET TO “Thin provisioned” If you select Thick, it will auto fill the space in your SSD, thin, it uses it as needed. Every disk should be thin provisioned in this setup.Navigate back to the top and click “Add network adapter.” You will be adding 8 more network adapters, for a grand total of 10.Make sure they’re all connected and configure them as shown above. Leave all the other settings under the networks default.CD and video card, leave default.Click Save and you should see a screen similar to above.
Repeat the above steps to create 3 total ESXi nested vms: vesxi7-1, vesxi7-2, vesxi7-3
12 vcpu (12, 10, 10)
128 GB memory
12 HD1 thin
4 HD2 thin
20 HD3 thin
200 HD4 thin
200 HD5 thin
200 HD6 thin
Network Adapter 1 vLAN10-Management
Network Adapter 2 vLAN10-Management
Network Adapter 3 vLAN20-vMotion
Network Adapter 4 vLAN20-vMotion
Network Adapter 5 vLAN30-vSAN
Network Adapter 6 vLAN30-vSAN
Network Adapter 7 vLAN40-vmNetwork
Network Adapter 8 vLAN40-vmNetwork
Network Adapter 9 vLAN50
Network Adapter 10 vLAN50
After you’ve finished, you’ll see 3 vms as shown above. Go ahead and power them on.
Powering on the Esxi hosts and setting management network
Load the console and press F2 to enter the setup. Default credentials are: root: VMware1!After authenticating, we need to change the root password!Now time to configure the management network, since it is likely using DHCP, and we want these ips to be static!Click on IPv4 Configuration and select the radio button next to “Set static IPv4 address and network configuration” You’ll need 3 IPs, 1 for each of the esxi hosts.Disable ipv6DNS server should already be here, since we configured DHCP on the pfsense router. However, if it is not. Specify them manually.Lastly make sure that your suffix is set. You don’t have to use home.lab for your envrionment, but make sure you’re consistent across the board.
8 cores, and its pretty beefy, and you can overclock it. I have quite a bit running and i’m only around 60-70% utilization. And up to 512gb of memory should be plenty. Also this model comes with an active fan on the cpu!
I bought 256gb of memory. Obviously you could get 4 sticks of 128 and just max it out, but my thought process is that if I actually exceed the 256, then it’s probably time to add a second server and get more cpu anyway.
Multiple
Ethernet cables monitor/keyboard for configuring and installing esxi on the supermicro surge protector/battery pack
You’ll obviously need some extra components that you might already have laying around..
9U was plenty big for my setup, again it would theoretically give you enough for a switch, power strip, 2 SuperMicro Mini Towers and at least another 1 or 2 U for something else.
I don’t have exact steps for the build, but if you’ve built a computer before, its basically the same thing. Ram goes in the ram slots, sata cables to your ssd.. All pretty easy.
Guide to install esxi7 on the newly built SuperMicro server. There are a ton of guides out there explaining how to install ESXi onto your host. Honestly, it’s not that difficult. But i’ll post my setup.
Version
VMware vSphere Hypervisor (ESXi) 7.0U2a
Install
Download the ISO file from vmware, should see a file like this: VMware-VMvisor-Installer-7.0U2a-17867351.x86_64.iso
Burn the ISO to a usb, microsd, ssd, etc. In my case, I had a USB laying around and burned it there. File size is around 400MB. I used Rufus 3.14
Insert the usb into SuperMicro usb slot in the back and power on, you will need a keyboard and a monitor hooked up as well.
You will see the ESXi installer loading..Still loading…Ready to install, press EnterF11 to accept the agreementIf you have my exact setup, you’ll see an option in here for the SATADOM, select that.Select your language.Set a complex root passwordYou should NOT see this if you’re installing on the SuperMicro board that I used in my setup. I’m showing this error just in case you run into it. If you do, you need to enable virtualization in your bios.F11 to repartition and installInstalling ESXI7 on the SATADOM..Press enter to reboot, and remove your usb.After it reboots, it will reboot ESXi. You should see a screen simliar to this, press F2 to enter setup and enter your root password.Go to configure management network, we need to statically set the ipv4 address and some other settings.Click on IPv4 Configuration and check the radio button next to “Set static IPv4 address and network configuration”. Add your address, subnet, and default gateway and press Enter.Next goto DNS Configuration and check the radio button next to “Use the following DNS server address and hostname.” Add your dns server (possibly your home modem’s default gateway) and you hostname, localhost is fine.Lastly under Custom DNS SUffixes, add a fqdn. You will need this later, don’t skip this step!Accept the changes. This will restart the management network and apply the changes.And finally, you should be able to access the UI by the IP address. (https://IPADDRESS)
If you followed all the steps above, you should now have ESXi7 installed on your supermicro and you should be able to access the UI. Username will be root, password will be the password you set in the above steps.
