bookmark_borderHome Lab – Virtual Router in your home lab – pfsense

I spent quite a bit of time researching solutions for virtual routers. I’m definitely not a r/s expert, so I wanted something that was easy to setup, and still provided lots of tools for troubleshooting. Initially I setup the lab using vyos router, and it met my needs. But I since switched to pfsense, and it’s absolutely amazing. Plus it comes with a pretty sweet firewall. So my guide here, will be a tutorial on how to deploy pfsense and configure it for home lab purposes. If you already have a router in mind, or a physical router, feel free to skip these steps.


Download pfsense community edition

https://www.pfsense.org/download/ – Download the ISO. Version used in my lab was 2.5.1. (Around 600MB)


Deploying the pfsense vm

Load the UI of your baremetal esxi7 and under Virtual Machines select “Create / Register VM”. Then select Next.
Name the vm “pfsense-router” and select the options as listed above.
Select the SSD
1 CPU should be fine, but feel free to make it 2 if you want. 2GB memory and 8GB HD is fine (Thin provisioned)
Select the CD/DVD Drive 1 and the dropdown to Datastore ISO file
Navigate the datastore to find the pfsense ISO file. If you haven’t uploaded it yet, then upload it to a directory (I created a directory called ISOs)
Setup should look like this. Click Next.
Summary, click Finish.
Power on the VM

Installing pfsense

Power on the VM and select the black GUI window, it will open the console window for this vm
Accept the agreement.
Install
Configure with default keymap
Auto (UFS) BIOS – Guided Disk Setup using BIOS boot method
Now let the installer run, should be fast, 1-2min tops.
Select No
Reboot, then exit the console. We need to add the networks to the vm.

Adding networks to pfsense vm

Click the vm and select Actions in the menu above, then “Edit settings”
Add 5 more network adapters and configure them as follows. Click Save and power on the VM

Initial config and disable firewall in pfsense cli

Back in the console, and you can see it is still rebooting…
After it has booted you will see this first time setup wizard. Select no for vlans.
Enter em0 for the WAN (If you don’t see this screen, don’t worry)
Leave this blank (If you don’t see this screen, don’t worry)
Ok, setup is complete and now we need to disable the firewall so we can continue setup in the GUI
Type 8 and press enter
Run the command: pfctl -d
By default the firewall is very restrictive so you won’t be able to access the GUI. We will reenable it later, after we’ve opened up the correct firewall rules πŸ™‚

GUI configuration of pfsense – Initial config of firewall

At this point you can access the GUI of the pfsense firewall. login with admin:pfsense
It’s a good idea to go ahead and reset the admin account password, click the link in the red banner at the top of the page.
Navigate over to the firewall tab, and select Rules. We need to configure some basic firewall rules to allow us access to the firewall. (Which is blocked by default)
Select the Green Add button and let’s add the first rule. As you can see in this image, I modified the destination port to SSH (22) and the Destination “This firewall (self)”
I also added a description. Click save.
DO NOT APPLY THE CHANGES YET! Your screen should look like this..
Go ahead and configure 2 more rules, in the same manner. for ICMP and port 443 (HTTPS)
Lastly we need to remove these 2 rules, by editing the WAN interface. Click on the gear icon on the right side.
Scroll all the way down on the page and you will see an option to uncheck these boxes. Uncheck “Block private networks and loopback addresses” and “Block bogon networks”
Go ahead and apply these changes.
Navigate back to the Firewall rules section and you’ll see those 2 rules are gone. All that is left is the 3 rules that you created. Go ahead and apply changes. It will take a minute, but if everything worked, you should still have access to the GUI.

Setting up the Interfaces

Next up, let’s rename and configure the interfaces for our WAN, then our 4 Internal vlans. Navigate to Interfaces -> Assignments
Click Add a few times until you add all the available interfaces. Should look like this (Don’t worry if your interface names are slightly different)
Now we’re going to go 1 by 1 through each of the interfaces and do the following:
1.Enable
2. Set Description
3. Static IPv4
4. IPv6 None
5. IPv4 Address and Mask (Select a static IP in your home network)
6. IPv4 Upstream gateway (Choose your gateway, ONLY SET THIS ON THE WAN)
And lastly, make sure these are unchecked. Save.
We’re going to repeat this process for each of the other interfaces. Here’s an example of the first vlan interface.
Same thing, uncheck these boxes. Save.
In the end, your interfaces should be setup like this. Check the network diagram if you are unsure of what IPs to use.

Firewall rules for each of the interfaces

I’m not going to go through each of the edit screens on how to add firewall rules, since I have shown examples in the above screenshots. Instead i’m just going to post screenshots of the firewall rules that I have added for each interface. In some instances I have just added a blanket allow all from any to any because its my home lab and pfsense is not my actual internet firewall πŸ™‚ If you want to keep it more secure, go for it.

Nothing to configure on the Floating interface.
Internet has the 3 rules we added initially. And my blanket allow everything rule πŸ™‚ (which essentially makes the first 3 rules useless)
The Anti-Lockout rule is set by default on the LAN interface, it’s not hurting anything so you can leave it. Add 2 more rules.
I started getting lazy and only adding ipv4. Honestly, I don’t use ipv6..
Same..
Same..
Last one.. And once they’re all set, you can Apply the changes.

Configure DHCP – Optional, but recommended

I would recommend setting up 50 or so IPs to be used for DHCP. It makes spinning up new vms in vcenter a little easier since it will auto select and IP, get the correct dns server, domain name, etc.
Configure these on all the interfaces EXCEPT The WAN (Internet) interface.

Click Enable, and setup using similar settings as above. I chose the range .100 – .150 in all my vlans. You can choose any range in the /24.
Set the DNS server
Set the domain name, and search list
Nothing here. Just hit save, then configure the rest.

Optional: Parse the xml and skip the step by step guide

This is my backup config. It probably won’t load for you if you try and do a system restore since I’ve removed cert info and passwords, etc. But if you prefer to parse this and setup pfsense, i’m making it available.

XML from pfsense router

bookmark_borderHome Lab – Configuring Nested ESXi7 on SuperMicro

We now have our SuperMicro server setup, and the esxi7 network settings are configured. It’s time to deploy our 3 nested ESXi7 vms. In later steps, we will configure them to use vSAN and be managed with vcenter.

Deploy 3 ESXi7 VMs

Navigate to the virtual machine tab. It’s time to setup our nested ESXi7 vms.
Click “Create / Register VM” and you’ll see the popup above. You can deploy esxi in 2 ways. We’re doing the ova method, so select the second option.

Methods to install nested ESXi in your lab:

  1. The method we used for the baremetal install. Basically mount the ISO as a (Datastore ISO) to a newly created VM, and go through the install process. This takes time to do for 3 vms, and i’m lazy πŸ™‚
  2. Use a preconfigured ova, and just modify it to our needs. We will be using this method. Head on over to William Lam’s website to download the ova. Filename: ESXi 7.0 Update 2a Virtual Appliance.
Specify the name and download the ova from the above link (William Lam’s website) and select that ova image here.
Select your storage device
The ova will take a few minutes to extract, be patient..
Once it’s ready, click I agree to accept the EUL and click Next.
Select VM Network, and Disk Provisioning Thin. Deselect Power on automatically. Next.
Leave all these default, and click Next.
Here’s the summary page, select Finish.

Configure the 3 nested ESXi7 VMs

Your screen should look like this. the vm is deployed, but it is powered down. If it’s not powered down, please do so now.
Click on the VM and goto the Actions menu, select Edit Settings.
We’re going to overprovision our environment pretty heavily here. My vsphere license allowed me to have 32 cpus, so I provisioned 3 ESXi hosts with 10, 10, and 12 CPUs. Depending on what your license is, you might change this number up or down slightly. Make sure “Expose hardware assisted virtualization to the guest OS” is checked!
yada yada, Warning saying be careful about over provisioning lol. Ignore it πŸ™‚ CPU is 10, memory is 128GB, everything else is default so far. Don’t worry about memory reservations under Memory tab, I left all that default.
Create 3 new disks, these will be used for vSAN. (You will create 3 disks on each of the nested esxi vms)
Pay attention to this step!!!!!!!! Set the hard disks to the GB as I listed above (Most importantly that the new disks are 200gb or more AND THAT THEY ARE ALL SET TO “Thin provisioned” If you select Thick, it will auto fill the space in your SSD, thin, it uses it as needed. Every disk should be thin provisioned in this setup.
Navigate back to the top and click “Add network adapter.” You will be adding 8 more network adapters, for a grand total of 10.
Make sure they’re all connected and configure them as shown above. Leave all the other settings under the networks default.
CD and video card, leave default.
Click Save and you should see a screen similar to above.

Repeat the above steps to create 3 total ESXi nested vms: vesxi7-1, vesxi7-2, vesxi7-3

  • 12 vcpu (12, 10, 10)
  • 128 GB memory
  • 12 HD1 thin
  • 4 HD2 thin
  • 20 HD3 thin
  • 200 HD4 thin
  • 200 HD5 thin
  • 200 HD6 thin
  • Network Adapter 1 vLAN10-Management
  • Network Adapter 2 vLAN10-Management
  • Network Adapter 3 vLAN20-vMotion
  • Network Adapter 4 vLAN20-vMotion
  • Network Adapter 5 vLAN30-vSAN
  • Network Adapter 6 vLAN30-vSAN
  • Network Adapter 7 vLAN40-vmNetwork
  • Network Adapter 8 vLAN40-vmNetwork
  • Network Adapter 9 vLAN50
  • Network Adapter 10 vLAN50
After you’ve finished, you’ll see 3 vms as shown above. Go ahead and power them on.

Powering on the Esxi hosts and setting management network

Load the console and press F2 to enter the setup. Default credentials are: root: VMware1!
After authenticating, we need to change the root password!
Now time to configure the management network, since it is likely using DHCP, and we want these ips to be static!
Click on IPv4 Configuration and select the radio button next to “Set static IPv4 address and network configuration”
You’ll need 3 IPs, 1 for each of the esxi hosts.
Disable ipv6
DNS server should already be here, since we configured DHCP on the pfsense router. However, if it is not. Specify them manually.
Lastly make sure that your suffix is set. You don’t have to use home.lab for your envrionment, but make sure you’re consistent across the board.

bookmark_borderHome Lab – SuperMicro Home Lab Components

Component List

I pretty much followed this guide, except for a few small changes since some of the parts were discontinued or changed model number.

QuanityNameDetails
1SAMSUNG 870 EVO Series 2.5β€³ 2TB SATA III V-NAND Internal Solid State Drive (SSD) MZ-77E2T0B/AM2TB SSD, in hind sight, I should have bought an 8TB. I’ll have to install a new one later..
1Supermicro SSD-DM032-SMCMVN1 32GB SATA DOMSATADOM installs right on the motherboard, and you will install the esxi image onto this drive. 32GB is plenty.
1SUPERMICRO MBD-M11SDV-8C+-LN4F-O Mini ITX Server Motherboard8 cores, and its pretty beefy, and you can overclock it. I have quite a bit running and i’m only around 60-70% utilization. And up to 512gb of memory should be plenty. Also this model comes with an active fan on the cpu!
2Noctua NF-A6x25 PWM, Premium Quiet Fan, 4-Pin (60mm, Brown)Honestly these don’t fit great, it is a snug fit. I would probably have went with just 1, or maybe a single larger fan. But they sure are quiet!
1Supermicro CSE-721TQ-350B 350W Mini-Tower ChassisThe tower
1256GB 4x64GB DDR4-2666 PC4-21300 2Rx4 RDIMM ECC Registered Memory by NEMIX RAMI bought 256gb of memory. Obviously you could get 4 sticks of 128 and just max it out, but my thought process is that if I actually exceed the 256, then it’s probably time to add a second server and get more cpu anyway.
MultipleEthernet cables
monitor/keyboard for configuring and installing esxi on the supermicro
surge protector/battery pack
You’ll obviously need some extra components that you might already have laying around..

Optional Component For Rack Enclosure

QuantityNameDetails
1NavePoint 9U Wall-Mount Network Cabinet Enclosure, 600mm Depth, Hinged Back, Swing Gate Server Cabinet, Locks, Pre-Assembled, Reversible Glass Front Door, 1 x L Brackets, 2 Fans, Cable Management9U was plenty big for my setup, again it would theoretically give you enough for a switch, power strip, 2 SuperMicro Mini Towers and at least another 1 or 2 U for something else.
1NETGEAR 24-Port Gigabit Ethernet Unmanaged Switch (JGS524) – Desktop or Rackmount, and Limited Lifetime Protection24 ports is quite a bit if this is all you have in your lab, but It would be good for future expansion
1AC Infinity MULTIFAN S7, Quiet Dual 120mm USB Fan, UL-Certified for Receiver DVR Playstation Xbox Computer Cabinet CoolingNot required, the rack comes with 2 fans. I found them to be pretty loud so I bought these fans, and they’re MUCH quieter.
1AC Infinity Turbo Fan Power Adapter, for MULTIFAN Series USB FansAdapter for the fans
1StarTech.com 8 Outlet Horizontal 1U Rack Mount PDU Power Strip for Network Server Racks – Surge Protection – 120V/15A – with 6 Ft Power Cord (RKPW081915), Black8 Outlets for power in the rack
1Amazon Basics RJ45 Cat-6 Ethernet Patch Internet Cable – 5 Feet (1.5 Meters), 10-PackEthernet cables

The Build

I built this a month or so ago so I’ll have to go back and get some pictures of the internal components, for now, follow the guide here: https://jorgedelacruz.uk/2020/10/05/supermicro-analysis-of-the-best-home-lab-server-2020-supermicro-m11sdv-8c-ln4f/ 

I don’t have exact steps for the build, but if you’ve built a computer before, its basically the same thing. Ram goes in the ram slots, sata cables to your ssd.. All pretty easy.

bookmark_borderHome Lab – Installing ESXi7 on your SuperMicro

Guide to install esxi7 on the newly built SuperMicro server. There are a ton of guides out there explaining how to install ESXi onto your host. Honestly, it’s not that difficult. But i’ll post my setup.

Version

VMware vSphere Hypervisor (ESXi) 7.0U2a

Install

Download the ISO file from vmware, should see a file like this: VMware-VMvisor-Installer-7.0U2a-17867351.x86_64.iso

Burn the ISO to a usb, microsd, ssd, etc. In my case, I had a USB laying around and burned it there. File size is around 400MB. I used Rufus 3.14

Insert the usb into SuperMicro usb slot in the back and power on, you will need a keyboard and a monitor hooked up as well.

You will see the ESXi installer loading..
Still loading…
Ready to install, press Enter
F11 to accept the agreement
If you have my exact setup, you’ll see an option in here for the SATADOM, select that.
Select your language.
Set a complex root password
You should NOT see this if you’re installing on the SuperMicro board that I used in my setup. I’m showing this error just in case you run into it. If you do, you need to enable virtualization in your bios.
F11 to repartition and install
Installing ESXI7 on the SATADOM..
Press enter to reboot, and remove your usb.
After it reboots, it will reboot ESXi. You should see a screen simliar to this, press F2 to enter setup and enter your root password.
Go to configure management network, we need to statically set the ipv4 address and some other settings.
Click on IPv4 Configuration and check the radio button next to “Set static IPv4 address and network configuration”. Add your address, subnet, and default gateway and press Enter.
Next goto DNS Configuration and check the radio button next to “Use the following DNS server address and hostname.” Add your dns server (possibly your home modem’s default gateway) and you hostname, localhost is fine.
Lastly under Custom DNS SUffixes, add a fqdn. You will need this later, don’t skip this step!
Accept the changes. This will restart the management network and apply the changes.
And finally, you should be able to access the UI by the IP address. (https://IPADDRESS)

If you followed all the steps above, you should now have ESXi7 installed on your supermicro and you should be able to access the UI. Username will be root, password will be the password you set in the above steps.