bookmark_borderTanzu Kubernetes on vCenter 7 ā€“ Deploy Avi Controller and Service Engines

In order for tkgs to function, we need the Avi load balancer deployed. This is a very easy step and if you prefer to follow the official documentation it can be found here: https://avinetworks.com/docs/latest/installing-avi-vantage-for-vmware-vcenter/

Download Avi ova from VMware portal.

Login to your account at https://my.vmware.com/ and go to Products and Accounts ->Products -> All Products

Select VMware NSX Advanced Load Balancer, and click View Download Components
Then click Go to Downloads
Then Download Now.
Under Software you will see the latest versions, at the time of writing we are currently on 21.1.1.
Select the version and the VMware file type (ova) and click download on the right side.

Deploy the OVA template in vCenter

This step is pretty easy, but i’ll include a few pictures.

Right click on your VM folder and select Deploy OVF Template
Select the avi controller ova file
Click next a few times until you get to the Customize Template. It’s not required but I would suggest adding a static IP address here for the Avi controller. I’m using 10.10.4.5. Add the mask and gateway, and no other settings are required. Click next and deploy the ova.

Configure Avi Controller

There’s only a few steps here required to configure the controller, then we can move into the tanzu side of the house.

Access the Avi controller by fqdn, and set a new password.
Set some additional System Settings, backup passphrase, dns resolver, dns search domain. Leave everything else default and Submit.
This is the screen you should be seeing now.

Configure Cloud

Navigate to Infrastructure -> Clouds and select the pencil “Edit” on Default-Cloud
Select VMware Cloud
Add in the credentials for vCenter and the IP address, select Next.
Select the vSAN Datacenter, click Next
Lastly configure the management network and static IP ranges.
Wait a few moments and your cloud should turn green.

Configure PodNetwork

Navigate to Infrastructure -> Networks and select PodNetwork. (If this does not exist, go back to vCenter and under the Networks tab, add a Distributed Port Group called “PodNetwork” under DSwitch.
Click edit on the PodNetwork and add the subnet and static range as listed above.

Configure Default route for SEs

Navigate to Infrastructure -> Routing and click Create.
Add 0.0.0.0/0 and the next hop of 10.10.4.1

Create IPAM profile and DNS profile and add them to the Cloud

IPAM allows Avi to auto allocate IP addresses to newly created virtualservices.

Navigate to Templates -> IPAM/DNS Profiles and click Create IPAM Profile.
Modify the settings to the above and click Save.
Modify the Dns profile settings to the above and click Save.
Lastly navigate back to the Infrastructure -> Clouds and click edit on the Default-Cloud
Add the ipam profile and dns profile to the cloud and click Save.

Create a controller certificate

Navigate to Templates -> Security -> SSL/TLS Certificates and click Create Controller Certificate
Create a new certificate called ControllerCert with the fqdn as common name and the IP address as a SAN name. Everything else is default.
Then navigate to Administration -> Settings -> Access Settings and click the pencil on the right to edit.
Modify the SSL/TLS Certificate (for the controller) to the newly created certificate and save.

Create a test vs

Add the name “test-vs” and set the Network by selecting VM Network, and the available subnet 10.10.4.0/24. Change Application Profile to “System-L4-Application” and the port to 443. Then on the bottom right, select Pool and in the drop down click Create Pool.
The pool name will prefill. Change the port to 443 and select System-TCP health monitor. Click Next
If you already have a server in mind then add it here. I always add the avi controller (by fqdn) because it’s fast and I know it will have connectivity to itself šŸ™‚ Click next and save the pool.
You will now see the pool in the drop down. Click Next through all the screens and save and create the virtual.
The virtual service will be marked down for a few minutes while Avi spins up some service engines to handle the traffic. Check back in 5 min or so.
After a few minutes the virtualservice will show Green and Avi is configured and ready to go.

bookmark_borderHome Lab ā€“ Overclocking the Supermicro M11SDV-8C+-LN4F (MBD-M11SDV-8C+-LN4F-O Mini ITX Server)

This is not a required step to build your home lab. It’s just an extra step that will give you a slight performance boost if you want it. Obviously you should know that modifying any of these BIOS settings can break your system and I nor Supermicro are responsible for it. I’m providing this as a reference on how I did it for my setup.

Manual for the board: https://www.supermicro.com/manuals/motherboard/EPYC3000/MNL-2172.pdf

Overclocking the Supermicro

Reboot and enter the BIOS

Reboot the device either by sshing into esxi and typing “reboot” or by resetting the power button on front of the Supermicro. Upon reboot, press F11 to enter the Aptio Setup Utility BIOS screen. (I don’t have any screenshots of this, but it’s pretty straightforward.)

Modify CPU Settings

There are 2 settings that I adjusted to increase my clock speed.

  • cTDP Control setting from default of 55W to 75W
  • Determinism Slider from default of Auto to Power

To modify these, navigate to the Advanced Tab, then select CPU Configuration. You will see the 2 options for cTDP and Determinism Slider, modify them to the settings listed above.

That is it. Save the settings and exit the BIOS.

The recommended and default power settings for this board is 55W
Here are the settings under NB Configuration.

bookmark_borderTanzu Kubernetes on vCenter 7 ā€“ Deploy an Application (Blue)

This is really the final step in setting up TKGs, testing the deployment. We will create a simple 2 pod deployment and use the Avi load balancer as the load balancer.

Deploy the Blue Application in Tanzu Guest Cluster

Login to the guest cluster and enable privileges

Run the following commands to login to the vSphere Tanzu cluster, and switch context to the new guest cluster that was created. By default Tanzu has a fair amount of Pod security, and we will be restricted in what we can create, unless we open up the access. Since this is a lab environment, it should not be an issue. The last command will essentially provide full access for creating services, deployments, pods, etc. More info: https://docs.vmware.com/en/VMware-vSphere/7.0/vmware-vsphere-with-tanzu/GUID-4CCDBB85-2770-4FB8-BF0E-5146B45C9543.html

kubectl vsphere login --vsphere-username administrator@vsphere.local --server=https://10.10.4.50 --insecure-skip-tls-verify --tanzu-kubernetes-cluster-namespace=dev --tanzu-kubernetes-cluster-name=tkg-cluster-01
kubectl config use-context tkg-cluster-01
kubectl create clusterrolebinding psp:authenticated --clusterrole=psp:vmware-system-privileged --group=system:authenticated

Create file blue-deployment-l4.yaml

Use nano/vi/vim or your favorite editor and create this file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: blue
spec:
  selector:
    matchLabels:
      app: blue
  replicas: 2
  template:
    metadata:
      labels:
        app: blue
    spec:
      containers:
      - name: blue
        image: mattadam07/bluegreen:latest
        ports:
        - containerPort: 5000
        env:
        - name: app_color
          value: "blue"
---
apiVersion: v1
kind: Service
metadata:
  name: blue
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 5000
    protocol: TCP
  selector:
    app: blue

Apply the blue-deployment-l4.yaml file

kubectl apply -f blue-deployment-lb.yaml
deployment.apps/blue created
service/blue created

Run “kubectl get pods” to see the status. You will see the following if done correctly

deployment.apps/blue created
service/blue created
kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
blue-c967796c6-p24kc   1/1     Running   0          76s
blue-c967796c6-sfk7s   1/1     Running   0          76s

Check the services and see if the LoadBalancer endpoint was created successfully. The IP 10.10.4.18 should now be accessible and you should be able to test it.

kubectl get services
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
blue         LoadBalancer   10.109.206.160   10.10.4.18    80:32242/TCP   4m4s
kubernetes   ClusterIP      10.96.0.1        <none>        443/TCP        4h47m
supervisor   ClusterIP      None             <none>        6443/TCP       4h47m

Validate the Avi LB VirtualService

Here is the newly created VirtualService. This was auto created through the built in AKO from TKGs. Note the IP address 10.10.4.18

Click edit on the Virtual Service and we can see that Application Profile is set for “System-L4-Application”, indicating this is an L4 vip. Additionally note that there is no Pool set at the bottom. This is actually done through an L4 Policy Set as shown below.

And lastly let’s test the URL: http://10.10.4.18

bookmark_borderTanzu Kubernetes on vCenter 7 – Deploy Guest Cluster

Now that we have the supervisor cluster up and running and our namespace created, we can deploy a guest cluster via the CLI. I installed an ubuntu 20 vm in vCenter for use as my jumpbox. I installed kubectl and the vsphere plugin in this environment. There are windows plugins and plugins for all the linux distros as well.

Install kubectl and vsphere plugin on jump server

Kubectl

You can download and install kubectl very easily (in linux) with these commands:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo mv kubectl /usr/local/bin/kubectl
sudo chmod +x /usr/local/bin/kubectl
kubectl version

vSphere Plugin

The easiest way to download this is to navigate to the supervisor cluster’s floating IP address. In my case it is 10.10.4.13

Select your OS and download the cli plugin. Upload to your jump box, and running the following commands.
sudo mv kubectl-vsphere /usr/local/bin/kubectl-vsphere
sudo chmod +x /usr/local/bin/kubectl-vsphere
kubectl vsphere

Login to Supervisor Cluster

OPTIONAL: Set Environment variable for vsphere password.

echo "KUBECTL_VSPHERE_PASSWORD='supersecretpassword123'" >> /etc/environment

Log into supervisor cluster and verify cluster is healthy

kubectl vsphere login --vsphere-username administrator@vsphere.local --server=https://10.10.4.50 --insecure-skip-tls-verify
kubectl config use-context dev
kubectl get pods --all-namespaces ### Should see a list of all the pods running
kubectl get nodes ### Everything should show Ready
kubectl get tanzukubernetesreleases ###Checkout the latest releases

Create yaml file to build guest cluster

Create a file called guest_cluster.yaml with the following content

---
apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TanzuKubernetesCluster
metadata:
  name: tkg-cluster-01
spec:
  topology:
    controlPlane:
      count: 1
      class: best-effort-small
      storageClass: vsan-default-storage-policy
    workers:
      count: 2
      class: best-effort-small
      storageClass: vsan-default-storage-policy
  distribution:
    version: v1.20.7

Deploy TKGs guest cluster

kubectl apply -f guest_cluster_tkgs.yaml
kubectl get cluster ### View the cluster status
kubectl get tanzukubernetescluster ### View the cluster status
Guest cluster is deploying
Still provisioning the cluster
Should be able to see the new vms spinning up in vCenter

bookmark_borderTanzu Kubernetes on vCenter 7 ā€“ Namespace Setup

After deploying the supervisor cluster, the next step is to setup the namespace where we will deploy our guest cluster.

Create Namespace

In the Menu click Workload Management. Then navigate to the Namespaces tab. Click Create Namespace.
Select the vSAN cluster and choose a name, I’m using “dev” Then select the workload network and finally add a description.
You should see the config status “Running” and kubernetes status “Active”. We need to configure Permissions, storage, capacity and usage, and the associated vm classes and content libraries for this namespace.

Configure Dev Namespace

Click Permissions and configure as shown above. If you’re using a different user, you can configure that here. I’m simply using the administrator for all access. Click OK.
Select Storage and choose the vSAN Default Storage Policy. Click OK.
Under Capacity and Usage, configure as shown above. I’m setting limits on memory and storage, but not CPU.
In the VM Service section, click Add VM Class and select the “best-effort-small” class. This will provide enough cpu and memory to the vms to handle a few deployments. If you need more, “best-effort_medium” would be a good fit as well.
Lastly, select the Add Content Library under VM Service, and add the kubernetes library.
Finished setup will look something like this.

bookmark_borderTanzu Kubernetes Content Library in vCenter 7

Before you can setup workload management in vCenter 7, you need to create a content library and setup the subscription to point to vmware’s library.

Add Content Library

Select the Menu and navigate to Content Libraries
Add a name for the content library and click Next.
Select Subscribed content library and add this Subscription URL: https://wp-content.vmware.com/v2/latest/lib.json
Additionally if you want to save space, select Download content when needed.
Yes to bypass the certificate warning.
Select the storage location, i’m using the vSAN datastore.
Review the summary page and click Finish.

bookmark_borderTanzu Kubernetes on vCenter 7 – Deploy Supervisor Cluster (WCP)

In this guide we will configure Workload Management for vCenter 7. We’ll be using vCenter Server Network (DSwitches) instead of NSX-T. Additionally we’ll be using the Avi Load Balancer (NSX Advanced Load Balancer).

Licensing for Supervisor Cluster

Right click on your vSAN cluster and navigate to Licensing. Select Assign Supervisor Cluster License and select the appropriate license. If you need to add a new license select Menu at the top -> Administration -> Licenses -> Add

Configuring Workload Management

Click the Menu and navigate to Workload Management, and you should see this page. (Assuming you licensed correctly). Click Get Started.
This alert is just informing you that Avi must already be preconfigured. If you haven’t done so yet, please do so now. Additionally we do not have NSX-T running in this lab, so vCenter Server Network is selected. Click Next.
Select the vSAN Cluster and click Next.
Pick the control plane size. I have found that Tiny was more than enough for my needs.
Select the default storage policy for control plane. I am using the vSAN Default Storage Policy. Click Next
Add in the details for the Avi load balancer. The name must be DNS compliant, so avi-controller-1 is simple and works.
Type: Avi
Controller IP: Use the IP and port here
Then add your username and password.
Add your Avi Controller Cert here as well. If you haven’t generated this yet, please do so now.
Again, as with everything VMware, make sure DNS works!

I’m using the 10.10.4.0/24 network for my management network. Select your starting range in that network and add your gateway. Add the dns server, search domain, and ntp server. Click Next.
Add in the Pod network (Workload Network) 10.10.5.0/24 is the network i’m using. Add the dns server then click Add for workload network.
In the popup add a name for the network and select the PodNetwork portgroup. Lastly add the gateway, subnet, and ip ranges. Click Save.
Everything should look like this. Click Next.
Select the kubernetes content library we created. Click Next.
All set! Click Finish.
You should see this screen. At this point go grab some coffee because this step takes quite a while, specially if your content library is set to “Download library content only when needed,” as mine is. It will download all the required ovas and start spinning up the supervisor cluster.
After a while (~45min for me) you should see your supervisor cluster up and running!
You can click the Menu and navigate to VMs and Templates and there should be 3 supervisor control plane vms running.

bookmark_borderHome Lab – Deploy vCenter 7 and vSAN on nested ESXi

If you’ve followed the guide this far, you’ve deployed 3 esxi hosts nested on your baremetal esxi install. This guide takes it a step further by deploying vcenter and creating a vSAN cluster on the esxi hosts.

Download vCenter Server

Login to your account at https://my.vmware.com/ and go to Products and Accounts ->Products -> All Products

Select VMware vSphere. View Download Components
Select your version and download the vCenter Server. I’m using VMware vCenter Server 7.0U2b with Enterprise Plus
Download the VMware vCenter Server Appliance (7.5GB)
VMware-VCSA-all-7.0.2-17958471.iso

Mount the ISO and use the install wizard to configure vCenter 7

I’m using windows 10, and it was relatively easy to mount the ISO. In Windows explorer, I just navigated over to the downloads directly where the ISO was, and double clicked it. Open the directory vcsa-ui-installer -> win32 -> installer.exe.

Stage 1


You should see a popup like this. Go ahead and click Install
Click Next to deploy vCenter server
Accept the EUL.
Put the IP/fqdn of the first esxi host, and the credentials.
Select Yes to accept the warning.
Specify a name for the vm, and set the root password
For deployment size I chose Tiny since it more than met my needs. If you need more, select Small.
Select “Install on a new vSAN cluster containing the target host” Feel free to modify the names.
We’re going to claim all the 200GB disks as capacity tier, and the 20GB disk as cache tier. The other disk we will not use. Additionally, I selected “Enable Thin Disk Mode” and “Enable Deduplication and compression” Since it’s a lab, i’m not too worried about a vSAN failure. Worst case, i’ll just rebuild the entire lab and get more practice.
Set the fqdn for vcenter, IP address and mask, default gateway, and dns server. vCenter is very picky about dns.. Make sure that the fqdn resolves and the ip address reverse lookup resolves as well.
Here’s the summary page. Go ahead and hit Finish then grab some coffee. This step takes a while.
Congratulations! It’s installed. Now onto stage 2 for some additional configuration. Click Continue

Stage 2

Into the setup wizard for stage 2. Click Next
You’re welcome to sync with a public ntp server (or private), I just selected the host for mine. Additionally, it’s a lab and ssh access to vCenter is very handy when troubleshooting issues later.
Set the SSO domain, I chose the default “vsphere.local” and enter your password.
Choose whether you want to join the CEIP.
Summary page. If all looks right, click Finish.
Stage 2 Completed. Vcenter is all setup. You can now access the UI:

vSAN Initial Setup

Launch vCenter in the browser.
Login with the administrator@vsphere.local account and password.
You’re going to see lots of alarms and warnings, don’t worry.. we’re going to fix it all in the next few steps.

Step 1: Cluster Basics

Navigate to the vSAN Cluster and select Configure. Then under Configuration click Quickstart. This provides a easy to use wizard for deploying HA and vSAN.
Step 1: Click Edit under Cluster Basics and make sure that all the options are turned on. vSphere DRS, vSphere HA, and vSAN.

Step 2: Add Hosts

Step 2: Under Add hosts, click ADD. Then add in the IP or fqdn of each of the other 2 esxi hosts. Then the user and passwords for each.
Select the hosts and click OK to accept the certificate security warning.
Summary of the hosts. Click Next
Ready to add them, click Finish. After you click finish, this will take some time. Just be patient.
Hosts are added, now on to step 3.

Step 3: Configure Hosts

Step 3: Click Configure under Configure cluster. I left all these settings default.
Set the vmnics as shown above. We will use this to setup vSAN and vMotion. Click Next.
I am using vlan20 (10.10.2.0/24) for my vmotion traffic. So I configured 3 interfaces for this traffic, 1 per esxi in the cluster. Also i’m not using vlans, so I have unchecked that box. Click Next.
Similarly, the vSAN vlan is vlan30 (10.10.3.0/24) and I configured 3 IP addresses on this network. Uncheck vlan if not in use. Click Next.
I left all of these settings default. You can turn on “Virtual Machine Monitoring” if you want. Everything else is fine as default. Click Next.
For the disks, select the “Group by:” as Host, and expand the hosts. You will see all the volumes that we created on the esxi setup.
Go through and claim the following:
200GB Claim as capacity tier
20GB Claim as cache tier
4GB do not claim
Click Next.
Skip this step, since we have already configured internet access.
Summary page. Review everything and click Finish. This step takes a while.. be patient.
Eventually everything will normalize and look like this. You can ignore those yellow alerts. As long as nothing is red, you will be fine.

Licensing vCenter

Click the Menu and navigate to Hosts and Clusters. Right click the vcenter1 instance, and select Assign License. Select the appropriate vCenter license and click Ok.

Licensing the ESXi hosts in vSAN cluster

Enter your licenses separated out by a new line. Click Next. Then you have the option to name your licenses, Next. Summary page. Save.
After adding the licenses, you will see them available here.
Select the Menu at the top and navigate to Hosts and Clusters
Right click on one of the ESXi hosts in the vSAN cluster and select Assign License. Then in the popup that will appear, you select the appropriate license. Repeat this steps for the other 2 ESXi hosts.

Set vSAN as default storage policy

Right click on the vCenter vm, and navigate to VM Policies, and select Edit VM Storage Policies
At the top select vSAN Default Storage Policy from the drop down and click Ok.

bookmark_borderHome Lab – pfsense xml dump

This is the xml dump from my pfsense router. If you prefer to use a step by step guide to configure your pfsense router, see This Guide.

<?xml version="1.0"?>
<pfsense>
	<version>21.5</version>
	<lastchange></lastchange>
	<system>
		<optimization>normal</optimization>
		<hostname>pfSense</hostname>
		<domain>home.arpa</domain>
		<group>
			<name>all</name>
			<description><![CDATA[All Users]]></description>
			<scope>system</scope>
			<gid>1998</gid>
		</group>
		<group>
			<name>admins</name>
			<description><![CDATA[System Administrators]]></description>
			<scope>system</scope>
			<gid>1999</gid>
			<member>0</member>
			<priv>page-all</priv>
		</group>
		<user>
			<name>admin</name>
			<descr><![CDATA[System Administrator]]></descr>
			<scope>system</scope>
			<groupname>admins</groupname>
			<bcrypt-hash></bcrypt-hash>
			<uid>0</uid>
			<priv>user-shell-access</priv>
			<expires></expires>
			<dashboardcolumns>2</dashboardcolumns>
			<authorizedkeys></authorizedkeys>
			<ipsecpsk></ipsecpsk>
			<webguicss>pfSense.css</webguicss>
		</user>
		<nextuid>2000</nextuid>
		<nextgid>2000</nextgid>
		<timeservers>2.pfsense.pool.ntp.org</timeservers>
		<webgui>
			<protocol>https</protocol>
			<loginautocomplete></loginautocomplete>
			<ssl-certref></ssl-certref>
			<port></port>
			<max_procs>2</max_procs>
			<nodnsrebindcheck></nodnsrebindcheck>
			<dashboardcolumns>2</dashboardcolumns>
			<nohttpreferercheck></nohttpreferercheck>
			<webguicss>pfSense.css</webguicss>
			<logincss>1e3f75;</logincss>
		</webgui>
		<disablenatreflection>yes</disablenatreflection>
		<disablesegmentationoffloading></disablesegmentationoffloading>
		<disablelargereceiveoffloading></disablelargereceiveoffloading>
		<ipv6allow></ipv6allow>
		<maximumtableentries>400000</maximumtableentries>
		<powerd_ac_mode>hadp</powerd_ac_mode>
		<powerd_battery_mode>hadp</powerd_battery_mode>
		<powerd_normal_mode>hadp</powerd_normal_mode>
		<bogons>
			<interval>monthly</interval>
		</bogons>
		<hn_altq_enable></hn_altq_enable>
		<already_run_config_upgrade></already_run_config_upgrade>
		<ssh>
			<enable>enabled</enable>
		</ssh>
		<serialspeed>115200</serialspeed>
		<primaryconsole>serial</primaryconsole>
		<sshguard_threshold></sshguard_threshold>
		<sshguard_blocktime></sshguard_blocktime>
		<sshguard_detection_time></sshguard_detection_time>
		<sshguard_whitelist></sshguard_whitelist>
		<language>en_US</language>
		<timezone>US/Central</timezone>
		<dnsserver>192.168.3.6</dnsserver>
		<dnsallowoverride></dnsallowoverride>
		<dns1host>ns1.home.lab</dns1host>
		<acb>
			<enable>yes</enable>
			<hint></hint>
			<frequency>cron</frequency>
			<hour>23</hour>
			<month>*</month>
			<day>*</day>
			<dow>*</dow>
			<numman></numman>
			<encryption_password></encryption_password>
		</acb>
	</system>
	<interfaces>
		<wan>
			<enable></enable>
			<if>em0</if>
			<descr><![CDATA[Internet]]></descr>
			<spoofmac></spoofmac>
			<ipaddr>192.168.3.7</ipaddr>
			<subnet>24</subnet>
			<gateway>WANGW_2</gateway>
			<ipaddrv6></ipaddrv6>
			<subnetv6></subnetv6>
			<gatewayv6></gatewayv6>
		</wan>
		<lan>
			<descr><![CDATA[VLAN10]]></descr>
			<if>em1</if>
			<enable></enable>
			<ipaddr>10.10.1.1</ipaddr>
			<subnet>24</subnet>
			<spoofmac></spoofmac>
		</lan>
		<opt1>
			<descr><![CDATA[VLAN20VMOTION]]></descr>
			<if>em2</if>
			<enable></enable>
			<ipaddr>10.10.2.1</ipaddr>
			<subnet>24</subnet>
			<spoofmac></spoofmac>
		</opt1>
		<opt2>
			<descr><![CDATA[VLAN30VSAN]]></descr>
			<if>em3</if>
			<enable></enable>
			<ipaddr>10.10.3.1</ipaddr>
			<subnet>24</subnet>
			<spoofmac></spoofmac>
		</opt2>
		<opt3>
			<descr><![CDATA[VLAN40VMNETWORK]]></descr>
			<if>em4</if>
			<enable></enable>
			<ipaddr>10.10.4.1</ipaddr>
			<subnet>24</subnet>
			<spoofmac></spoofmac>
		</opt3>
		<opt4>
			<descr><![CDATA[VLAN50]]></descr>
			<if>em5</if>
			<enable></enable>
			<ipaddr>10.10.5.1</ipaddr>
			<subnet>24</subnet>
			<spoofmac></spoofmac>
		</opt4>
	</interfaces>
	<staticroutes></staticroutes>
	<dhcpd>
		<opt1>
			<range>
				<from>10.10.2.100</from>
				<to>10.10.2.150</to>
			</range>
			<enable></enable>
			<failover_peerip></failover_peerip>
			<defaultleasetime></defaultleasetime>
			<maxleasetime></maxleasetime>
			<netmask></netmask>
			<gateway></gateway>
			<domain>home.lab</domain>
			<domainsearchlist>home.lab</domainsearchlist>
			<ddnsdomain></ddnsdomain>
			<ddnsdomainprimary></ddnsdomainprimary>
			<ddnsdomainsecondary></ddnsdomainsecondary>
			<ddnsdomainkeyname></ddnsdomainkeyname>
			<ddnsdomainkeyalgorithm>hmac-md5</ddnsdomainkeyalgorithm>
			<ddnsdomainkey></ddnsdomainkey>
			<mac_allow></mac_allow>
			<mac_deny></mac_deny>
			<ddnsclientupdates>allow</ddnsclientupdates>
			<tftp></tftp>
			<ldap></ldap>
			<nextserver></nextserver>
			<filename></filename>
			<filename32></filename32>
			<filename64></filename64>
			<filename32arm></filename32arm>
			<filename64arm></filename64arm>
			<rootpath></rootpath>
			<numberoptions></numberoptions>
			<dhcpleaseinlocaltime></dhcpleaseinlocaltime>
			<dnsserver>192.168.3.6</dnsserver>
		</opt1>
		<opt2>
			<range>
				<from>10.10.3.100</from>
				<to>10.10.3.150</to>
			</range>
			<enable></enable>
			<failover_peerip></failover_peerip>
			<defaultleasetime></defaultleasetime>
			<maxleasetime></maxleasetime>
			<netmask></netmask>
			<gateway></gateway>
			<domain>home.lab</domain>
			<domainsearchlist>home.lab</domainsearchlist>
			<ddnsdomain></ddnsdomain>
			<ddnsdomainprimary></ddnsdomainprimary>
			<ddnsdomainsecondary></ddnsdomainsecondary>
			<ddnsdomainkeyname></ddnsdomainkeyname>
			<ddnsdomainkeyalgorithm>hmac-md5</ddnsdomainkeyalgorithm>
			<ddnsdomainkey></ddnsdomainkey>
			<mac_allow></mac_allow>
			<mac_deny></mac_deny>
			<ddnsclientupdates>allow</ddnsclientupdates>
			<tftp></tftp>
			<ldap></ldap>
			<nextserver></nextserver>
			<filename></filename>
			<filename32></filename32>
			<filename64></filename64>
			<filename32arm></filename32arm>
			<filename64arm></filename64arm>
			<rootpath></rootpath>
			<numberoptions></numberoptions>
			<dhcpleaseinlocaltime></dhcpleaseinlocaltime>
			<dnsserver>192.168.3.6</dnsserver>
		</opt2>
		<opt3>
			<range>
				<from>10.10.4.100</from>
				<to>10.10.4.150</to>
			</range>
			<enable></enable>
			<failover_peerip></failover_peerip>
			<defaultleasetime></defaultleasetime>
			<maxleasetime></maxleasetime>
			<netmask></netmask>
			<gateway></gateway>
			<domain>home.lab</domain>
			<domainsearchlist>home.lab</domainsearchlist>
			<ddnsdomain></ddnsdomain>
			<ddnsdomainprimary></ddnsdomainprimary>
			<ddnsdomainsecondary></ddnsdomainsecondary>
			<ddnsdomainkeyname></ddnsdomainkeyname>
			<ddnsdomainkeyalgorithm>hmac-md5</ddnsdomainkeyalgorithm>
			<ddnsdomainkey></ddnsdomainkey>
			<mac_allow></mac_allow>
			<mac_deny></mac_deny>
			<ddnsclientupdates>allow</ddnsclientupdates>
			<tftp></tftp>
			<ldap></ldap>
			<nextserver></nextserver>
			<filename></filename>
			<filename32></filename32>
			<filename64></filename64>
			<filename32arm></filename32arm>
			<filename64arm></filename64arm>
			<rootpath></rootpath>
			<numberoptions></numberoptions>
			<dhcpleaseinlocaltime></dhcpleaseinlocaltime>
			<dnsserver>192.168.3.6</dnsserver>
		</opt3>
		<opt4>
			<range>
				<from>10.10.5.100</from>
				<to>10.10.5.150</to>
			</range>
			<enable></enable>
			<failover_peerip></failover_peerip>
			<defaultleasetime></defaultleasetime>
			<maxleasetime></maxleasetime>
			<netmask></netmask>
			<gateway></gateway>
			<domain>home.lab</domain>
			<domainsearchlist>home.lab</domainsearchlist>
			<ddnsdomain></ddnsdomain>
			<ddnsdomainprimary></ddnsdomainprimary>
			<ddnsdomainsecondary></ddnsdomainsecondary>
			<ddnsdomainkeyname></ddnsdomainkeyname>
			<ddnsdomainkeyalgorithm>hmac-md5</ddnsdomainkeyalgorithm>
			<ddnsdomainkey></ddnsdomainkey>
			<mac_allow></mac_allow>
			<mac_deny></mac_deny>
			<ddnsclientupdates>allow</ddnsclientupdates>
			<tftp></tftp>
			<ldap></ldap>
			<nextserver></nextserver>
			<filename></filename>
			<filename32></filename32>
			<filename64></filename64>
			<filename32arm></filename32arm>
			<filename64arm></filename64arm>
			<rootpath></rootpath>
			<numberoptions></numberoptions>
			<dhcpleaseinlocaltime></dhcpleaseinlocaltime>
			<dnsserver>192.168.3.6</dnsserver>
		</opt4>
		<lan>
			<range>
				<from>10.10.1.100</from>
				<to>10.10.1.150</to>
			</range>
			<enable></enable>
			<failover_peerip></failover_peerip>
			<defaultleasetime></defaultleasetime>
			<maxleasetime></maxleasetime>
			<netmask></netmask>
			<dnsserver>192.168.3.6</dnsserver>
			<gateway></gateway>
			<domain>home.lab</domain>
			<domainsearchlist>home.lab</domainsearchlist>
			<ddnsdomain></ddnsdomain>
			<ddnsdomainprimary></ddnsdomainprimary>
			<ddnsdomainsecondary></ddnsdomainsecondary>
			<ddnsdomainkeyname></ddnsdomainkeyname>
			<ddnsdomainkeyalgorithm>hmac-md5</ddnsdomainkeyalgorithm>
			<ddnsdomainkey></ddnsdomainkey>
			<mac_allow></mac_allow>
			<mac_deny></mac_deny>
			<ddnsclientupdates>allow</ddnsclientupdates>
			<tftp></tftp>
			<ldap></ldap>
			<nextserver></nextserver>
			<filename></filename>
			<filename32></filename32>
			<filename64></filename64>
			<filename32arm></filename32arm>
			<filename64arm></filename64arm>
			<rootpath></rootpath>
			<numberoptions></numberoptions>
			<dhcpleaseinlocaltime></dhcpleaseinlocaltime>
		</lan>
	</dhcpd>
	<dhcpdv6></dhcpdv6>
	<snmpd>
		<syslocation></syslocation>
		<syscontact></syscontact>
		<rocommunity>public</rocommunity>
	</snmpd>
	<diag>
		<ipv6nat>
			<ipaddr></ipaddr>
		</ipv6nat>
	</diag>
	<syslog>
		<filterdescriptions>1</filterdescriptions>
	</syslog>
	<filter>
		<rule>
			<id></id>
			<tracker>1625536415</tracker>
			<type>pass</type>
			<interface>wan</interface>
			<ipprotocol>inet</ipprotocol>
			<tag></tag>
			<tagged></tagged>
			<max></max>
			<max-src-nodes></max-src-nodes>
			<max-src-conn></max-src-conn>
			<max-src-states></max-src-states>
			<statetimeout></statetimeout>
			<statetype><![CDATA[keep state]]></statetype>
			<os></os>
			<protocol>tcp</protocol>
			<source>
				<any></any>
			</source>
			<destination>
				<network>(self)</network>
				<port>22</port>
			</destination>
			<descr><![CDATA[Allow 22 to pfSense]]></descr>
			<created>
				<time>1625536415</time>
				<username><![CDATA[admin@192.168.2.94 (Local Database)]]></username>
			</created>
			<updated>
				<time>1625536955</time>
				<username><![CDATA[admin@192.168.2.94 (Local Database)]]></username>
			</updated>
		</rule>
		<rule>
			<id></id>
			<tracker>1625535224</tracker>
			<type>pass</type>
			<interface>wan</interface>
			<ipprotocol>inet</ipprotocol>
			<tag></tag>
			<tagged></tagged>
			<max></max>
			<max-src-nodes></max-src-nodes>
			<max-src-conn></max-src-conn>
			<max-src-states></max-src-states>
			<statetimeout></statetimeout>
			<statetype><![CDATA[keep state]]></statetype>
			<os></os>
			<protocol>icmp</protocol>
			<icmptype>any</icmptype>
			<source>
				<any></any>
			</source>
			<destination>
				<network>(self)</network>
			</destination>
			<descr><![CDATA[Allow Ping to pfSense]]></descr>
			<updated>
				<time>1625535224</time>
				<username><![CDATA[admin@192.168.2.94 (Local Database)]]></username>
			</updated>
			<created>
				<time>1625535224</time>
				<username><![CDATA[admin@192.168.2.94 (Local Database)]]></username>
			</created>
		</rule>
		<rule>
			<id></id>
			<tracker>1625535067</tracker>
			<type>pass</type>
			<interface>wan</interface>
			<ipprotocol>inet</ipprotocol>
			<tag></tag>
			<tagged></tagged>
			<max></max>
			<max-src-nodes></max-src-nodes>
			<max-src-conn></max-src-conn>
			<max-src-states></max-src-states>
			<statetimeout></statetimeout>
			<statetype><![CDATA[keep state]]></statetype>
			<os></os>
			<protocol>tcp</protocol>
			<source>
				<any></any>
			</source>
			<destination>
				<network>(self)</network>
				<port>443</port>
			</destination>
			<descr><![CDATA[Allow 443 to pfSense]]></descr>
			<updated>
				<time>1625535067</time>
				<username><![CDATA[admin@192.168.2.94 (Local Database)]]></username>
			</updated>
			<created>
				<time>1625535067</time>
				<username><![CDATA[admin@192.168.2.94 (Local Database)]]></username>
			</created>
		</rule>
		<rule>
			<id></id>
			<tracker>1625673442</tracker>
			<type>pass</type>
			<interface>wan</interface>
			<ipprotocol>inet</ipprotocol>
			<tag></tag>
			<tagged></tagged>
			<max></max>
			<max-src-nodes></max-src-nodes>
			<max-src-conn></max-src-conn>
			<max-src-states></max-src-states>
			<statetimeout></statetimeout>
			<statetype><![CDATA[keep state]]></statetype>
			<os></os>
			<source>
				<any></any>
			</source>
			<destination>
				<any></any>
			</destination>
			<descr></descr>
			<updated>
				<time>1625673442</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</updated>
			<created>
				<time>1625673442</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</created>
		</rule>
		<rule>
			<type>pass</type>
			<ipprotocol>inet</ipprotocol>
			<descr><![CDATA[Default allow LAN to any rule]]></descr>
			<interface>lan</interface>
			<tracker>0100000101</tracker>
			<source>
				<network>lan</network>
			</source>
			<destination>
				<any></any>
			</destination>
		</rule>
		<rule>
			<type>pass</type>
			<ipprotocol>inet6</ipprotocol>
			<descr><![CDATA[Default allow LAN IPv6 to any rule]]></descr>
			<interface>lan</interface>
			<tracker>0100000102</tracker>
			<source>
				<network>lan</network>
			</source>
			<destination>
				<any></any>
			</destination>
		</rule>
		<rule>
			<id></id>
			<tracker>1626104310</tracker>
			<type>pass</type>
			<interface>opt1</interface>
			<ipprotocol>inet</ipprotocol>
			<tag></tag>
			<tagged></tagged>
			<max></max>
			<max-src-nodes></max-src-nodes>
			<max-src-conn></max-src-conn>
			<max-src-states></max-src-states>
			<statetimeout></statetimeout>
			<statetype><![CDATA[keep state]]></statetype>
			<os></os>
			<source>
				<network>opt1</network>
			</source>
			<destination>
				<any></any>
			</destination>
			<descr><![CDATA[Default allow to any rule]]></descr>
			<updated>
				<time>1626104310</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</updated>
			<created>
				<time>1626104310</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</created>
		</rule>
		<rule>
			<id></id>
			<tracker>1626104295</tracker>
			<type>pass</type>
			<interface>opt2</interface>
			<ipprotocol>inet</ipprotocol>
			<tag></tag>
			<tagged></tagged>
			<max></max>
			<max-src-nodes></max-src-nodes>
			<max-src-conn></max-src-conn>
			<max-src-states></max-src-states>
			<statetimeout></statetimeout>
			<statetype><![CDATA[keep state]]></statetype>
			<os></os>
			<source>
				<network>opt2</network>
			</source>
			<destination>
				<any></any>
			</destination>
			<descr><![CDATA[Default allow to any rule]]></descr>
			<updated>
				<time>1626104295</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</updated>
			<created>
				<time>1626104295</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</created>
		</rule>
		<rule>
			<id></id>
			<tracker>1625687815</tracker>
			<type>pass</type>
			<interface>opt3</interface>
			<ipprotocol>inet</ipprotocol>
			<tag></tag>
			<tagged></tagged>
			<max></max>
			<max-src-nodes></max-src-nodes>
			<max-src-conn></max-src-conn>
			<max-src-states></max-src-states>
			<statetimeout></statetimeout>
			<statetype><![CDATA[keep state]]></statetype>
			<os></os>
			<source>
				<network>opt3</network>
			</source>
			<destination>
				<any></any>
			</destination>
			<descr><![CDATA[Default allow LAN to any rule]]></descr>
			<updated>
				<time>1625687815</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</updated>
			<created>
				<time>1625687815</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</created>
		</rule>
		<rule>
			<id></id>
			<tracker>1626104265</tracker>
			<type>pass</type>
			<interface>opt4</interface>
			<ipprotocol>inet</ipprotocol>
			<tag></tag>
			<tagged></tagged>
			<max></max>
			<max-src-nodes></max-src-nodes>
			<max-src-conn></max-src-conn>
			<max-src-states></max-src-states>
			<statetimeout></statetimeout>
			<statetype><![CDATA[keep state]]></statetype>
			<os></os>
			<source>
				<network>opt4</network>
			</source>
			<destination>
				<any></any>
			</destination>
			<descr><![CDATA[Default allow to any rule]]></descr>
			<updated>
				<time>1626104265</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</updated>
			<created>
				<time>1626104265</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</created>
		</rule>
		<separator>
			<wan></wan>
			<lan></lan>
			<opt3></opt3>
			<opt4></opt4>
			<opt2></opt2>
			<opt1></opt1>
		</separator>
	</filter>
	<ipsec></ipsec>
	<aliases></aliases>
	<proxyarp></proxyarp>
	<cron>
		<item>
			<minute>*/1</minute>
			<hour>*</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/usr/sbin/newsyslog</command>
		</item>
		<item>
			<minute>1</minute>
			<hour>3</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/etc/rc.periodic daily</command>
		</item>
		<item>
			<minute>15</minute>
			<hour>4</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>6</wday>
			<who>root</who>
			<command>/etc/rc.periodic weekly</command>
		</item>
		<item>
			<minute>30</minute>
			<hour>5</hour>
			<mday>1</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/etc/rc.periodic monthly</command>
		</item>
		<item>
			<minute>1,31</minute>
			<hour>0-5</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/usr/bin/nice -n20 adjkerntz -a</command>
		</item>
		<item>
			<minute>1</minute>
			<hour>3</hour>
			<mday>1</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/usr/bin/nice -n20 /etc/rc.update_bogons.sh</command>
		</item>
		<item>
			<minute>1</minute>
			<hour>1</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/usr/bin/nice -n20 /etc/rc.dyndns.update</command>
		</item>
		<item>
			<minute>*/60</minute>
			<hour>*</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot</command>
		</item>
		<item>
			<minute>30</minute>
			<hour>12</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/usr/bin/nice -n20 /etc/rc.update_urltables</command>
		</item>
		<item>
			<minute>1</minute>
			<hour>0</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/usr/bin/nice -n20 /etc/rc.update_pkg_metadata</command>
		</item>
		<item>
			<minute>0</minute>
			<hour>23</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/usr/bin/nice -n20 /usr/local/bin/php /usr/local/sbin/execacb.php</command>
		</item>
	</cron>
	<wol></wol>
	<rrd>
		<enable></enable>
	</rrd>
	<widgets>
		<sequence>system_information:col1:open:0,interfaces:col2:open:0</sequence>
		<period>10</period>
	</widgets>
	<openvpn></openvpn>
	<dnshaper></dnshaper>
	<unbound>
		<enable></enable>
		<dnssec></dnssec>
		<active_interface></active_interface>
		<outgoing_interface></outgoing_interface>
		<custom_options></custom_options>
		<hideidentity></hideidentity>
		<hideversion></hideversion>
		<dnssecstripped></dnssecstripped>
	</unbound>
	<revision>
		<time>1626966645</time>
		<description><![CDATA[admin@192.168.3.155 (Local Database): main]]></description>
		<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
	</revision>
	<ppps></ppps>
	<gateways>
		<gateway_item>
			<interface>wan</interface>
			<gateway>192.168.3.1</gateway>
			<name>WANGW_2</name>
			<weight>1</weight>
			<ipprotocol>inet</ipprotocol>
			<descr><![CDATA[Interface wan Gateway]]></descr>
		</gateway_item>
		<defaultgw4>WANGW_2</defaultgw4>
		<defaultgw6></defaultgw6>
	</gateways>
	<cert>
		<refid></refid>
		<descr><![CDATA[webConfigurator default ()]]></descr>
		<type>server</type>
		<crt></crt>
		<prv></prv>
	</cert>
	<installedpackages>
		<package>
			<name>Open-VM-Tools</name>
			<descr><![CDATA[VMware Tools is a suite of utilities that enhances the performance of the virtual machine's guest operating system and improves management of the virtual machine.]]></descr>
			<website>http://open-vm-tools.sourceforge.net/</website>
			<version>10.1.0_5,1</version>
			<pkginfolink>https://docs.netgate.com/pfsense/en/latest/packages/open-vm-tools.html</pkginfolink>
			<configurationfile>open-vm-tools.xml</configurationfile>
			<logging>
				<logfilename>vmware-vmsvc-root.log</logfilename>
			</logging>
			<include_file>/usr/local/pkg/open-vm-tools.inc</include_file>
		</package>
		<service>
			<name>vmware-guestd</name>
			<rcfile>vmware-guestd.sh</rcfile>
			<custom_php_service_status_command>mwexec(&quot;/usr/local/etc/rc.d/vmware-guestd status&quot;) == 0;</custom_php_service_status_command>
			<description><![CDATA[VMware Guest Daemon]]></description>
		</service>
		<service>
			<name>vmware-kmod</name>
			<rcfile>vmware-kmod.sh</rcfile>
			<custom_php_service_status_command>mwexec(&quot;/usr/local/etc/rc.d/vmware-kmod status&quot;) == 0;</custom_php_service_status_command>
			<description><![CDATA[VMware Kernel Modules]]></description>
		</service>
	</installedpackages>
	<vlans></vlans>
	<shaper></shaper>
</pfsense>

bookmark_borderHome Lab – Configuring the baremetal ESXi7

So now that we have our SuperMicro server setup, and we’ve installed ESXi7, the next step is to do some basic configuration on our BareMetal ESXi7. We will setup the network, view the storage, and prepare the esxi for the nested esxi environment.

Log in to the UI of the baremetal Esxi device, and you’ll see a screen like this

Network Configuration

Navigate to the Networking tab, and select physical Nics. As you can see I have 4 physical NICs on my SuperMicro, and I have my ethernet plugged into vmnic1.
Navigate to the Virtual Switches tab, we will need to create 1 more virtual switch.
Click “Add standard virtual switch” and configure using the details above. Make sure to open the Security tab and Accept all the options: Promiscuous Mode, MAC address changes, and Forged transmits. 
Click add and you will see a screen like this.
Navigate to the port groups tab, it’s time to create our networks.
Create the first network with the options listed above. We’re going to repeat these steps to create 4 more networks.
Here’s a list of the networks that you should have. Note that the vSwitch is vSwitch1 for vlan10-50. I’m not doing vlan tagging in my setup, feel free to do it on your end if you prefer.

Storage Configuration

You should see your SSD here. I have a 2GB SSD, but you should see your physical disk listed here.

This is basically the whole setup for the baremetal. Next step will be deploying the esxi vms.