bookmark_borderTanzu Kubernetes on vCenter 7 – Deploy Guest Cluster

Now that we have the supervisor cluster up and running and our namespace created, we can deploy a guest cluster via the CLI. I installed an ubuntu 20 vm in vCenter for use as my jumpbox. I installed kubectl and the vsphere plugin in this environment. There are windows plugins and plugins for all the linux distros as well.

Install kubectl and vsphere plugin on jump server

Kubectl

You can download and install kubectl very easily (in linux) with these commands:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo mv kubectl /usr/local/bin/kubectl
sudo chmod +x /usr/local/bin/kubectl
kubectl version

vSphere Plugin

The easiest way to download this is to navigate to the supervisor cluster’s floating IP address. In my case it is 10.10.4.13

Select your OS and download the cli plugin. Upload to your jump box, and running the following commands.
sudo mv kubectl-vsphere /usr/local/bin/kubectl-vsphere
sudo chmod +x /usr/local/bin/kubectl-vsphere
kubectl vsphere

Login to Supervisor Cluster

OPTIONAL: Set Environment variable for vsphere password.

echo "KUBECTL_VSPHERE_PASSWORD='supersecretpassword123'" >> /etc/environment

Log into supervisor cluster and verify cluster is healthy

kubectl vsphere login --vsphere-username administrator@vsphere.local --server=https://10.10.4.50 --insecure-skip-tls-verify
kubectl config use-context dev
kubectl get pods --all-namespaces ### Should see a list of all the pods running
kubectl get nodes ### Everything should show Ready
kubectl get tanzukubernetesreleases ###Checkout the latest releases

Create yaml file to build guest cluster

Create a file called guest_cluster.yaml with the following content

---
apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TanzuKubernetesCluster
metadata:
  name: tkg-cluster-01
spec:
  topology:
    controlPlane:
      count: 1
      class: best-effort-small
      storageClass: vsan-default-storage-policy
    workers:
      count: 2
      class: best-effort-small
      storageClass: vsan-default-storage-policy
  distribution:
    version: v1.20.7

Deploy TKGs guest cluster

kubectl apply -f guest_cluster_tkgs.yaml
kubectl get cluster ### View the cluster status
kubectl get tanzukubernetescluster ### View the cluster status
Guest cluster is deploying
Still provisioning the cluster
Should be able to see the new vms spinning up in vCenter

bookmark_borderTanzu Kubernetes on vCenter 7 ā€“ Namespace Setup

After deploying the supervisor cluster, the next step is to setup the namespace where we will deploy our guest cluster.

Create Namespace

In the Menu click Workload Management. Then navigate to the Namespaces tab. Click Create Namespace.
Select the vSAN cluster and choose a name, I’m using “dev” Then select the workload network and finally add a description.
You should see the config status “Running” and kubernetes status “Active”. We need to configure Permissions, storage, capacity and usage, and the associated vm classes and content libraries for this namespace.

Configure Dev Namespace

Click Permissions and configure as shown above. If you’re using a different user, you can configure that here. I’m simply using the administrator for all access. Click OK.
Select Storage and choose the vSAN Default Storage Policy. Click OK.
Under Capacity and Usage, configure as shown above. I’m setting limits on memory and storage, but not CPU.
In the VM Service section, click Add VM Class and select the “best-effort-small” class. This will provide enough cpu and memory to the vms to handle a few deployments. If you need more, “best-effort_medium” would be a good fit as well.
Lastly, select the Add Content Library under VM Service, and add the kubernetes library.
Finished setup will look something like this.

bookmark_borderTanzu Kubernetes Content Library in vCenter 7

Before you can setup workload management in vCenter 7, you need to create a content library and setup the subscription to point to vmware’s library.

Add Content Library

Select the Menu and navigate to Content Libraries
Add a name for the content library and click Next.
Select Subscribed content library and add this Subscription URL: https://wp-content.vmware.com/v2/latest/lib.json
Additionally if you want to save space, select Download content when needed.
Yes to bypass the certificate warning.
Select the storage location, i’m using the vSAN datastore.
Review the summary page and click Finish.

bookmark_borderTanzu Kubernetes on vCenter 7 – Deploy Supervisor Cluster (WCP)

In this guide we will configure Workload Management for vCenter 7. We’ll be using vCenter Server Network (DSwitches) instead of NSX-T. Additionally we’ll be using the Avi Load Balancer (NSX Advanced Load Balancer).

Licensing for Supervisor Cluster

Right click on your vSAN cluster and navigate to Licensing. Select Assign Supervisor Cluster License and select the appropriate license. If you need to add a new license select Menu at the top -> Administration -> Licenses -> Add

Configuring Workload Management

Click the Menu and navigate to Workload Management, and you should see this page. (Assuming you licensed correctly). Click Get Started.
This alert is just informing you that Avi must already be preconfigured. If you haven’t done so yet, please do so now. Additionally we do not have NSX-T running in this lab, so vCenter Server Network is selected. Click Next.
Select the vSAN Cluster and click Next.
Pick the control plane size. I have found that Tiny was more than enough for my needs.
Select the default storage policy for control plane. I am using the vSAN Default Storage Policy. Click Next
Add in the details for the Avi load balancer. The name must be DNS compliant, so avi-controller-1 is simple and works.
Type: Avi
Controller IP: Use the IP and port here
Then add your username and password.
Add your Avi Controller Cert here as well. If you haven’t generated this yet, please do so now.
Again, as with everything VMware, make sure DNS works!

I’m using the 10.10.4.0/24 network for my management network. Select your starting range in that network and add your gateway. Add the dns server, search domain, and ntp server. Click Next.
Add in the Pod network (Workload Network) 10.10.5.0/24 is the network i’m using. Add the dns server then click Add for workload network.
In the popup add a name for the network and select the PodNetwork portgroup. Lastly add the gateway, subnet, and ip ranges. Click Save.
Everything should look like this. Click Next.
Select the kubernetes content library we created. Click Next.
All set! Click Finish.
You should see this screen. At this point go grab some coffee because this step takes quite a while, specially if your content library is set to “Download library content only when needed,” as mine is. It will download all the required ovas and start spinning up the supervisor cluster.
After a while (~45min for me) you should see your supervisor cluster up and running!
You can click the Menu and navigate to VMs and Templates and there should be 3 supervisor control plane vms running.

bookmark_borderDeploy vCenter 7 and vSAN on nested ESXi

If you’ve followed the guide this far, you’ve deployed 3 esxi hosts nested on your baremetal esxi install. This guide takes it a step further by deploying vcenter and creating a vSAN cluster on the esxi hosts.

Download vCenter Server

Login to your account at https://my.vmware.com/ and go to Products and Accounts ->Products -> All Products

Select VMware vSphere. View Download Components
Select your version and download the vCenter Server. I’m using VMware vCenter Server 7.0U2b with Enterprise Plus
Download the VMware vCenter Server Appliance (7.5GB)
VMware-VCSA-all-7.0.2-17958471.iso

Mount the ISO and use the install wizard to configure vCenter 7

I’m using windows 10, and it was relatively easy to mount the ISO. In Windows explorer, I just navigated over to the downloads directly where the ISO was, and double clicked it. Open the directory vcsa-ui-installer -> win32 -> installer.exe.

Stage 1


You should see a popup like this. Go ahead and click Install
Click Next to deploy vCenter server
Accept the EUL.
Put the IP/fqdn of the first esxi host, and the credentials.
Select Yes to accept the warning.
Specify a name for the vm, and set the root password
For deployment size I chose Tiny since it more than met my needs. If you need more, select Small.
Select “Install on a new vSAN cluster containing the target host” Feel free to modify the names.
We’re going to claim all the 200GB disks as capacity tier, and the 20GB disk as cache tier. The other disk we will not use. Additionally, I selected “Enable Thin Disk Mode” and “Enable Deduplication and compression” Since it’s a lab, i’m not too worried about a vSAN failure. Worst case, i’ll just rebuild the entire lab and get more practice.
Set the fqdn for vcenter, IP address and mask, default gateway, and dns server. vCenter is very picky about dns.. Make sure that the fqdn resolves and the ip address reverse lookup resolves as well.
Here’s the summary page. Go ahead and hit Finish then grab some coffee. This step takes a while.
Congratulations! It’s installed. Now onto stage 2 for some additional configuration. Click Continue

Stage 2

Into the setup wizard for stage 2. Click Next
You’re welcome to sync with a public ntp server (or private), I just selected the host for mine. Additionally, it’s a lab and ssh access to vCenter is very handy when troubleshooting issues later.
Set the SSO domain, I chose the default “vsphere.local” and enter your password.
Choose whether you want to join the CEIP.
Summary page. If all looks right, click Finish.
Stage 2 Completed. Vcenter is all setup. You can now access the UI:

vSAN Initial Setup

Launch vCenter in the browser.
Login with the administrator@vsphere.local account and password.
You’re going to see lots of alarms and warnings, don’t worry.. we’re going to fix it all in the next few steps.

Step 1: Cluster Basics

Navigate to the vSAN Cluster and select Configure. Then under Configuration click Quickstart. This provides a easy to use wizard for deploying HA and vSAN.
Step 1: Click Edit under Cluster Basics and make sure that all the options are turned on. vSphere DRS, vSphere HA, and vSAN.

Step 2: Add Hosts

Step 2: Under Add hosts, click ADD. Then add in the IP or fqdn of each of the other 2 esxi hosts. Then the user and passwords for each.
Select the hosts and click OK to accept the certificate security warning.
Summary of the hosts. Click Next
Ready to add them, click Finish. After you click finish, this will take some time. Just be patient.
Hosts are added, now on to step 3.

Step 3: Configure Hosts

Step 3: Click Configure under Configure cluster. I left all these settings default.
Set the vmnics as shown above. We will use this to setup vSAN and vMotion. Click Next.
I am using vlan20 (10.10.2.0/24) for my vmotion traffic. So I configured 3 interfaces for this traffic, 1 per esxi in the cluster. Also i’m not using vlans, so I have unchecked that box. Click Next.
Similarly, the vSAN vlan is vlan30 (10.10.3.0/24) and I configured 3 IP addresses on this network. Uncheck vlan if not in use. Click Next.
I left all of these settings default. You can turn on “Virtual Machine Monitoring” if you want. Everything else is fine as default. Click Next.
For the disks, select the “Group by:” as Host, and expand the hosts. You will see all the volumes that we created on the esxi setup.
Go through and claim the following:
200GB Claim as capacity tier
20GB Claim as cache tier
4GB do not claim
Click Next.
Skip this step, since we have already configured internet access.
Summary page. Review everything and click Finish. This step takes a while.. be patient.
Eventually everything will normalize and look like this. You can ignore those yellow alerts. As long as nothing is red, you will be fine.

Licensing vCenter

Click the Menu and navigate to Hosts and Clusters. Right click the vcenter1 instance, and select Assign License. Select the appropriate vCenter license and click Ok.

Licensing the ESXi hosts in vSAN cluster

Enter your licenses separated out by a new line. Click Next. Then you have the option to name your licenses, Next. Summary page. Save.
After adding the licenses, you will see them available here.
Select the Menu at the top and navigate to Hosts and Clusters
Right click on one of the ESXi hosts in the vSAN cluster and select Assign License. Then in the popup that will appear, you select the appropriate license. Repeat this steps for the other 2 ESXi hosts.

Set vSAN as default storage policy

Right click on the vCenter vm, and navigate to VM Policies, and select Edit VM Storage Policies
At the top select vSAN Default Storage Policy from the drop down and click Ok.

bookmark_borderpfsense – Home Lab xml dump

This is the xml dump from my pfsense router. If you prefer to use a step by step guide to configure your pfsense router, see This Guide.

<?xml version="1.0"?>
<pfsense>
	<version>21.5</version>
	<lastchange></lastchange>
	<system>
		<optimization>normal</optimization>
		<hostname>pfSense</hostname>
		<domain>home.arpa</domain>
		<group>
			<name>all</name>
			<description><![CDATA[All Users]]></description>
			<scope>system</scope>
			<gid>1998</gid>
		</group>
		<group>
			<name>admins</name>
			<description><![CDATA[System Administrators]]></description>
			<scope>system</scope>
			<gid>1999</gid>
			<member>0</member>
			<priv>page-all</priv>
		</group>
		<user>
			<name>admin</name>
			<descr><![CDATA[System Administrator]]></descr>
			<scope>system</scope>
			<groupname>admins</groupname>
			<bcrypt-hash></bcrypt-hash>
			<uid>0</uid>
			<priv>user-shell-access</priv>
			<expires></expires>
			<dashboardcolumns>2</dashboardcolumns>
			<authorizedkeys></authorizedkeys>
			<ipsecpsk></ipsecpsk>
			<webguicss>pfSense.css</webguicss>
		</user>
		<nextuid>2000</nextuid>
		<nextgid>2000</nextgid>
		<timeservers>2.pfsense.pool.ntp.org</timeservers>
		<webgui>
			<protocol>https</protocol>
			<loginautocomplete></loginautocomplete>
			<ssl-certref></ssl-certref>
			<port></port>
			<max_procs>2</max_procs>
			<nodnsrebindcheck></nodnsrebindcheck>
			<dashboardcolumns>2</dashboardcolumns>
			<nohttpreferercheck></nohttpreferercheck>
			<webguicss>pfSense.css</webguicss>
			<logincss>1e3f75;</logincss>
		</webgui>
		<disablenatreflection>yes</disablenatreflection>
		<disablesegmentationoffloading></disablesegmentationoffloading>
		<disablelargereceiveoffloading></disablelargereceiveoffloading>
		<ipv6allow></ipv6allow>
		<maximumtableentries>400000</maximumtableentries>
		<powerd_ac_mode>hadp</powerd_ac_mode>
		<powerd_battery_mode>hadp</powerd_battery_mode>
		<powerd_normal_mode>hadp</powerd_normal_mode>
		<bogons>
			<interval>monthly</interval>
		</bogons>
		<hn_altq_enable></hn_altq_enable>
		<already_run_config_upgrade></already_run_config_upgrade>
		<ssh>
			<enable>enabled</enable>
		</ssh>
		<serialspeed>115200</serialspeed>
		<primaryconsole>serial</primaryconsole>
		<sshguard_threshold></sshguard_threshold>
		<sshguard_blocktime></sshguard_blocktime>
		<sshguard_detection_time></sshguard_detection_time>
		<sshguard_whitelist></sshguard_whitelist>
		<language>en_US</language>
		<timezone>US/Central</timezone>
		<dnsserver>192.168.3.6</dnsserver>
		<dnsallowoverride></dnsallowoverride>
		<dns1host>ns1.home.lab</dns1host>
		<acb>
			<enable>yes</enable>
			<hint></hint>
			<frequency>cron</frequency>
			<hour>23</hour>
			<month>*</month>
			<day>*</day>
			<dow>*</dow>
			<numman></numman>
			<encryption_password></encryption_password>
		</acb>
	</system>
	<interfaces>
		<wan>
			<enable></enable>
			<if>em0</if>
			<descr><![CDATA[Internet]]></descr>
			<spoofmac></spoofmac>
			<ipaddr>192.168.3.7</ipaddr>
			<subnet>24</subnet>
			<gateway>WANGW_2</gateway>
			<ipaddrv6></ipaddrv6>
			<subnetv6></subnetv6>
			<gatewayv6></gatewayv6>
		</wan>
		<lan>
			<descr><![CDATA[VLAN10]]></descr>
			<if>em1</if>
			<enable></enable>
			<ipaddr>10.10.1.1</ipaddr>
			<subnet>24</subnet>
			<spoofmac></spoofmac>
		</lan>
		<opt1>
			<descr><![CDATA[VLAN20VMOTION]]></descr>
			<if>em2</if>
			<enable></enable>
			<ipaddr>10.10.2.1</ipaddr>
			<subnet>24</subnet>
			<spoofmac></spoofmac>
		</opt1>
		<opt2>
			<descr><![CDATA[VLAN30VSAN]]></descr>
			<if>em3</if>
			<enable></enable>
			<ipaddr>10.10.3.1</ipaddr>
			<subnet>24</subnet>
			<spoofmac></spoofmac>
		</opt2>
		<opt3>
			<descr><![CDATA[VLAN40VMNETWORK]]></descr>
			<if>em4</if>
			<enable></enable>
			<ipaddr>10.10.4.1</ipaddr>
			<subnet>24</subnet>
			<spoofmac></spoofmac>
		</opt3>
		<opt4>
			<descr><![CDATA[VLAN50]]></descr>
			<if>em5</if>
			<enable></enable>
			<ipaddr>10.10.5.1</ipaddr>
			<subnet>24</subnet>
			<spoofmac></spoofmac>
		</opt4>
	</interfaces>
	<staticroutes></staticroutes>
	<dhcpd>
		<opt1>
			<range>
				<from>10.10.2.100</from>
				<to>10.10.2.150</to>
			</range>
			<enable></enable>
			<failover_peerip></failover_peerip>
			<defaultleasetime></defaultleasetime>
			<maxleasetime></maxleasetime>
			<netmask></netmask>
			<gateway></gateway>
			<domain>home.lab</domain>
			<domainsearchlist>home.lab</domainsearchlist>
			<ddnsdomain></ddnsdomain>
			<ddnsdomainprimary></ddnsdomainprimary>
			<ddnsdomainsecondary></ddnsdomainsecondary>
			<ddnsdomainkeyname></ddnsdomainkeyname>
			<ddnsdomainkeyalgorithm>hmac-md5</ddnsdomainkeyalgorithm>
			<ddnsdomainkey></ddnsdomainkey>
			<mac_allow></mac_allow>
			<mac_deny></mac_deny>
			<ddnsclientupdates>allow</ddnsclientupdates>
			<tftp></tftp>
			<ldap></ldap>
			<nextserver></nextserver>
			<filename></filename>
			<filename32></filename32>
			<filename64></filename64>
			<filename32arm></filename32arm>
			<filename64arm></filename64arm>
			<rootpath></rootpath>
			<numberoptions></numberoptions>
			<dhcpleaseinlocaltime></dhcpleaseinlocaltime>
			<dnsserver>192.168.3.6</dnsserver>
		</opt1>
		<opt2>
			<range>
				<from>10.10.3.100</from>
				<to>10.10.3.150</to>
			</range>
			<enable></enable>
			<failover_peerip></failover_peerip>
			<defaultleasetime></defaultleasetime>
			<maxleasetime></maxleasetime>
			<netmask></netmask>
			<gateway></gateway>
			<domain>home.lab</domain>
			<domainsearchlist>home.lab</domainsearchlist>
			<ddnsdomain></ddnsdomain>
			<ddnsdomainprimary></ddnsdomainprimary>
			<ddnsdomainsecondary></ddnsdomainsecondary>
			<ddnsdomainkeyname></ddnsdomainkeyname>
			<ddnsdomainkeyalgorithm>hmac-md5</ddnsdomainkeyalgorithm>
			<ddnsdomainkey></ddnsdomainkey>
			<mac_allow></mac_allow>
			<mac_deny></mac_deny>
			<ddnsclientupdates>allow</ddnsclientupdates>
			<tftp></tftp>
			<ldap></ldap>
			<nextserver></nextserver>
			<filename></filename>
			<filename32></filename32>
			<filename64></filename64>
			<filename32arm></filename32arm>
			<filename64arm></filename64arm>
			<rootpath></rootpath>
			<numberoptions></numberoptions>
			<dhcpleaseinlocaltime></dhcpleaseinlocaltime>
			<dnsserver>192.168.3.6</dnsserver>
		</opt2>
		<opt3>
			<range>
				<from>10.10.4.100</from>
				<to>10.10.4.150</to>
			</range>
			<enable></enable>
			<failover_peerip></failover_peerip>
			<defaultleasetime></defaultleasetime>
			<maxleasetime></maxleasetime>
			<netmask></netmask>
			<gateway></gateway>
			<domain>home.lab</domain>
			<domainsearchlist>home.lab</domainsearchlist>
			<ddnsdomain></ddnsdomain>
			<ddnsdomainprimary></ddnsdomainprimary>
			<ddnsdomainsecondary></ddnsdomainsecondary>
			<ddnsdomainkeyname></ddnsdomainkeyname>
			<ddnsdomainkeyalgorithm>hmac-md5</ddnsdomainkeyalgorithm>
			<ddnsdomainkey></ddnsdomainkey>
			<mac_allow></mac_allow>
			<mac_deny></mac_deny>
			<ddnsclientupdates>allow</ddnsclientupdates>
			<tftp></tftp>
			<ldap></ldap>
			<nextserver></nextserver>
			<filename></filename>
			<filename32></filename32>
			<filename64></filename64>
			<filename32arm></filename32arm>
			<filename64arm></filename64arm>
			<rootpath></rootpath>
			<numberoptions></numberoptions>
			<dhcpleaseinlocaltime></dhcpleaseinlocaltime>
			<dnsserver>192.168.3.6</dnsserver>
		</opt3>
		<opt4>
			<range>
				<from>10.10.5.100</from>
				<to>10.10.5.150</to>
			</range>
			<enable></enable>
			<failover_peerip></failover_peerip>
			<defaultleasetime></defaultleasetime>
			<maxleasetime></maxleasetime>
			<netmask></netmask>
			<gateway></gateway>
			<domain>home.lab</domain>
			<domainsearchlist>home.lab</domainsearchlist>
			<ddnsdomain></ddnsdomain>
			<ddnsdomainprimary></ddnsdomainprimary>
			<ddnsdomainsecondary></ddnsdomainsecondary>
			<ddnsdomainkeyname></ddnsdomainkeyname>
			<ddnsdomainkeyalgorithm>hmac-md5</ddnsdomainkeyalgorithm>
			<ddnsdomainkey></ddnsdomainkey>
			<mac_allow></mac_allow>
			<mac_deny></mac_deny>
			<ddnsclientupdates>allow</ddnsclientupdates>
			<tftp></tftp>
			<ldap></ldap>
			<nextserver></nextserver>
			<filename></filename>
			<filename32></filename32>
			<filename64></filename64>
			<filename32arm></filename32arm>
			<filename64arm></filename64arm>
			<rootpath></rootpath>
			<numberoptions></numberoptions>
			<dhcpleaseinlocaltime></dhcpleaseinlocaltime>
			<dnsserver>192.168.3.6</dnsserver>
		</opt4>
		<lan>
			<range>
				<from>10.10.1.100</from>
				<to>10.10.1.150</to>
			</range>
			<enable></enable>
			<failover_peerip></failover_peerip>
			<defaultleasetime></defaultleasetime>
			<maxleasetime></maxleasetime>
			<netmask></netmask>
			<dnsserver>192.168.3.6</dnsserver>
			<gateway></gateway>
			<domain>home.lab</domain>
			<domainsearchlist>home.lab</domainsearchlist>
			<ddnsdomain></ddnsdomain>
			<ddnsdomainprimary></ddnsdomainprimary>
			<ddnsdomainsecondary></ddnsdomainsecondary>
			<ddnsdomainkeyname></ddnsdomainkeyname>
			<ddnsdomainkeyalgorithm>hmac-md5</ddnsdomainkeyalgorithm>
			<ddnsdomainkey></ddnsdomainkey>
			<mac_allow></mac_allow>
			<mac_deny></mac_deny>
			<ddnsclientupdates>allow</ddnsclientupdates>
			<tftp></tftp>
			<ldap></ldap>
			<nextserver></nextserver>
			<filename></filename>
			<filename32></filename32>
			<filename64></filename64>
			<filename32arm></filename32arm>
			<filename64arm></filename64arm>
			<rootpath></rootpath>
			<numberoptions></numberoptions>
			<dhcpleaseinlocaltime></dhcpleaseinlocaltime>
		</lan>
	</dhcpd>
	<dhcpdv6></dhcpdv6>
	<snmpd>
		<syslocation></syslocation>
		<syscontact></syscontact>
		<rocommunity>public</rocommunity>
	</snmpd>
	<diag>
		<ipv6nat>
			<ipaddr></ipaddr>
		</ipv6nat>
	</diag>
	<syslog>
		<filterdescriptions>1</filterdescriptions>
	</syslog>
	<filter>
		<rule>
			<id></id>
			<tracker>1625536415</tracker>
			<type>pass</type>
			<interface>wan</interface>
			<ipprotocol>inet</ipprotocol>
			<tag></tag>
			<tagged></tagged>
			<max></max>
			<max-src-nodes></max-src-nodes>
			<max-src-conn></max-src-conn>
			<max-src-states></max-src-states>
			<statetimeout></statetimeout>
			<statetype><![CDATA[keep state]]></statetype>
			<os></os>
			<protocol>tcp</protocol>
			<source>
				<any></any>
			</source>
			<destination>
				<network>(self)</network>
				<port>22</port>
			</destination>
			<descr><![CDATA[Allow 22 to pfSense]]></descr>
			<created>
				<time>1625536415</time>
				<username><![CDATA[admin@192.168.2.94 (Local Database)]]></username>
			</created>
			<updated>
				<time>1625536955</time>
				<username><![CDATA[admin@192.168.2.94 (Local Database)]]></username>
			</updated>
		</rule>
		<rule>
			<id></id>
			<tracker>1625535224</tracker>
			<type>pass</type>
			<interface>wan</interface>
			<ipprotocol>inet</ipprotocol>
			<tag></tag>
			<tagged></tagged>
			<max></max>
			<max-src-nodes></max-src-nodes>
			<max-src-conn></max-src-conn>
			<max-src-states></max-src-states>
			<statetimeout></statetimeout>
			<statetype><![CDATA[keep state]]></statetype>
			<os></os>
			<protocol>icmp</protocol>
			<icmptype>any</icmptype>
			<source>
				<any></any>
			</source>
			<destination>
				<network>(self)</network>
			</destination>
			<descr><![CDATA[Allow Ping to pfSense]]></descr>
			<updated>
				<time>1625535224</time>
				<username><![CDATA[admin@192.168.2.94 (Local Database)]]></username>
			</updated>
			<created>
				<time>1625535224</time>
				<username><![CDATA[admin@192.168.2.94 (Local Database)]]></username>
			</created>
		</rule>
		<rule>
			<id></id>
			<tracker>1625535067</tracker>
			<type>pass</type>
			<interface>wan</interface>
			<ipprotocol>inet</ipprotocol>
			<tag></tag>
			<tagged></tagged>
			<max></max>
			<max-src-nodes></max-src-nodes>
			<max-src-conn></max-src-conn>
			<max-src-states></max-src-states>
			<statetimeout></statetimeout>
			<statetype><![CDATA[keep state]]></statetype>
			<os></os>
			<protocol>tcp</protocol>
			<source>
				<any></any>
			</source>
			<destination>
				<network>(self)</network>
				<port>443</port>
			</destination>
			<descr><![CDATA[Allow 443 to pfSense]]></descr>
			<updated>
				<time>1625535067</time>
				<username><![CDATA[admin@192.168.2.94 (Local Database)]]></username>
			</updated>
			<created>
				<time>1625535067</time>
				<username><![CDATA[admin@192.168.2.94 (Local Database)]]></username>
			</created>
		</rule>
		<rule>
			<id></id>
			<tracker>1625673442</tracker>
			<type>pass</type>
			<interface>wan</interface>
			<ipprotocol>inet</ipprotocol>
			<tag></tag>
			<tagged></tagged>
			<max></max>
			<max-src-nodes></max-src-nodes>
			<max-src-conn></max-src-conn>
			<max-src-states></max-src-states>
			<statetimeout></statetimeout>
			<statetype><![CDATA[keep state]]></statetype>
			<os></os>
			<source>
				<any></any>
			</source>
			<destination>
				<any></any>
			</destination>
			<descr></descr>
			<updated>
				<time>1625673442</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</updated>
			<created>
				<time>1625673442</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</created>
		</rule>
		<rule>
			<type>pass</type>
			<ipprotocol>inet</ipprotocol>
			<descr><![CDATA[Default allow LAN to any rule]]></descr>
			<interface>lan</interface>
			<tracker>0100000101</tracker>
			<source>
				<network>lan</network>
			</source>
			<destination>
				<any></any>
			</destination>
		</rule>
		<rule>
			<type>pass</type>
			<ipprotocol>inet6</ipprotocol>
			<descr><![CDATA[Default allow LAN IPv6 to any rule]]></descr>
			<interface>lan</interface>
			<tracker>0100000102</tracker>
			<source>
				<network>lan</network>
			</source>
			<destination>
				<any></any>
			</destination>
		</rule>
		<rule>
			<id></id>
			<tracker>1626104310</tracker>
			<type>pass</type>
			<interface>opt1</interface>
			<ipprotocol>inet</ipprotocol>
			<tag></tag>
			<tagged></tagged>
			<max></max>
			<max-src-nodes></max-src-nodes>
			<max-src-conn></max-src-conn>
			<max-src-states></max-src-states>
			<statetimeout></statetimeout>
			<statetype><![CDATA[keep state]]></statetype>
			<os></os>
			<source>
				<network>opt1</network>
			</source>
			<destination>
				<any></any>
			</destination>
			<descr><![CDATA[Default allow to any rule]]></descr>
			<updated>
				<time>1626104310</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</updated>
			<created>
				<time>1626104310</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</created>
		</rule>
		<rule>
			<id></id>
			<tracker>1626104295</tracker>
			<type>pass</type>
			<interface>opt2</interface>
			<ipprotocol>inet</ipprotocol>
			<tag></tag>
			<tagged></tagged>
			<max></max>
			<max-src-nodes></max-src-nodes>
			<max-src-conn></max-src-conn>
			<max-src-states></max-src-states>
			<statetimeout></statetimeout>
			<statetype><![CDATA[keep state]]></statetype>
			<os></os>
			<source>
				<network>opt2</network>
			</source>
			<destination>
				<any></any>
			</destination>
			<descr><![CDATA[Default allow to any rule]]></descr>
			<updated>
				<time>1626104295</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</updated>
			<created>
				<time>1626104295</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</created>
		</rule>
		<rule>
			<id></id>
			<tracker>1625687815</tracker>
			<type>pass</type>
			<interface>opt3</interface>
			<ipprotocol>inet</ipprotocol>
			<tag></tag>
			<tagged></tagged>
			<max></max>
			<max-src-nodes></max-src-nodes>
			<max-src-conn></max-src-conn>
			<max-src-states></max-src-states>
			<statetimeout></statetimeout>
			<statetype><![CDATA[keep state]]></statetype>
			<os></os>
			<source>
				<network>opt3</network>
			</source>
			<destination>
				<any></any>
			</destination>
			<descr><![CDATA[Default allow LAN to any rule]]></descr>
			<updated>
				<time>1625687815</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</updated>
			<created>
				<time>1625687815</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</created>
		</rule>
		<rule>
			<id></id>
			<tracker>1626104265</tracker>
			<type>pass</type>
			<interface>opt4</interface>
			<ipprotocol>inet</ipprotocol>
			<tag></tag>
			<tagged></tagged>
			<max></max>
			<max-src-nodes></max-src-nodes>
			<max-src-conn></max-src-conn>
			<max-src-states></max-src-states>
			<statetimeout></statetimeout>
			<statetype><![CDATA[keep state]]></statetype>
			<os></os>
			<source>
				<network>opt4</network>
			</source>
			<destination>
				<any></any>
			</destination>
			<descr><![CDATA[Default allow to any rule]]></descr>
			<updated>
				<time>1626104265</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</updated>
			<created>
				<time>1626104265</time>
				<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
			</created>
		</rule>
		<separator>
			<wan></wan>
			<lan></lan>
			<opt3></opt3>
			<opt4></opt4>
			<opt2></opt2>
			<opt1></opt1>
		</separator>
	</filter>
	<ipsec></ipsec>
	<aliases></aliases>
	<proxyarp></proxyarp>
	<cron>
		<item>
			<minute>*/1</minute>
			<hour>*</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/usr/sbin/newsyslog</command>
		</item>
		<item>
			<minute>1</minute>
			<hour>3</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/etc/rc.periodic daily</command>
		</item>
		<item>
			<minute>15</minute>
			<hour>4</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>6</wday>
			<who>root</who>
			<command>/etc/rc.periodic weekly</command>
		</item>
		<item>
			<minute>30</minute>
			<hour>5</hour>
			<mday>1</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/etc/rc.periodic monthly</command>
		</item>
		<item>
			<minute>1,31</minute>
			<hour>0-5</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/usr/bin/nice -n20 adjkerntz -a</command>
		</item>
		<item>
			<minute>1</minute>
			<hour>3</hour>
			<mday>1</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/usr/bin/nice -n20 /etc/rc.update_bogons.sh</command>
		</item>
		<item>
			<minute>1</minute>
			<hour>1</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/usr/bin/nice -n20 /etc/rc.dyndns.update</command>
		</item>
		<item>
			<minute>*/60</minute>
			<hour>*</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot</command>
		</item>
		<item>
			<minute>30</minute>
			<hour>12</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/usr/bin/nice -n20 /etc/rc.update_urltables</command>
		</item>
		<item>
			<minute>1</minute>
			<hour>0</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/usr/bin/nice -n20 /etc/rc.update_pkg_metadata</command>
		</item>
		<item>
			<minute>0</minute>
			<hour>23</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command>/usr/bin/nice -n20 /usr/local/bin/php /usr/local/sbin/execacb.php</command>
		</item>
	</cron>
	<wol></wol>
	<rrd>
		<enable></enable>
	</rrd>
	<widgets>
		<sequence>system_information:col1:open:0,interfaces:col2:open:0</sequence>
		<period>10</period>
	</widgets>
	<openvpn></openvpn>
	<dnshaper></dnshaper>
	<unbound>
		<enable></enable>
		<dnssec></dnssec>
		<active_interface></active_interface>
		<outgoing_interface></outgoing_interface>
		<custom_options></custom_options>
		<hideidentity></hideidentity>
		<hideversion></hideversion>
		<dnssecstripped></dnssecstripped>
	</unbound>
	<revision>
		<time>1626966645</time>
		<description><![CDATA[admin@192.168.3.155 (Local Database): main]]></description>
		<username><![CDATA[admin@192.168.3.155 (Local Database)]]></username>
	</revision>
	<ppps></ppps>
	<gateways>
		<gateway_item>
			<interface>wan</interface>
			<gateway>192.168.3.1</gateway>
			<name>WANGW_2</name>
			<weight>1</weight>
			<ipprotocol>inet</ipprotocol>
			<descr><![CDATA[Interface wan Gateway]]></descr>
		</gateway_item>
		<defaultgw4>WANGW_2</defaultgw4>
		<defaultgw6></defaultgw6>
	</gateways>
	<cert>
		<refid></refid>
		<descr><![CDATA[webConfigurator default ()]]></descr>
		<type>server</type>
		<crt></crt>
		<prv></prv>
	</cert>
	<installedpackages>
		<package>
			<name>Open-VM-Tools</name>
			<descr><![CDATA[VMware Tools is a suite of utilities that enhances the performance of the virtual machine's guest operating system and improves management of the virtual machine.]]></descr>
			<website>http://open-vm-tools.sourceforge.net/</website>
			<version>10.1.0_5,1</version>
			<pkginfolink>https://docs.netgate.com/pfsense/en/latest/packages/open-vm-tools.html</pkginfolink>
			<configurationfile>open-vm-tools.xml</configurationfile>
			<logging>
				<logfilename>vmware-vmsvc-root.log</logfilename>
			</logging>
			<include_file>/usr/local/pkg/open-vm-tools.inc</include_file>
		</package>
		<service>
			<name>vmware-guestd</name>
			<rcfile>vmware-guestd.sh</rcfile>
			<custom_php_service_status_command>mwexec(&quot;/usr/local/etc/rc.d/vmware-guestd status&quot;) == 0;</custom_php_service_status_command>
			<description><![CDATA[VMware Guest Daemon]]></description>
		</service>
		<service>
			<name>vmware-kmod</name>
			<rcfile>vmware-kmod.sh</rcfile>
			<custom_php_service_status_command>mwexec(&quot;/usr/local/etc/rc.d/vmware-kmod status&quot;) == 0;</custom_php_service_status_command>
			<description><![CDATA[VMware Kernel Modules]]></description>
		</service>
	</installedpackages>
	<vlans></vlans>
	<shaper></shaper>
</pfsense>

bookmark_borderConfiguring the baremetal ESXi7

So now that we have our SuperMicro server setup, and we’ve installed ESXi7, the next step is to do some basic configuration on our BareMetal ESXi7. We will setup the network, view the storage, and prepare the esxi for the nested esxi environment.

Log in to the UI of the baremetal Esxi device, and you’ll see a screen like this

Network Configuration

Navigate to the Networking tab, and select physical Nics. As you can see I have 4 physical NICs on my SuperMicro, and I have my ethernet plugged into vmnic1.
Navigate to the Virtual Switches tab, we will need to create 1 more virtual switch.
Click “Add standard virtual switch” and configure using the details above. Make sure to open the Security tab and Accept all the options: Promiscuous Mode, MAC address changes, and Forged transmits. 
Click add and you will see a screen like this.
Navigate to the port groups tab, it’s time to create our networks.
Create the first network with the options listed above. We’re going to repeat these steps to create 4 more networks.
Here’s a list of the networks that you should have. Note that the vSwitch is vSwitch1 for vlan10-50. I’m not doing vlan tagging in my setup, feel free to do it on your end if you prefer.

Storage Configuration

You should see your SSD here. I have a 2GB SSD, but you should see your physical disk listed here.

This is basically the whole setup for the baremetal. Next step will be deploying the esxi vms.

bookmark_borderVirtual Router in your home lab – pfsense

I spent quite a bit of time researching solutions for virtual routers. I’m definitely not a r/s expert, so I wanted something that was easy to setup, and still provided lots of tools for troubleshooting. Initially I setup the lab using vyos router, and it met my needs. But I since switched to pfsense, and it’s absolutely amazing. Plus it comes with a pretty sweet firewall. So my guide here, will be a tutorial on how to deploy pfsense and configure it for home lab purposes. If you already have a router in mind, or a physical router, feel free to skip these steps.


Download pfsense community edition

https://www.pfsense.org/download/ – Download the ISO. Version used in my lab was 2.5.1. (Around 600MB)


Deploying the pfsense vm

Load the UI of your baremetal esxi7 and under Virtual Machines select “Create / Register VM”. Then select Next.
Name the vm “pfsense-router” and select the options as listed above.
Select the SSD
1 CPU should be fine, but feel free to make it 2 if you want. 2GB memory and 8GB HD is fine (Thin provisioned)
Select the CD/DVD Drive 1 and the dropdown to Datastore ISO file
Navigate the datastore to find the pfsense ISO file. If you haven’t uploaded it yet, then upload it to a directory (I created a directory called ISOs)
Setup should look like this. Click Next.
Summary, click Finish.
Power on the VM

Installing pfsense

Power on the VM and select the black GUI window, it will open the console window for this vm
Accept the agreement.
Install
Configure with default keymap
Auto (UFS) BIOS – Guided Disk Setup using BIOS boot method
Now let the installer run, should be fast, 1-2min tops.
Select No
Reboot, then exit the console. We need to add the networks to the vm.

Adding networks to pfsense vm

Click the vm and select Actions in the menu above, then “Edit settings”
Add 5 more network adapters and configure them as follows. Click Save and power on the VM

Initial config and disable firewall in pfsense cli

Back in the console, and you can see it is still rebooting…
After it has booted you will see this first time setup wizard. Select no for vlans.
Enter em0 for the WAN (If you don’t see this screen, don’t worry)
Leave this blank (If you don’t see this screen, don’t worry)
Ok, setup is complete and now we need to disable the firewall so we can continue setup in the GUI
Type 8 and press enter
Run the command: pfctl -d
By default the firewall is very restrictive so you won’t be able to access the GUI. We will reenable it later, after we’ve opened up the correct firewall rules šŸ™‚

GUI configuration of pfsense – Initial config of firewall

At this point you can access the GUI of the pfsense firewall. login with admin:pfsense
It’s a good idea to go ahead and reset the admin account password, click the link in the red banner at the top of the page.
Navigate over to the firewall tab, and select Rules. We need to configure some basic firewall rules to allow us access to the firewall. (Which is blocked by default)
Select the Green Add button and let’s add the first rule. As you can see in this image, I modified the destination port to SSH (22) and the Destination “This firewall (self)”
I also added a description. Click save.
DO NOT APPLY THE CHANGES YET! Your screen should look like this..
Go ahead and configure 2 more rules, in the same manner. for ICMP and port 443 (HTTPS)
Lastly we need to remove these 2 rules, by editing the WAN interface. Click on the gear icon on the right side.
Scroll all the way down on the page and you will see an option to uncheck these boxes. Uncheck “Block private networks and loopback addresses” and “Block bogon networks”
Go ahead and apply these changes.
Navigate back to the Firewall rules section and you’ll see those 2 rules are gone. All that is left is the 3 rules that you created. Go ahead and apply changes. It will take a minute, but if everything worked, you should still have access to the GUI.

Setting up the Interfaces

Next up, let’s rename and configure the interfaces for our WAN, then our 4 Internal vlans. Navigate to Interfaces -> Assignments
Click Add a few times until you add all the available interfaces. Should look like this (Don’t worry if your interface names are slightly different)
Now we’re going to go 1 by 1 through each of the interfaces and do the following:
1.Enable
2. Set Description
3. Static IPv4
4. IPv6 None
5. IPv4 Address and Mask (Select a static IP in your home network)
6. IPv4 Upstream gateway (Choose your gateway, ONLY SET THIS ON THE WAN)
And lastly, make sure these are unchecked. Save.
We’re going to repeat this process for each of the other interfaces. Here’s an example of the first vlan interface.
Same thing, uncheck these boxes. Save.
In the end, your interfaces should be setup like this. Check the network diagram if you are unsure of what IPs to use.

Firewall rules for each of the interfaces

I’m not going to go through each of the edit screens on how to add firewall rules, since I have shown examples in the above screenshots. Instead i’m just going to post screenshots of the firewall rules that I have added for each interface. In some instances I have just added a blanket allow all from any to any because its my home lab and pfsense is not my actual internet firewall šŸ™‚ If you want to keep it more secure, go for it.

Nothing to configure on the Floating interface.
Internet has the 3 rules we added initially. And my blanket allow everything rule šŸ™‚ (which essentially makes the first 3 rules useless)
The Anti-Lockout rule is set by default on the LAN interface, it’s not hurting anything so you can leave it. Add 2 more rules.
I started getting lazy and only adding ipv4. Honestly, I don’t use ipv6..
Same..
Same..
Last one.. And once they’re all set, you can Apply the changes.

Configure DHCP – Optional, but recommended

I would recommend setting up 50 or so IPs to be used for DHCP. It makes spinning up new vms in vcenter a little easier since it will auto select and IP, get the correct dns server, domain name, etc.
Configure these on all the interfaces EXCEPT The WAN (Internet) interface.

Click Enable, and setup using similar settings as above. I chose the range .100 – .150 in all my vlans. You can choose any range in the /24.
Set the DNS server
Set the domain name, and search list
Nothing here. Just hit save, then configure the rest.

Optional: Parse the xml and skip the step by step guide

This is my backup config. It probably won’t load for you if you try and do a system restore since I’ve removed cert info and passwords, etc. But if you prefer to parse this and setup pfsense, i’m making it available.

XML from pfsense router

bookmark_borderConfiguring Nested ESXi7 on SuperMicro

We now have our SuperMicro server setup, and the esxi7 network settings are configured. It’s time to deploy our 3 nested ESXi7 vms. In later steps, we will configure them to use vSAN and be managed with vcenter.

Deploy 3 ESXi7 VMs

Navigate to the virtual machine tab. It’s time to setup our nested ESXi7 vms.
Click “Create / Register VM” and you’ll see the popup above. You can deploy esxi in 2 ways. We’re doing the ova method, so select the second option.

Methods to install nested ESXi in your lab:

  1. The method we used for the baremetal install. Basically mount the ISO as a (Datastore ISO) to a newly created VM, and go through the install process. This takes time to do for 3 vms, and i’m lazy šŸ™‚
  2. Use a preconfigured ova, and just modify it to our needs. We will be using this method. Head on over to William Lam’s website to download the ova. Filename: ESXi 7.0 Update 2a Virtual Appliance.
Specify the name and download the ova from the above link (William Lam’s website) and select that ova image here.
Select your storage device
The ova will take a few minutes to extract, be patient..
Once it’s ready, click I agree to accept the EUL and click Next.
Select VM Network, and Disk Provisioning Thin. Deselect Power on automatically. Next.
Leave all these default, and click Next.
Here’s the summary page, select Finish.

Configure the 3 nested ESXi7 VMs

Your screen should look like this. the vm is deployed, but it is powered down. If it’s not powered down, please do so now.
Click on the VM and goto the Actions menu, select Edit Settings.
We’re going to overprovision our environment pretty heavily here. My vsphere license allowed me to have 32 cpus, so I provisioned 3 ESXi hosts with 10, 10, and 12 CPUs. Depending on what your license is, you might change this number up or down slightly. Make sure “Expose hardware assisted virtualization to the guest OS” is checked!
yada yada, Warning saying be careful about over provisioning lol. Ignore it šŸ™‚ CPU is 10, memory is 128GB, everything else is default so far. Don’t worry about memory reservations under Memory tab, I left all that default.
Create 3 new disks, these will be used for vSAN. (You will create 3 disks on each of the nested esxi vms)
Pay attention to this step!!!!!!!! Set the hard disks to the GB as I listed above (Most importantly that the new disks are 200gb or more AND THAT THEY ARE ALL SET TO “Thin provisioned” If you select Thick, it will auto fill the space in your SSD, thin, it uses it as needed. Every disk should be thin provisioned in this setup.
Navigate back to the top and click “Add network adapter.” You will be adding 8 more network adapters, for a grand total of 10.
Make sure they’re all connected and configure them as shown above. Leave all the other settings under the networks default.
CD and video card, leave default.
Click Save and you should see a screen similar to above.

Repeat the above steps to create 3 total ESXi nested vms: vesxi7-1, vesxi7-2, vesxi7-3

  • 12 vcpu (12, 10, 10)
  • 128 GB memory
  • 12 HD1 thin
  • 4 HD2 thin
  • 20 HD3 thin
  • 200 HD4 thin
  • 200 HD5 thin
  • 200 HD6 thin
  • Network Adapter 1 vLAN10-Management
  • Network Adapter 2 vLAN10-Management
  • Network Adapter 3 vLAN20-vMotion
  • Network Adapter 4 vLAN20-vMotion
  • Network Adapter 5 vLAN30-vSAN
  • Network Adapter 6 vLAN30-vSAN
  • Network Adapter 7 vLAN40-vmNetwork
  • Network Adapter 8 vLAN40-vmNetwork
  • Network Adapter 9 vLAN50
  • Network Adapter 10 vLAN50
After you’ve finished, you’ll see 3 vms as shown above. Go ahead and power them on.

Powering on the Esxi hosts and setting management network

Load the console and press F2 to enter the setup. Default credentials are: root: VMware1!
After authenticating, we need to change the root password!
Now time to configure the management network, since it is likely using DHCP, and we want these ips to be static!
Click on IPv4 Configuration and select the radio button next to “Set static IPv4 address and network configuration”
You’ll need 3 IPs, 1 for each of the esxi hosts.
Disable ipv6
DNS server should already be here, since we configured DHCP on the pfsense router. However, if it is not. Specify them manually.
Lastly make sure that your suffix is set. You don’t have to use home.lab for your envrionment, but make sure you’re consistent across the board.