Clone a Nested ESXi VM for Nested vSphere Lab Environments

Posted on by Matt Adam

Update. Don’t clone an esxi VM to create a nested environment. There is a better way! Use a kickstart script and install it directly from the ISO.
When I originally started this blog post, I thought the best/only way to create a bunch of esxi vms for nesting, was to do it via cloning. I had a nice ansible script to clone the VM, deploy all the network settings, etc. But I ran into quite a few problems, as you’ll see below in the original blog post, and eventually decided it wasn’t worth it. The final problem that I couldn’t solve was it seems the cloned esxi VMs couldn’t be added to a vsan, since there was some uuid caching going on between the cloned VMs. I know there are some commands out there to run to resolve this issue, they didn’t work for me.
And even if they had worked, like I said there is a better, cleaner way to accomplish this.

So why not just remove this blog altogether? Simple. Google searchability. If you are wanting to automate the creation of a bunch of esxi hosts for testing purposes, and all you know to search for is cloning, then hopefully you’ll find this page and learn there is a better way. Install via the ISO automatically using a kickstarter script (KS).

Original Post Below..

I’m in the process of building out a nested VCF lab in my environment. A small VCF lab requires 4 management esxi hosts, and 3 additional hosts for each workload domain. So at a minimum that is 7 different nested hosts that will need to be configured in the environment.
During the process of using the VMware Cloud Builder (VCF deployment tool) I destroyed many of those nested ESXi hosts. Basically every time the cloud builder fails, you need to recreate the esxi hosts to reset their configuration. And since I’m just learning VCF, I made a lot of mistakes on the json configuration to build out the environment. With every mistake, means a manual recreation of 4 esxi hosts for the management domain in VCF. After about 2 or so manual rebuilds, I decided to automate that task.

So this guide is really for anyone wanting to quickly create multiple nested ESXi VMs for their use. I’ll walk through the steps on how to create the “template” ESXI vm to use in cloning the rest. As well as some of the takeaways I’ve learned in the process. Quick shoutout to William Lam, his blog post on this topic was a big help in putting the final pieces together.

Creating the template ESXI Host

The process here is pretty straightforward. If you’ve ever created an ubuntu/centos VM in vSphere/vCenter before, you’ve likely done all the required steps for this. We’re going to create a VM and attach the ESXI ISO as a disk, and go through the GUI to install it on the disk. That’s all for this step.

Download the latest ISO

Navigate to your customer connect page at VMware and navigate to Products and Accounts -> All Products

Select VMware vSphere -> View Download Components

I’m downloading the Enterprise Plus version of the vSphere Hypervisor. Go to Downloads.

Download the VMware vSphere Hypervisor (ESXi ISO) image.

Create the Template ESXi VM

Open the GUI for your esxi host where we will be nesting these environments, and let’s create the template ESXI host.

Click Create / Register VM and select Create a new virtual machine.

Name the template “esxi-template”. I added a -new since I already have a working template created. Set the Compatibility, Guest OS family, and Guest OS version as shown above.

Select your datastore and click Next.

Customize your nested ESXi settings. I am creating 4 hosts at 8 vCPU each, and 64GB of memory for each of them. If you want anything different please make those changes here.

Checklist:

  • Under the CPU, make sure you check the box for “Expose hardware assisted virtualization to the guest OS”
  • Under the Hard disk 1, for Disk Provisioning select “Thin provisioned”
  • For the Network Adapter 1, the automation script will auto attach the required networks, so for this template all you will need is the first Network Adapter.
  • On the CD/DVD Drive 1, select the Datastore ISO file, and you will need to select a folder and upload the ISO that was previously downloaded.

See pics below for detailed views of the above.

When the settings are all correct, click Next to see the summary, and click Finish to deploy the VM.

Power on the esxi-template VM.

After the VM powers on, you will need to install vSphere Hypervisor onto the disk. That process is outlined in an earlier blog post: Install ESXi

After the installation, and the reboot, you should eventually see something like this:

Configure the Template ESXi VM

There are a few things we have to configure on the template ESXi VM in order to make it cloneable. These items are pointed out in William’s blog post above.

  • Enable DHCP or set a static IP. We will need to ssh into the esxi host, and modify a few settings
  • Enable ssh on the esxi host
  • Run 3 commands on the esxi host to fix the mac address and the system uuid from persisting to the clones.

Set an IP address for the ESXi VM

If you already have DHCP in your environment (I would definitely recommend) then you should see something like the below.

If you see a DHCP address, and you can reach it. Skip to the next step.
If you don’t see an Address, or it is not reachable. Please see my blog post on how to configure basic settings on an ESXi host: Configure the ESXi Host

Enable SSH on the ESXi VM

To enable ssh on the ESXi, you will need to login to the UI, and enable it.

Login with root and your password.

Select Actions -> Services -> Enable Secure Shell (SSH)

Test the ssh as shown above, make sure you can ssh into your esxi host.

Final Configuration on the ESXi template VM

There are 3 commands that need to be run. Again for more detailed explanations, see William’s blog post above. But basically when cloning an ESXi VM, the MAC address and the system UUID will persist. Fortunately there is an easy solution, just run the following 3 commands directly on the template VM.

# Run the following on the ESXi template VM

esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1
sed -i 's#/system/uuid.*##' /etc/vmware/esx.conf
/sbin/auto-backup.sh

After running those 3 commands, you can shutdown the template vm. It is ready to be cloned.

Automating the rest with Ansible

Removed the script since it is recommended not to use cloning.