AKO – L7 Ingress on vSphere with Kubernetes (WCP

Long title, but basically this is a guide to deploy L7 ingress on top of your WCP and NSX-T setup. If you’ve followed my previous guides, you should have NSX and WCP configured, and a supervisor cluster and guest cluster already configured. We will need to add a little bit more config to NSX-T and then spin up an Avi Controller and service engines. Finally we will deploy AKO and configure L7 ingress.

Table of Contents

NSX Networking

Create Avi T1 in NSX

By this point we’ve created a few T1 gateways so the process is the same for this one. I called it Avi-T1 and under Route Advertisements make sure to check them all.

[All Static Routes, All NAT IP’s, All DNS Forwarder Routes, All LB VIP Routes, All Connected Segments & Service Ports, All LB SNAT IP Routes, All IPSec Local Endpoints]

Create 2 Avi Segments under the Avi T1

Create 2 segments, one for the Vip/Data traffic and another for management.

Deploy and Configure Avi (NSX Advanced Load Balancer)

In the interest of not duplicating work, see this guide: https://mattadam.com/2021/10/12/tanzu-kubernetes-on-vcenter-7-deploy-avi-controller-and-service-engines/ Stop at the Configure Cloud section, and that is where this guide will start.

Create Credentials

We need to create the nsx manager and vcenter credentials, for use in the cloud.

Navigate to Administration -> User Credentials and click Create.

Create the first set of credentials for NSX-T.

Configure NSX-T Cloud

After the Avi controller is deployed, log in and navigate to Infrastructure -> Clouds. Then select Create NSX-T Cloud.

Set the cloud name, type is NSX-T Cloud. Then the object prefix as avi.

In the NSX-T section set the nsx manager address (ip or fqdn) and the pre-created credentials. Then click Connect

For the management network, set the Transport zone as nsx-overlay-transportzone (Overlay) and set the Avi-T1 router as well as the Avi-Mgmt segment.

For the data network, similarly you will use the Avi-T1 router and the overlay segment is Avi-Vip

For vCenter server, set the name, address, and credentials created in the earlier step. Also if you haven’t already created a new content library in vCenter do so now and refresh the page. The content library is used for Avi to store the SE images.

We will set the IPAM and DNS profile later, Set the DNS resolver and click Save.

Set static IP Ranges (Optional)

If you have a dhcp range configured in your networks, you likely can skip this step. Just verify that the networks have been discovered by Avi. Navigate to Infrastructure -> Cloud Resources -> Networks and select the “nsxt” cloud

select the Avi-Mgmt network and click Edit on the right side.

Set the subnet and an IP Range that can be used for the SEs.

Do the same thing for the Avi-Vip network.

Set static routes for Vip and management subnets

This is done in the avi cli.

#Exact commands used:

shell #login to the avi shell with credentials

configure vrfcontext Avi-T1 #Enter submode
static_routes #Enter submode
next_hop 172.16.40.1
route_id 2
prefix 0.0.0.0/0
save
save

configure vrfcontext management #Enter submode
static_routes #Enter submode
next_hop 172.16.30.1
route_id 3
prefix 0.0.0.0/0
save
save

ssh into the avi controller, type “shell” and then your credentials.

Type switchto cloud nsxt

Configure the vrfcontext Avi-T1 as shown above.

Configure the vrfcontext management as shown above.

Create a DNS and IPAM profile.

Navigate to Templates -> Profiles -> IPAM/DNS Profiles

Lastly navigate back to the nsxt cloud we created earlier, and add in the IPAM and DNS profile.

Install AKO on K8s Cluster

In this step we’ll deploy AKO and a test deployment with an ingress to ensure everything is working.

Exact Commands:

#login to cluster
kubectl vsphere login --vsphere-username administrator@vsphere.local --server=https://10.10.1.2 --insecure-skip-tls-verify --tanzu-kubernetes-cluster-namespace=dev --tanzu-kubernetes-cluster-name=tkg-cluster-01
kubectl config use-context tkg-cluster-01

#Helm Setup
kubectl create ns avi-system
helm repo add ako https://projects.registry.vmware.com/chartrepo/ako
helm show values ako/ako --version 1.7.1 > values.yaml
nano values.yaml

#Example values.yaml file edits

clusterName: my-cluster #Change the name here
layer7Only: true #Set this to true. NSX will handle L4. Avi will handle L7.
nsxtT1LR: '/infra/tier-1s/Avi-T1' #Set this to the Avi T1 from NSX.

vipNetworkList: [] - Comment out this line and replace it with the following:
  vipNetworkList:
   - networkName: Avi-Vip
     cidr: 172.16.40.0/24

serviceType: NodePort #I am using NodePort for my lab.
shardVSSize: SMALL #Small will create a small number of VS for sharding.

ControllerSettings:
  serviceEngineGroupName: Default-Group
  controllerVersion: '21.1.4'
  cloudName: nsxt
  controllerHost: 'avi-controller.home.lab'
  tenantName: admin   

avicredentials:
  username: "admin"
  password: "password123"

After saving the values.yaml file, the following commands are used to install AKO.

helm install  ako/ako  --generate-name --version 1.7.1 -f /ako/values.yaml namespace=avi-system

2 thoughts on “AKO – L7 Ingress on vSphere with Kubernetes (WCP – TKGs) with NSX-T”

Michel AL Daher says:

August 9, 2022 at 8:38 am

Hello Matt,

first thank you for the blog very helpful, i have a small question why do we need to add the static routes on the VRF, it should not be automated ?

1. Matt Adam says:
  
  August 9, 2022 at 8:52 am
  
  Without a default gateway, the nodes wouldn’t know how to reach any other network. So you must set the DG in the management vrf as well as the vip/data vrf.