This post is intended to outline the steps required to configure AVI with vSphere with Tanzu deployed with NSX Networking. It will focus on the AVI configuration pieces specifically. It is assumed NSX and vSphere have been set up with the requisite functionality. This is for initial Supervisor setup and does not include installing AVI as L7 ingress controller into Workload clusters.
Software Versions used in this demo
| Software | Version |
|---|---|
| vSphere | >=8.0u2 |
| NSX | >=4.1.1 |
| Avi | >=22.1.4 * Avi Software version 30.x is not supported at time of writing ** Requires Avi Enterprise License |
AKO pod in the Supervisor is deployed as part of Workload Management. Within the pod are 2 containers. AKO-MGR and AKO-INFRA. AKO Infra adds networks to the cloud as well as IPAM as NCP creates the infrastructure. Troubleshooting steps for the two pods will be shown at the end of the document.
High Level Diagram
Note: SE Data segments are DHCP only – This is why it is critical to set DHCP in the NSX Cloud Connector
Same SE can attach up to 9 Data Segments. This means 9 namespaces per SE is allowed.
Deploy AVI Controller
This guide will not cover initial controller deployment. Please review the relevant KB for Controller deployment
Set up NSX-T cloud
As a pre-requisite to NSX-T Cloud creation we need to either choose an existing or create a new segment/vlan backed network for SE to Controller connectivity. We also need to create a T1 and overlay segment for data. The data T1 will not be used for the vSphere with Tanzu environment but it is a mandatory field for the NSX-T cloud creation. To setup the cloud perform the following;
- From the controller UI Navigate to Infrastructure > Clouds and select Create
- Choose NSX CLoud
- The below page will show. Give the cloud a name and an object prefix. These are unique to the setup.
- It is critical that the enable DHCP box is ticked. DHCP is used for AVI Data segments as described below.
- NSX Manager Details
- Add the NSX Manager address and the credentials used.
- Once the AVI controller can connect to NSX Manager select the Transport Zone. This must be the same transport zone used on the VDS switch that you will configure in Workload Manager. It is not supported for multiple clouds to share a Transport Zone
- Select the T1 (if overlay mode) and segment for AVI MGMT. This network is used for the SE to connect to the AVI Controller
- Select the T1 we defined earlier for Data. This T1 will not be used during workload manager. It is a mandatory field for cloud creation
- Add the vcenter connected to NSX that we want to use for compute. At present we only support 1 vcenter per NSX cloud for Workload Management
- On the Ipam field select the three dots and click create . This will create an IPAM profile. Leave the IPAM profile in its default setting. We just need a blank IPAM profile attached to the cloud. AKO will use this IPAM
- Click Save – When the cloud goes green you can proceed to the next step.
Service Engine Group Selection
All of the various options for Service Engine Groups are outside of the scope of this doc but if you wish to modify SEG settings before Workload Management install. You can modify the Default Service Engine Group for the NSX-T cloud you just created. This SEG will be used as a template for the SEG that will be built for Workload Management.
Register AVI with NSX Manager
Once the cloud has been created we need to register the AVI controller with NSX Manager. This is accomplished with the below API call from NSX-Manager. The call will set the enforcement point on the NSX-Manager so that it will use AVI as the LB vs NSX LB. It also creates the users that the two AKO will use.
We also need to change the AVI Portal certificate on the AVI controller as we cannot use the default certificate. This certificate can be self signed, signed by a private CA* or a Public CA.
It is imperative that the certificates contain the ip addresses of all Controllers and Controller VIP (if present) in the IP SAN field.
*If you are using a private CA you must upload the CA to NSX Managers by following this link.
To create a self signed certificate and change the Portal cert follow the following steps. If you already have a certificate upload it and its chain to Templates>Security>TLS Certificates and follow the steps below to change the Portal cert.
Create Self Signed Certificate
- Navigate to Templates>Security>TLS Certificates
- Click Create and choose Controller Certificate
- Enter the name of the cert and the Common Name of the controller
- Add the Ip addresses of the controller and VIP in the SAN field
- Click Save
Change AVI portal cert
Once you have created the certificate you can now add it as the Portal cert.
- Navigate to Administration>System Settings
- Click edit and under the Access setting add the new certificate as below to the SSL/TLS certificate section.
- Click Save.
One you have uploaded the cert the API call is as below
Note: If using NSX 4.1.2 or above the dns_server field or ntp_server fields are not required. If you include them they will overwrite the settings on AVI Controller
When you have run the API call and if it is successful you will see a similar response.
To confirm NSX has successfully registered AVI perform a GET to https://<nsx-mgr-ip>/policy/api/v1/infra/sites/default/enforcement-points/alb-endpoint
The status field should show Deactivate Provider . If so AVI has been successfully registered as the enforcement point.
Once AVI controller has been registered with NSX – You can now proceed with the Workload Manager Wizard
Workload Management Wizard
Choose NSX-T networking from the Wizard and select whether deployment is 3 Zone or Single Cluster. The decision tree for this is outside the scope of this document.
You will then choose the networking options for the management network. This network is for Supervisor management and AVI is not used for load balancing in any way for this network. This is the MGMT Network as depicted in the diagram above. An example of a completed configuration is shown below. The network can be the mgmt network the Infrastructure is deployed on (NSX Manager etc or another Management Network. We must assign 5 concurrent IPs from this network. 3 will be used for each Supervisor node and one will be used as a VIP for said nodes and the remaining is used for administrative purposes.
Once the management networking is complete – select Next and it will take you to the Workload Network page. A completed example is shown below.
These fields configure the networking that is used for the Supervisor namespace. A breakdown of the fields is shown below
| Field | Info |
|---|---|
| vSphere Distributed Switch | This is the VDS switch that the NSX-T Overlay Transport Zone is attached to.AVI controller will use this Switch to define which cloud to use on the AVI controller. At this time it is not supported to have multiple clouds on the AVI controller share Transport Zone as if so Workload Manager will select the first cloud with the TZ |
| Edge Cluster | The Edge Cluster the T0 is tied too |
| DNS Servers | DNS servers that the nodes will use for DNS |
| Services CIDR | This is the CIDR used for internal Kubernetes cluster IP services. Kube-DNS KUBE-API for example |
| Tier 0 gateway | The T0 that is tied to the edge cluster and VDS switch that we are leveraging |
| Nat Mode | This is a toggle button and is used if the nodes/pods are not routable external to NSX |
| Namespace Network | This Network is the workload network for this namespace in its entirety. It will be broken up into Subnets defined in the Namespace Network Field. This is the network that the Supervisor Nodes and any workload nodes within the namespace will use |
| Namespace Subnet Prefix | Subnet prefix used to break the Namespace Network up to Cluster Workload Networks |
| Ingress | This network will be added to the IPAM on the AVI controller. It is used for external API and L4/L7 Services provided by the SupervisorThis network will be configured by AKO-INFRA pod as the IPAM network for the NSX-Cloud – it will be put in the global VRF and IPs are pulled as /32s and placed on workloadmanager T1s as required. A further breakdown of this is below. This is not AKO providing L7 Ingress |
| Egress | If Nat mode is toggled this is the range that will be used as SNAT for external traffic from within the clusters. |
When you have completed the fields select Next and complete out the remaining Workload Management config. The remaining pieces are not AVI specifc so will not be covered here.
How ingress networking works
When supervisor or workload clusters are created NCP builds a T1 that connects to the T0 defined above for every namespace.
A Segment is then created for Supervisor/Workload node connectivity. This subnet comes out of the Namespace Network and is then sized accordingly to the Namespace Subnet Prefix. For example above the Namespace network is 10.244.0.0/21 and the Namespace Subnet Prefix is /28 so the first Workload segment for the supervisor would be 10.244.0.0/28.
NCP also creates a segment on the same T1 for AVI to connect to for Data. This segment is non routable outside and is taken from CGNAT range. DHCP is enabled on the segment.
When NCP creates the segment it tells the AKO-INFRA pod to add the namespace T1 and the newly created Data Segment to the NSX-T cloud. The AVI SE then sends a DHCP request and obtains an ip/ gateway and default route from NSX-T so it can plumb its NIC in the segment. This is why it is imperative that DHCP is enabled on the cloud as discussed previously.
The NSX-T Cloud will show the T1 and Data Segment as show
Once AKO-Infra has added the segment to the cloud it then uses one of the IPs from the ingress field above, in this case 192.168.68.240/28, to use as the actual VIP. The AVI Controller via the cloud connector and NSX Manager then adds the static routing to the T1 and advertises the network up to the T0 and beyond.
The NSX-T Topology should look similar to below upon creation.
When Workload Manager completes you should see the Virtual Service for Kube API for the supervisor
Troubleshooting
If the install process is hanging and you need to troubleshoot the AKO pods – perform the following steps
SSH as root to vcenter running the sup cluster
Run the following script as root
/usr/lib/vmware-wcp/decryptK8Pwd.py
This will give you the ip for the management side of the Supervisor and the password as below
SSH into the supervisor with the above credentials and you will now be able to run kubectl commands against the supervisor.
The AKO pod is deployed into the vmware-system-ako namespace and to log AKO infra pod for example run the following command
Kubectl logs vmware-system-ako-ako-controller-xxxxxx -n vmware-system-ako -c infra
This should show if there any errors on the AKO side