Kubernetes – vBrainstorm.com

This post is the first in a series of creating Tanzu Kubernetes on vSphere 8. I’m doing this my home lab, which I have decided to create with an Orange theme, because its not an Apple lab. (Yes, that’s cringy dad-joke territory.)

So what is done up to this point in the lab?

Purchased a Ubiquiti Dream Machine Pro as my core switch & router
Setup several VLANs for vSphere use
- Management VLAN
- vMotion VLAN
- FT VLAN
- Storage VLAN for iSCSI
- VSAN VLAN (for later)
- Servers VLAN for VM workloads
Purchased 3 x HP Z640 workstations with Xeon processors, RAM and an SSD boot disk
Setup iSCSI from a Synology NAS
Added dual 10Gb NICs to each host and added a Ubiquiti 8 port 10Gb Switch
Used 1 0 G t e k b r a n d TwinAx/DAC 10Gb cables to connect things up
Installed ESXi 8.0.0 on each host
Deployed vCenter 8.0.0
Configured a Standard vSwitch and Distributed vSwitch
Setup iSCSI software adapters, configured, connected and provisioned 2 x 1TB VMFS datastores

Deploying TKG

Tanzu Kubernetes Grid (TKG) with vSphere 8 has gotten significantly simpler, in my eyes. The wizards in vCenter 8 are great and walking a novice through the deployment of TKG in a simplistic way, however, there are somethings that flow better with the workflow if you’ve completed prep before hand.

Checklist of items to complete before the Workload Management wizard

vSphere Enterprise Plus licensing is required because DRS (and vSphere Distributed Switch) is required. HA also needs to be enabled.
You will need to make some basic network decisions around load balancing.
- You will either need to use native vSphere Distributed Switches (VDS) or NSX
- For vSphere Distributed Switches, you need a load balancer. Either deploying NSX Advanced Load Balancer or HAProxy load balancer are you two options.
- For NSX, you’ll use their internal load balancing features of NSX.
- NSX-T is supported and you can use a vSphere Distributed Switch (a converged distributed switch) but not a NSX Distributed Switch/N-DVS). You must migrate if you use N-VDS.
You need to setup and define vSphere Zones or your Tanzu is limited to a single vSphere cluster. Personally recommend setting up a zone (see below).
A storage policy is required.
Management Network – control plane VMs to the Supervisor cluster
Workload Network – traffic to the Kubernetes API, workloads, and services

vSphere Zones

Most customers are probably not leveraging vSphere Zones. This feature in vSphere is for multiple zone deployments when a customer scales to multiple colocation datacenters or clouds. I would recommend that even as you start your Tanzu journey, even if you only have a single zone – go ahead and create a single zone. This is one step in future-proofing your Tanzu installation. The whole idea of Kubernetes is enabling your cloud-native applications to be deployed across multiple clouds for resiliency. By creating your first zone, you create your first step in this journey. For more detail, see this article.

If you choose not to use vSphere Zones, you’ll need to choose a cluster to assign your Tanzu deployment onto.

New Networks for Tanzu and HAProxy (or NSX)

I created 3 new networks for Tanzu and HAProxy. The first is a Kubernetes Control Plane network. The second is a Load Balancer network. The third is a Kubernetes workload network.

The Kubernetes Control plane will be used for the HAProxy management network. This is where Tanzu will talk to HAProxy to configure and programmatically setup external access to your Kubernetes workloads. “Programatically” is the key here. Kubernetes will reach out to HAProxy and make all the magic happen automatically. While I’m talking about HAProxy – the same is true with NSX Advanced Load Balancer (aka AVI Networks) or NSX-T’s built-in load balancer.

The Load Balancer network is where your external endpoints for your workloads are going to be exposed. It is a range of IP addresses. For your production workloads, this is the network that should scale with the number of workloads you plan to have. Keep in mind, this is for the single zone (back to the previous concept) – so enough external IP addresses

HA Proxy Download and Config

Review the requirements of HAProxy Load Balancer from VMware. You can obtain the current OVA from VMware’s GitHub for HAProxy – https://github.com/haproxytech/vmware-haproxy.

As you deploy, take note of the following things that you will need during the configuration. Network settings and some backend network configuration for the HAProxy configuration is definitely required.

HAProxy Load Balancer Controller Endpoint – this is the API endpoint that Tanzu will call to configure HAProxy. This is the primary integration point of HAProxy with your Kubernetes and vSphere clusters.
Username and password configured during the HAProxy deployment
Virtual IP Address Ranges
TLS Certificate from the HAProxy appliance

I loaded the HAProxy OVA into my Content Library (aka The Barn) and then cloned a VM out from this location. It is a standard OVA deployment and you’ll be prompted for configuration to deploy. This guide does a great job of walking you through the HAProxy deployment.

Wizard, wizard

Welcome. The pre-requisites are listed and linked to documentation.

Decisions, decisions – What vCenter Server and what network technology will you use? The wizard adjust based on your choices. For a base level in the lab, I’m deploying VDS.

Next step is choosing where to deploy your Supervisor cluster – the Kubernetes cluster that controls all workload Kubernetes clusters in your deployment. This is where you can choose and associate a vSphere Zone to allow for scale and multi-zone resiliency.

You’ll choose a storage policy – easy peasy.

Next, your load balancer configuration – as noted from above for HAProxy, will be required to input and associate the stand-alone HAProxy or HAProxy cluster.

Next, you’ll use your planned management network

Lastly, your workloads will need a place to run. This is where the Kubernetes API, workloads and services will run – but the API needs to be accessible to interact with other services. Behind the covers, the Kubernetes cluster will run in its own private IP space, that is not of real concern.

Deploy.

Monitor.

Done.

For users getting started on vSphere 8 and Tanzu, after you get everything setup and configured, you’ll likely hit a snag. VMware’s GitHub hosts a quick start guide and YAML manifest for deploying your first Tanzu workload cluster. It will fail with the following error:

Error from server (unable to find a compatible full version matching version hint "1.18" and default OS labels: "os-arch=amd64,os-name=photon,os-type=linux,os-version=3.0".

vCenter and Kubernetes Versioning

Why does this error? This is a brand new cluster, after all? The answer is a versioning issue.

Tanzu Kubernetes Grid is tied to the vCenter version and since we are running vSphere 8 and vCenter 8, the release has specific Kubernetes releases it supports. K8s 1.18 is not compatible. vCenter 8.0.0.10100 releases with Tanzu Kubernetes Grid 1.6.1. The compatible versions for vCenter 8.0.0.10100 are Kubernetes 1.21, 1.22 and 1.23. The supervisor cluster deployed with vCenter 8.0.0.10100 is Kubernetes 1.23.10.

My VMware team has told me that each vCenter will ship with a specific K8 release and upgrading vCenter is the way to upgrade your Tanzu Kubernetes versions. In addition to the vCenter and Tanzu Supervisor Cluster being tied at the hip, VMware will support the current and two prior K8 releases on each vCenter version.

VMware, for its part, doesn’t do much to help the confusion of versioning – as Tanzu has its own versioning, separate from vCenter and separate from the Kubernetes releases. Tanzu Kubernetes Grid version 1.6.1 has K8 version 1.23.10 as its release. TKG 1.5.4 was K8 1.22.9, TKG 1.4.2 was K8 1.21.8.

Demo YAML File

Source: https://raw.githubusercontent.com/vsphere-tmm/vsphere-with-tanzu-quick-start/master/manifests/tkc.yaml

apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TanzuKubernetesCluster
metadata:
  name: tkc-1
spec:
  distribution:
    version: v1.18
  topology:
    controlPlane:
      class: guaranteed-small
      count: 1
      storageClass: kubernetes-demo-storage
    workers:
      class: guaranteed-small
      count: 1
      storageClass: kubernetes-demo-storage
  settings:
    storage:
      classes: ["kubernetes-demo-storage"]           #Named PVC storage classes
      defaultClass: kubernetes-demo-storage          #Default PVC storage class

Troubleshooting the Issue

The first thought is, can my supervisor cluster see the library of Kubernetes images? How can I verify that? Google results point me to a post from Anthony S p i t e r i w i t h s o m e h e l p f u l t r o u b l e s h o o t i n g c o m m a n d s . After verifying that my content library was linked correctly, I ran the following command:

kubectl get virtualmachineimages

This output is a good sign. I can see that I have a LOT of images that are available to this namespace. That’s promising, but no silver bullet why I’m getting this error. Digging a bit further into Google results, I hit another kubectl command:

kubectl get tanzukubernetesreleases

BINGO. In the output, I noticed that the version I requested in the YAML was not “READY” or “COMPATIBLE” in the list.

What needs to be altered to allow this to run on vCenter 8 and Tanzu Kubernetes Grid 1.6.1?

Its a simple change – just adjust the distribution version number to 1.21, 1.22, or 1.23. The error is very helpful, in that it says it can’t find a compatible version.

Working YAML

apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TanzuKubernetesCluster
metadata:
  name: tkc-1
spec:
  distribution:
    version: v1.23
  topology:
    controlPlane:
      class: guaranteed-small
      count: 1
      storageClass: kubernetes-demo-storage
    workers:
      class: guaranteed-small
      count: 1
      storageClass: kubernetes-demo-storage
  settings:
    storage:
      classes: ["kubernetes-demo-storage"]            #Named PVC storage classes
      defaultClass: kubernetes-demo-storage           #Default PVC storage class

Further assumptions – for this file to work, you need to have created a “kubernetes-demo-storage” storage policy and assigned it to the namespace where you’re running this YAML. You also need to have the default VM Class of “guaranteed-small” assigned to the namespace and available for consumption. If that’s in place, run kubectl from the same directory as the YAML file and voila, you will get a working Tanzu Kubernetes Cluster for workloads!

kubectl apply -f ./tkc.yml

Happy clustering!