Raspberry Pi Kubernetes Cluster

Table of Contents

Summary

I have a few Raspberry Pi’s laying around from a cardano stake pool I used to run. So I decided to use them to build a simple K3s K8s cluster. I’ll likely move my DNS server here, and that’s about it. In any case, here is what i’m doing for the setup.

Components

3 – Raspberry Pi Model B 8GB
3 – SanDisk 32GB Class 10 MicroSD Card
3 – Canakit including the fan, case etc. OPTIONAL

Build

Build the Raspberry Pis, there’s quite a few docs on this already. Then you need an OS to run. Pretty much anything will work fine, but i’m using the default Rasbian. Flash the Rasberry PI OS onto the MicroSD, using the Raspberry Pi Imager.

Be sure to go into the configuration and enable ssh and set a password. Alternatively you can create a file called “ssh” and add that to the boot folder of the MicroSD.

Initial Configuration

Plug one of the PIs in to the power supply, and connect an ethernet cable.

# SSH Into the node and become root
ssh pi@DHCPIP
sudo su

###
### Optional Network and Vlan configuration ###
# If not using vlans, skip this
apt install vlan -y
nano /etc/network/interfaces.d/vlans

# Add the following (3 is the vlanID, so if you want vlan20, use eth0.20)
auto eth0.3
iface eth0.3 inet manual
  vlan-raw-device eth0

# If wanting to use DHCP, skip this
nano /etc/dhcpcd.conf

# Again either eth0 or eth0.X where X is the vlanID
interface eth0.3
static ip_address=192.168.3.80
static routers=192.168.3.1
static domain_name_servers=192.168.3.6

### Optional Network and Vlan configuration ###
###


# Set the hostname
nano /etc/hostname
node01 # I am using hostname "node01"

nano /etc/hosts

# Edit this line:
127.0.1.1               raspberrypi
# With this:
127.0.1.1               node01

# Update all the packages
sudo su
apt install dnsutils -y
apt update && sudo apt full-upgrade -y
reboot

# Install docker and some additional things
sudo su
apt install -y docker.io
sed -i '$ s/$/ cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1/' /boot/cmdline.txt
reboot

Install K3s

This super condensed guide is loosely based off of this guide here. K3s is pretty easy to install. You basically run a command on one of the raspberry pis, and it will configure this server as master. Then you run a “join” command on the workers, and it will automatically install K3s and join the master.

CNIs and LB/Ingress

By default K3s comes with flannel. For my purposes this will be fine. It also comes with Klipper for L4, and Traefik for L7 Ingress.
I’m going to replace the Klipper with MetalLB, so that I can get a single external IP address to load balance to to my K8s nodes.

K3s on the Master

# Install K3s on the master node
### Check out the K3s documentation for all the flags: https://docs.k3s.io/reference/server-config ###

sudo su
# Default install with no options:
curl -sfL https://get.k3s.io | sh -

# Install and specify a node-ip and cluster-cidr
# --disable servicelb is used to disable Klipper
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC=" --cluster-cidr=172.16.0.0/16 --node-ip 192.168.3.80 --disable servicelb" sh -

# Get the join token
cat /var/lib/rancher/k3s/server/node-token

K3s on the Worker

# Install K3s on the worker node
### Check out the K3s documentation for all the flags: https://docs.k3s.io/reference/server-config ###

# Default install with no options:
curl -sfL https://get.k3s.io | K3S_URL=https://192.168.3.81:6443 K3S_TOKEN=K....f0::server:617....2f sh -

# Install and specify a node-ip
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--node-ip 192.168.3.81" K3S_URL=https://192.168.3.80:6443 K3S_TOKEN=K....f0::server:617....2f sh -

Install MetalLB

After K3s is installed and the nodes are clustered, you can install MetalLB.

# Install Command, be sure to replace the version with the latest version
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml

Lastly we need to apply a yaml file to declare the MetalLB configuration. I have an example file below. Set the addresses range to any IPs that you want to use as the load balancer virtual IP address.

---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: mainpool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.3.90-192.168.3.99

---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: dnspool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.3.6/32

---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: advertisemetallb
  namespace: metallb-system

# Apply the MetalLB config
kubectl apply -f metallb.yaml

Testing

K3s is pretty great because it’s easy to install. A few commands, and all the binaries are there, and the nodes automatically form a cluster. You can even use the kubectl commands without any further installation.

k3s even has their own binary. I haven’t used it yet though.

MetalLB is up and running, see the pods named speaker-x. You will see 1 pod per node in the k8s cluster.

Accessing the Kubernetes Environment

If you are on the master node, your kubeconfig file should be present, and you should be able to run the above commands. The default kubeconfig location in K3s is /etc/rancher/k3s/k3s.yaml.
If you want to access the K8s environment from a different server, you will need to copy the kubeconfig file.

Step 1

# Run this on the master, and copy the output
sudo cat /etc/rancher/k3s/k3s.yaml

Step 2

# Now on the server in which you want to access the k8s cluster, create the kube folder and create the config file.
mkdir ~/.kube
nano ~/.kube/config

# Paste in the contents from the previous step
# Modify the server.
    server: https://127.0.0.1:6443
# Change to the IP of your master node
    server: https://192.168.9.2:6443

Test the Kubectl commands now, you should see them working.

Deploy an Ingress and test

Here’s a quick example of a hello-world nginx Ingress you can test with.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world-nginx
spec:
  selector:
    matchLabels:
      app: hello-world
  replicas: 3
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80



---
apiVersion: v1
kind: Service
metadata:
  name: hello-world
spec:
  ports:
    - port: 80
      protocol: TCP
  selector:
    app:  hello-world


---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world
  annotations:
    kubernetes.io/ingress.class: "traefik"
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: hello-world
            port:
              number: 80

Uninstall

Uninstall K3s on the master node

# Run this command
/usr/local/bin/k3s-uninstall.sh

Uninstall K3s on the worker nodes

# Run this command
/usr/local/bin/k3s-agent-uninstall.sh