Table of Contents
Summary
I have a few Raspberry Pi’s laying around from a cardano stake pool I used to run. So I decided to use them to build a simple K3s K8s cluster. I’ll likely move my DNS server here, and that’s about it. In any case, here is what i’m doing for the setup.
Components
- 3 – Raspberry Pi Model B 8GB
- 3 – SanDisk 32GB Class 10 MicroSD Card
- 3 – Canakit including the fan, case etc. OPTIONAL
Build
Build the Raspberry Pis, there’s quite a few docs on this already. Then you need an OS to run. Pretty much anything will work fine, but i’m using the default Rasbian. Flash the Rasberry PI OS onto the MicroSD, using the Raspberry Pi Imager.
Be sure to go into the configuration and enable ssh and set a password. Alternatively you can create a file called “ssh” and add that to the boot folder of the MicroSD.
Initial Configuration
Plug one of the PIs in to the power supply, and connect an ethernet cable.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 |
# SSH Into the node and become root ssh pi@DHCPIP sudo su ### ### Optional Network and Vlan configuration ### # If not using vlans, skip this apt install vlan -y nano /etc/network/interfaces.d/vlans # Add the following (3 is the vlanID, so if you want vlan20, use eth0.20) auto eth0.3 iface eth0.3 inet manual vlan-raw-device eth0 # If wanting to use DHCP, skip this nano /etc/dhcpcd.conf # Again either eth0 or eth0.X where X is the vlanID interface eth0.3 static ip_address=192.168.3.80 static routers=192.168.3.1 static domain_name_servers=192.168.3.6 ### Optional Network and Vlan configuration ### ### # Set the hostname nano /etc/hostname node01 # I am using hostname "node01" nano /etc/hosts # Edit this line: 127.0.1.1 raspberrypi # With this: 127.0.1.1 node01 # Update all the packages sudo su apt install dnsutils -y apt update && sudo apt full-upgrade -y reboot # Install docker and some additional things sudo su apt install -y docker.io sed -i '$ s/$/ cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1/' /boot/cmdline.txt reboot |
Install K3s
This super condensed guide is loosely based off of this guide here. K3s is pretty easy to install. You basically run a command on one of the raspberry pis, and it will configure this server as master. Then you run a “join” command on the workers, and it will automatically install K3s and join the master.
CNIs and LB/Ingress
By default K3s comes with flannel. For my purposes this will be fine. It also comes with Klipper for L4, and Traefik for L7 Ingress.
I’m going to replace the Klipper with MetalLB, so that I can get a single external IP address to load balance to to my K8s nodes.
K3s on the Master
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# Install K3s on the master node ### Check out the K3s documentation for all the flags: https://docs.k3s.io/reference/server-config ### sudo su # Default install with no options: curl -sfL https://get.k3s.io | sh - # Install and specify a node-ip and cluster-cidr # --disable servicelb is used to disable Klipper curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC=" --cluster-cidr=172.16.0.0/16 --node-ip 192.168.3.80 --disable servicelb" sh - # Get the join token cat /var/lib/rancher/k3s/server/node-token |
K3s on the Worker
|
1 2 3 4 5 6 7 8 |
# Install K3s on the worker node ### Check out the K3s documentation for all the flags: https://docs.k3s.io/reference/server-config ### # Default install with no options: curl -sfL https://get.k3s.io | K3S_URL=https://192.168.3.81:6443 K3S_TOKEN=K....f0::server:617....2f sh - # Install and specify a node-ip curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--node-ip 192.168.3.81" K3S_URL=https://192.168.3.80:6443 K3S_TOKEN=K....f0::server:617....2f sh - |
Install MetalLB
After K3s is installed and the nodes are clustered, you can install MetalLB.
|
1 2 |
# Install Command, be sure to replace the version with the latest version kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml |
Lastly we need to apply a yaml file to declare the MetalLB configuration. I have an example file below. Set the addresses range to any IPs that you want to use as the load balancer virtual IP address.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
--- apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: mainpool namespace: metallb-system spec: addresses: - 192.168.3.90-192.168.3.99 --- apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: dnspool namespace: metallb-system spec: addresses: - 192.168.3.6/32 --- apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: advertisemetallb namespace: metallb-system |
|
1 2 |
# Apply the MetalLB config kubectl apply -f metallb.yaml |
Testing
K3s is pretty great because it’s easy to install. A few commands, and all the binaries are there, and the nodes automatically form a cluster. You can even use the kubectl commands without any further installation.
Accessing the Kubernetes Environment
If you are on the master node, your kubeconfig file should be present, and you should be able to run the above commands. The default kubeconfig location in K3s is /etc/rancher/k3s/k3s.yaml.
If you want to access the K8s environment from a different server, you will need to copy the kubeconfig file.
Step 1
|
1 2 |
# Run this on the master, and copy the output sudo cat /etc/rancher/k3s/k3s.yaml |
Step 2
|
1 2 3 4 5 6 7 8 9 |
# Now on the server in which you want to access the k8s cluster, create the kube folder and create the config file. mkdir ~/.kube nano ~/.kube/config # Paste in the contents from the previous step # Modify the server. server: https://127.0.0.1:6443 # Change to the IP of your master node server: https://192.168.9.2:6443 |
Test the Kubectl commands now, you should see them working.
Deploy an Ingress and test
Here’s a quick example of a hello-world nginx Ingress you can test with.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
--- apiVersion: apps/v1 kind: Deployment metadata: name: hello-world-nginx spec: selector: matchLabels: app: hello-world replicas: 3 template: metadata: labels: app: hello-world spec: containers: - name: nginx image: nginx ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: hello-world spec: ports: - port: 80 protocol: TCP selector: app: hello-world --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: hello-world annotations: kubernetes.io/ingress.class: "traefik" spec: rules: - http: paths: - path: / pathType: Prefix backend: service: name: hello-world port: number: 80 |
Uninstall
Uninstall K3s on the master node
|
1 2 |
# Run this command /usr/local/bin/k3s-uninstall.sh |
Uninstall K3s on the worker nodes
|
1 2 |
# Run this command /usr/local/bin/k3s-agent-uninstall.sh |