Overview

My goal for this project was to setup and manage a Kubernetes cluster using Saltstack, the idea being that since I wanted to learn them both if I could automate the setup of the cluster I’d be killing two birds with one stone.

I found a few good blog posts on this but all the ones I found lacked the level of automation I was looking for, some had hard coded values and others had manual commands needing to be run on the minions. For me this wasn’t good enough for what I wanted to do and learn so I’m writing this post detailing how I created my setup and what I learned along the way.

This blog post will have the setup steps I took as reflected in my current setup, if you’re following this blog and not using the same equipment or operating systems your mileage may vary.

Requirements

I did my setup on four Raspberry Pi 4b 8GB models using the 64-bit beta version of Raspbian. Any hardware should be sufficient but you may encounter errors if you try running this on non-debian based distros (the package names may not be the same for certain software).

Repository setup1

Clone my Raspberry Pi Lab Cluster GitHub Repo into /srv on the device you’re denoting as your Salt Master.

Headless Raspberry Pi setup

After flashing your SD card with Raspbian there’s a couple of changes you have to make to enable the ability to SSH into it.

  1. Create an empty file named ssh in the root of the device (/). This tells the device you want SSH enabled.
  2. If using wireless create a file named wpa_supplicant.conf with the following details inside it, obviously changing any details to reflect your setup.
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=US
network={
  ssid="YOUR WIFI SSID HERE"
  psk="YOUR WIFI PSK HERE"
}

When done insert the SD card into the Pi and power it on, after 30-60 seconds you should be able to connect to it with ssh. The default credentials are pi:raspberry.

Once connected to the Pi you should change your hostname with sudo raspi-config. This is because we’re using hostnames within Saltstack instead of IPs, if we leave the default hostname of raspberrypi we’ll have issues.

Setting up the Salt Master

To keep things simple I used the bootstrap-salt.sh script from here2

Once downloaded run the following command sudo ./bootstrap-salt.sh -M -A [SALT-MASTER-HOSTNAME]

After that’s finished installing we have to make a couple of changes to our /etc/salt/master file. This is because we have custom pillar extensions to get certain information from our Kubernetes master for our nodes to use.

  1. Change the extension_modules setting to extension_modules: /srv/ext
  2. Change the ext_pillar setting to:
ext_pillar:
  - kubetoken: []
  - kubecert: []

Once this is done you need to reboot the salt master with sudo systemctl restart salt-master.

Setting up Salt Minions

We use the same bootstrap script2 from earlier but without the -M flag: sudo ./bootstrap-salt.sh -A [SALT-MASTER-HOSTNAME]

When you run this command it’ll bootstrap your minion and tell it to check in with the master at the hostname you provided. After this is done you shouldn’t have to connect to the minion directly anymore and everything can be managed from the master node.

Now we have to accept our minion keys on the salt master. We do this by listing all the keys with sudo salt-key -L and then accepting the keys with sudo salt-key -a minion-id-here.

If you’re following along you should have two IDs ready to be accepted, the master and the minion. Accept both.

To test connectivity to your salt minions you can run sudo salt '*' test.ping on the salt master, you should get some output like this:

rpi-cluster-master:
  True
rpi-cluster-minion-1:
  True

Configuring user accounts

Create an ssh key and copy the public key for ssh key authentication into the the salt/ssh_keys folder in a file named [username].id_rsa.pub.3

I’m using the default Raspberry Pi user of pi, if you wish to change it you can do so in the users.sls file and naming the above mentioned public key to your user.

Configuring a Raspberry Pi device to run Kubernetes

This is where the post starts diverging from other blogs I’ve read. I wanted to remove as many manual tasks as I could as a learning exercise so I did things a bit differently from here on out.

I setup a Saltstack orchestration that would:

  • Set an sbc:rpi-4b-64bit grain on the minion to be used for group targeting rpi-based minions
  • Run a specific configuration saltstate file that does three things:
    • Disable HDMI output (because it’s a headless server and we don’t need it)
    • Setup /boot/cmdline.txt as a managed file (so we can control it to add some cgroup settings to the end of it on every raspberry pi node we setup)
    • Reboot the server to boot with the new /boot/cmdline.txt modifications
  • Run a separate orchestration named setup-server that will:
    • Configure the user accounts from earlier
    • Run some administrative tasks like disabling the swapfile
    • Install Docker and setup a managed daemon.json file
    • Install the Kubernetes packages

To run this orchestration: sudo salt-run state.orch orch.setup-raspberrypi-4b-64bit pillar='{"target-minion": "[MINION-HOSTNAME-HERE]"}'45

In short this orchestration sets up a the server with everything needed to run Kubernetes and Docker.

Configuring Kubernetes

  • To configure the master: sudo salt-run state.orch orch.setup-k8s-master pillar='{"target-minion": "[MASTER-HOSTNAME-HERE]"}'
  • To configure minions: sudo salt-run state.orch orch.setup-k8s-minion pillar='{"target-minion": "[minion-HOSTNAME-HERE]"}'

After you have configured both a master and a minion you should be able to run sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes on the Salt Master.6

pi@rpi-cluster-master:/srv $ sudo kubectl  --kubeconfig /etc/kubernetes/admin.conf get nodes
NAME                   STATUS   ROLES    AGE   VERSION
rpi-cluster-master     Ready    master   19m   v1.19.2
rpi-cluster-minion-1   Ready    <none>   15m   v1.19.2
rpi-cluster-minion-2   Ready    <none>   14m   v1.19.2
rpi-cluster-minion-3   Ready    <none>   75s   v1.19.2

Conclusion

We now have a setup that allows us to configure the entire environment with two Saltstack commands per node we want to add to our k8s cluster.

Unfortunately I wasn’t able to eliminate all of the manual tasks I wanted to but I’m fairly happy with the automation I’ve created. I still want to automate creating the ssh and wpa_supplicant.conf files, and changing the hostname but since that can’t be done with Saltstack it’s outside the scope of this project.

So this was a high-level overview of my setup and how to deploy it. In the future I may write a follow-up post diving into the specific SLS files and explaining how everything works, particularly how we manage the kubeadm init and join stuff. For those curious you can review the source in GitHub, I’ve tried to comment it explaining things.

  1. At the time of writing this the GitHub Repo is at commit b475486682ad6a12181124268f1de6b8cb679929, if you’re trying to follow along and the repo has been updated since I wrote this simply revert to that commit. ↩︎

  2. To copy+paste: curl -o bootstrap-salt.sh https://bootstrap.saltstack.com ↩︎ ↩︎

  3. Password-based authentication WILL BE DISABLED for the managed user. If you don’t configure the key-based authentication you will not be able to login as that user. ↩︎

  4. If you’re not using a raspberry pi you can replace orch.setup-raspberrypi-4b-64bit with orch.setup-server. This orchestration does everything minus the RPI-specific stuff. ↩︎

  5. If your Salt Master is also a Raspberry Pi you should run this orchestration and target it in the passed in pillar data too. ↩︎

  6. You can now copy the /etc/kubernetes/admin.conf file from your Salt Master to your local computer and control the cluster from there. ↩︎