Let’s go through some of the design principles for my refurbished cluster. I got the farm of Raspberries in early 2019 and I have been expanding a bit after that. I have had this setup running previously, so I am starting with some assumptions. These are not set in stone, but are supposed to limit my options to avoid wasting my time on trying to do everything.
- Avoid SD cards like the plague
- Ignore Wifi
- Run read-only
- Have a golden image
- Trust DHCP
- Boot from NFS
All of these opinions are based on annoying stuff happening.
Avoid SD cards like the plague
A Raspberry Pi boots by default from an SD card. However, the cards will easily reach their lifetime of writes. This can happen by accident by switching on logging by accident, it will happen by running a conventional OS image that doesn’t know to avoid writing to disk and I have read some discussions about heat doing its damage.
The more recent Raspberry Pi firmwares support booting from USB or network through PXE. The model 3B+ has an issue in its boot ROM that fails if DHCP is too slow. I have plenty of those units and I may need to boot from SD, but will not boot the OS from there.
There will also be 2-4 nodes with attached SSD storage that, even over NFS, will beat the read and write performance of the SD cards.
Ignore Wifi
All modern Raspberries have Wifi chips and that would be an easy way to get wireless access. However, there is no PXE for Wifi and cluster traffic will potentially bother my Netflix moments. Also, I prefer to practice in ways that would work in a datacenter. So, antennae off.
Run read-only
The cluster will have a dozen devices with the same architecture and operating system. Nodes may come and go as I allocate them for test use or buy new ones. So, I will be running the root filesystem read-only and follow the “cattle, not pets” idea as much as I can.
I will be using a layered filesystem, but I am not current on which one.
There have been many over the years: onionfs, unionfs and overlayfs.
It seems overlayfs is used by Raspbian, so I am happy with checking that
first.
Have a golden image
It is possible to start with a stock Raspbian image, however SSH is not
enabled, the default user is named pi and many other defaults that are
not the way they should be in a cluster. I have absolutely no wish to
enable SSH, install keys or run upgrades on each. I used to have a
Raspbian image preseeded my way from the latest stable version and I
intend to follow that idea again.
There are security updates and significant changes to the system that must be performed occasionally, but I want to figure out a way to update on one writable host, freeze changes and reboot individual nodes to the new version.
Trust DHCP
In order to avoid allocating static IP addresses to all hosts, I will allocate a VLAN from my home network to Raspberries and assume anything physically connected to the network to boot through PXE. If the host has previously existed based on MAC address, it will regain its automatic identity. Otherwise, it should run through an automated process of firmware upgrades to the desired version and then be joined as a new host to the cluster.
I will configure another VLAN to handle Kubernetes traffic and use the default VLAN as a management network.
Boot from NFS
In order to avoid SD cards and make use of 2-4 SSDs, there needs to be network storage. I have successfully ran iSCSI on Raspberries before, but that may prevent a clean reboot.
So, I will need a means to mount or extract my golden image on the NFS host and keep it up-to date. I have previously used a docker container for this, so it could potentially run on the cluster itself.
The Raspberry Pi networking performance on a model 3B+ is less than 300 Mbps, so in order not to saturate the SSD hosts, I will prefer attaching storage to the 4B units.