A while back, I had the chance to set up a Kubernetes cluster on a group of GPU-enabled servers at my workplace. Each server had 8 GTX1080Ti NVIDIA GPUs in it, and 512GB of ram, and 72 CPU cores.
The criteria were as follows:
- Resource management – The user should not need to manage the resources of the cluster. The containers should be put into the appropriate server that can serve its requirements. Easily resolved by Kubernetes.
- Easy horizontal scaling : One should be able to easily add new servers to the cluster.
- Offline Install : There is and will be a security air gap between the servers and the external internet world, meaning that we would have to install Kubernetes in an offline way.
- GPU Acceleration: Our users who run tensorflow algorithms required GPU acceleration within their containers.
I jumped at the chance to experiment with containers and K8s and volunteered. The setup looks something like this:
Kubeadm is an installer for Kubernetes, and is well-supported by the K8s community. For our offline installation, we mirror-ed closely the Kubeadm setup steps whenever we could.
However, Kubeadm is still largely online-based, an internet connection is assumed.
You will need a PC with internet access to do this installation for downloading of RPMs and Docker Images. I am doing this on a CentOS 7 system so the repository handling will be yum-based.
Download the Required Kubernetes RPMs
You can use yum –downloadonly to download all required rpms. See the link below for a guide on how to use yum –downloadonly.
The required files are
- kubeadm 1.8.1
- kubectl 1.8.1
- kubelet 1.8.1
- kubernetes-cni 0.5.1-1
- ebtables 2.0.10-15
- ethtool 4.8-1
It’s all here in a zip archive for you lazy ones. These are the ones that I tested against. Of course, you may want to download the latest and greatest ones. Link
Install Kubeadm and Friends
Install kubeadm, kubectl, kubelet and kubernetes-cni and start kubelet services.
yum install *.rpm
To install all the rpm files that you downloaded.
Follow the steps to install Docker here: https://kubernetes.io/docs/setup/independent/install-kubeadm/
It should be available in the repository of your linux distribution. Configure your distribution for local repository. Alternatively, download the docker rpm as well on a live internet machine. (same steps as above using yum –download-only)
yum install docker
Getting the System Ready
Enable and Start Docker Service
systemctl enable docker && systemctl start docker
Turn off the swap file of the system.
sudo swapoff -a
Remove/comment swap entries in /etc/fstab.
sudo vim /etc/fstab
It looks something like this.
/dev/VolGroup00/LogVol02 swap swap defaults 0 0
In k8s.conf, add the lines
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
Download the Docker Images
Kubeadm runs most required Kubernetes components on container images. Which is a great design as the underlying operating system is kept relatively “clean” compared to something like OpenStack.
You can inspect an existing kubeadm installation to see the container list after installing kubeadm on an online machine.
sudo docker images
Those that start with gcr.io/google_containers and weaveworks are what you need.
You can pull the container images one-by-one individually using the following command. Example here is pulling the kube-apiserver-amd64, v1.8.1.
docker pull gcr.io/google_containers/kube-apiserver-amd64:v1.8.1
docker save gcr.io/google_containers/kube-apiserver-amd64:v1.8.1 > kube-apiserver.tar
<repeat for all required containers>
Alternatively, the easy way is to download all the tar files I used for my working install here. Link
Load Docker Images into the Kubernetes Computer
CentOS should come with Python to run python scripts.
If you have downloaded my containers zip bundle v1.8.1_k8s_containers.zip, you can run :
to load all containers in the directory into your Kubernetes Computer’s docker repository. It’s just a script to run docker load command for all the containers in the directory.
At this juncture, your system is ready to run kubeadm. The next part will focus on how to set up the master K8s node and getting the slave nodes to join it. It should have been fairly straightforward, but there were some nitty gritty things that I encountered along the way. Also, I will cover the additional setup required to set up GPU acceleration for each of the 8-GPU equipped servers.
Part 2 – Master, Workers and GPUs