Avi :)

K3S with metallb and nginx-ingress

I followed Greg Jeanmart’s tutorial for this a few years back, but things have changed, as they always do in k8s! I’m also lazy and like to steer close to Helm installing stuff.

This is all on a Debian machine who’s hostname is nuc

Install K3s

First, install k3s, without servicelb (we will use metallb) and without traefik (we will use nginx-ingress):

export K3S_KUBECONFIG_MODE="644" export INSTALL_K3S_EXEC=" --disable servicelb --disable traefik"
curl -sfL https://get.k3s.io | sudo sh -

And you should have a k3s running:

$ ps aux | grep bin/k3[s]
root 4021 101 2.5 1126500 407404 ? Ssl 18:09 0:36 /usr/local/bin/k3s server

Grab the kubeconfig and check:

$ mkdir -p ~/.kube/
$ sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
$ kubectl get nodes
NAME   STATUS   ROLES                  AGE   VERSION
nuc    Ready    control-plane,master   11m   v1.26.4+k3s1

Add the Helm stable chart for the next bits:

$ helm repo add stable https://charts.helm.sh/stable

Install metallb

This is very different; metallb used to be configured using configMaps because that’s traditionally where we put configs. But now it uses customResources presumably for some thoroughly good technical reasons. k3s doesn’t use pod security so we don’t need to label the namespace.

I like namespaces at the best of times, but now that a metallb install has all its CRs kicking about I definitely want to put it in one on its own, but there’s no great need to.

First, install metallb from helm

kubectl create ns metallb
helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb --namespace metallb
helm install metallb metallb/metallb

Now, we create two yaml documents. First, an address pool, which tells metallb which addresses it can use:

$ cat > IPAddressPool.yaml <<EOF
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: k3s-nuc-pool
  namespace: metallb
spec:
  addresses:
  - 192.168.2.100-192.168.2.250
EOF

Secondly, a layer 2 advertisement, which you can think of as roughly an instruction to metallb to make use of an address pool. If you don’t name any address pool in it, all pools are advertised (or used), the only reason to name it here is so it doesn’t look weird and pointless:

$ cat > L2Advertisement.yaml <<EOF
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: k3s-nuc-l2advertisment
  namespace: metallb
spec:
  ipAddressPools:
  - k3s-nuc-pool
EOF

Make sure you’ve got the address pool that you’d like to use configured, and then apply these

kubectl -n metallb apply -f ./IPAddressPool.yaml 
kubectl -n metallb apply -f ./L2Advertisement.yaml

And you should now have a controller and a speaker:

$ kubectl get pods -n metallb
NAME                                  READY   STATUS    RESTARTS   AGE
metallb-controller-777d84cdd5-k76kt   1/1     Running   0          8m26s
metallb-speaker-7xblk                 1/1     Running   0          8m26s

Install nginx-ingress

First, install it from helm:

helm upgrade --install ingress-nginx ingress-nginx \
--repo https://kubernetes.github.io/ingress-nginx \
--namespace ingress-nginx --create-namespace

And then wait a bit, it can take some time but you should get a controller pod:

$ kubectl get pods --namespace=ingress-nginx
NAME                                       READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-f6c55fdc8-4b4d4   0/1     Running   0          37s

Testing!

Cribbing from the nginx project, create a tiny deployment. demo.localdev.me should always resolve to 127.0.0.1 so is really handy when testing locally with name-based routing:

kubectl create deployment demo --image=httpd --port=80
kubectl expose deployment demo
kubectl create ingress demo-localhost --class=nginx --rule="demo.localdev.me/*=demo:80"

We can now use a port-forward to check this:

kubectl port-forward --namespace=ingress-nginx service/ingress-nginx-controller 8080:80

And then, in another shell (or perhaps more ideally not) run a curl. You should get a document back that says ‘It Works!’ and not a 404:

$ curl http://demo.localdev.me:8080/
Handling connection for 8080
<html><body><h1>It works!</h1></body></html>

Now we know the ingress is working we can check if the load-balancer is, too. Nginx will have tried to create a load-balancer:

$ kubectl get service ingress-nginx-controller --namespace=ingress-nginx
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
ingress-nginx-controller   LoadBalancer   10.43.147.53   192.168.2.101   80:30031/TCP,443:32045/TCP   18m

So now, from another host, we should be able to hit demo.localdev.me at that external IP (192.168.2.101 for me, but use what’s in your kubectl output):

$ curl 192.168.2.101 -Hhost:demo.localdev.me
<html><body><h1>It works!</h1></body></html>

Posted

in

by

avi

Tags:

, , ,