running on kubernetes about subspace

[agonbar]

I'm do not use Kubernetes that much, but as far a I know:

kubernetes/kubernetes#18269

Kubernetes has no way to allow a container to run on the host network, thus making the iptables scripts fail. A good option could be to migrate into an alternative that doesn't require to use iptables on the host machine. Collaboration into this could turn the project into something really usable.

[outbackdingo]

If this were true imm curious how kilo https://github.com/squat/kilo/ accomplishes it ? squat/kilo@7051b9f

[outbackdingo]

re: a comment from the developer at kilo
never heard of subspace but it looks very cool!
regarding that issue, kubernetes absolutely allows you to put a pod in the host network, e.g.:
https://github.com/squat/kilo/blob/master/manifests/kilo-k3s.yaml#L99

Note, you will likely also need: to make the pod privileged (or at lease give it CAP_NET_ADMIN) https://github.com/squat/kilo/blob/master/manifests/kilo-k3s.yaml#L112

[squat]

Hi subspace ppl :) outbackdingo pointed me to subspace earlier in the week and I found the project very cool!
I wrote a shim to allow using Subspace as the UI to define and manage the peers for Kilo. This means that Kilo takes care of the underlying WireGuard for the whole Kubernetes cluster and Subspace provides the SAML and web interfaces for user and peer management.
Under the hood, kilosubspace fakes the wg command so that when Subspace invokes wg set, it actually creates/deletes peers through the Kubernetes API.

https://github.com/squat/kilosubspace

[linktohack]

This should work:

---
# Source: stack/templates/stack.yml
apiVersion: v1
data:
  auth: REDACTED
kind: Secret
metadata:
  name: subspace-default-basic-auth
type: Opaque
---
# Source: stack/templates/stack.yml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: wireguard-bin-wg
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    name: bin-wg
    namespace: wireguard
  hostPath:
    path: /usr/bin
  persistentVolumeReclaimPolicy: Delete
---
# Source: stack/templates/stack.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: bin-wg
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: manual
---
# Source: stack/templates/stack.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: subspace-data
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: local-path
---
# Source: stack/templates/stack.yml
apiVersion: v1
kind: Service
metadata:
  name: subspace-loadbalancer-tcp
spec:
  ports:
  - name: tcp-51820
    port: 51820
    protocol: TCP
    targetPort: 51820
  selector:
    service: subspace
  type: LoadBalancer
---
# Source: stack/templates/stack.yml
apiVersion: v1
kind: Service
metadata:
  name: subspace-loadbalancer-udp
spec:
  ports:
  - name: udp-51820
    port: 51820
    protocol: UDP
    targetPort: 51820
  selector:
    service: subspace
  type: LoadBalancer
---
# Source: stack/templates/stack.yml
apiVersion: v1
kind: Service
metadata:
  name: subspace
spec:
  ports:
  - name: tcp-80
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    service: subspace
  type: ClusterIP
---
# Source: stack/templates/stack.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: subspace
spec:
  selector:
    matchLabels:
      service: subspace
  template:
    metadata:
      labels:
        service: subspace
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node-role.kubernetes.io/master
                operator: In
                values:
                - "true"
      containers:
      - env:
        - name: SUBSPACE_HTTP_HOST
          value: wireguard.kube.linktohack.com
        - name: SUBSPACE_HTTP_INSECURE
          value: "true"
        - name: SUBSPACE_LETSENCRYPT
          value: "false"
        image: subspacecloud/subspace:latest
        name: subspace
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
        volumeMounts:
        - mountPath: /usr/bin/wg
          name: bin-wg
          subPath: wg
        - mountPath: /data
          name: subspace-data
      volumes:
      - name: bin-wg
        persistentVolumeClaim:
          claimName: bin-wg
      - name: subspace-data
        persistentVolumeClaim:
          claimName: subspace-data
---
# Source: stack/templates/stack.yml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/auth-realm: traefik
    ingress.kubernetes.io/auth-secret: subspace-default-basic-auth
    ingress.kubernetes.io/auth-type: basic
  name: subspace-default
spec:
  rules:
  - host: REDACTED
    http:
      paths:
      - backend:
          serviceName: subspace
          servicePort: tcp-80
        path: /

[linktohack]

Generated via my helm chart

~/Downloads/sup❯ helm -n wireguard upgrade --install subspace link/stack -f wireguard.yml --dry-run

And the stack:

services:
  subspace:
    image: subspacecloud/subspace:latest
    cap_add:
      - NET_ADMIN
    ports:
      - 51820:51820
      - 51820:51820/udp
    volumes:
      - bin_wg:/usr/bin/wg
      - subspace_data:/data
    environment:
      - SUBSPACE_HTTP_HOST=wireguard.kube.linktohack.com
      - SUBSPACE_HTTP_INSECURE=true
      - SUBSPACE_LETSENCRYPT=false
    deploy:
      placement:
        constraints:
          - node.role == manager
      labels:
        - traefik.port=80
        - traefik.frontend.rule=Host:REDACTED
        - traefik.frontend.auth.basic=REDACTED
volumes:
  bin_wg:
    driver_opts:
      type: none
      device: /usr/bin
    subPath: wg
  subspace_data:
    driver_opts:
      type: local-path

[mzupan]

@linktohack how are you setting up bin-wg via the manual storageclass? Is that just mounting in /usr/bin on the local node so you have wireguard installed on all k8s nodes?

[linktohack]

Yes, you have to install wireguard on the [specific] node. WG is a kernel module so there is no other way atm. Advantage? You can configure the nodes to communicate via WG, too.

[squat]

You can also use userspace implementations of WireGuard on nodes where you can't install the WireGuard kernel module, e.g. boringtun. There are good Docker packagings of the project, e.g. https://github.com/leonnicolas/boringtun. Using these, you can run a DaemonSet to put boringtun on every node. Even better, you can limit the DaemonSet to only nodes that don't have the kernel module installed by using a node labeler that labels based on kernel module availability: https://github.com/leonnicolas/nkml

An end-to-end example of this can be found in the Kilo documentation: https://github.com/squat/kilo/blob/master/docs/userspace-wireguard.md

[mike-serchenia]

also set MTU sth like 1280