dragonflyoss / helm-charts

Dragonfly Helm Charts

Dragonfly Community Helm Charts

Helm charts for Dragonfly Community.

Introduction

Dragonfly is an open source P2P-based file distribution and image acceleration system. It is hosted by the Cloud Native Computing Foundation (CNCF) as an Incubating Level Project.

Installation

Please refer to the document to install Dragonfly & Nydus on Kubernetes.

Documentation

• Install Dragonfly Stack on Kubernetes
• Install Dragonfly on Kubernetes
• Install Nydus on Kubernetes
• Install Dragonfly & Nydus on Kubernetes

Community

Join the conversation and help the community.

• Slack Channel: #dragonfly on CNCF Slack
• Discussion Group: [email protected]
• Developer Group: [email protected]
• Github Discussions: [Dragonfly Discussion Forum][discussion]
• Twitter: @dragonfly_oss
• DingTalk: 22880028764

License

Apache 2.0 License. Please see License File for more information.

helm-charts's Issues

wrong values in helm chart template

Bug report:

$ helm pull dragonfly/dragonfly
$ tar -zxf dragonfly-1.1.33.tgz
$ grep -r Values.metrics ./dragonfly/templates/
./templates/manager/metrics-svc.yaml:{{ toYaml .Values.metrics.service.labels | indent 4 }}
./templates/seed-peer/metrics-svc.yaml:{{ toYaml .Values.metrics.service.labels | indent 4 }}
./templates/seed-client/metrics-svc.yaml:{{ toYaml .Values.metrics.service.labels | indent 4 }}
./templates/trainer/metrics-svc.yaml:{{ toYaml .Values.metrics.service.labels | indent 4 }}
./templates/scheduler/metrics-svc.yaml:{{ toYaml .Values.metrics.service.labels | indent 4 }}

i think should be
.Values.${APPNAME}.metrics.service.labels

fix please

dfinit: restart-container-runtime restart loop

Bug report:

The restart-container-runtime init container is configured to restart the container runtime without any conditions. As a result, the pod remains in an unready state (NotReady) perpetually. This happens because the container runtime is continuously being restarted, preventing the pod from reaching a stable, ready state.

The restart should only happen once if the configuration is changed. So that the next loop could be marked as ready.

Expected behavior:

Daemonset should start normally.

How to reproduce it:

values.yaml with

client:
  enable: true
  config:
    verbose: true
  dfinit:
    enable: true
    config:
      verbose: true
      containerRuntime:
        containerd:
          registries:
            - hostNamespace: docker.io
              serverAddr: https://index.docker.io
              capabilities: ["pull", "resolve"]
            - hostNamespace: ghcr.io
              serverAddr: https://ghcr.io
              capabilities: ["pull", "resolve"]

Environment:

• Dragonfly version: v2.1.49 (chart v1.1.67)
• OS: Linux`
• Kernel (e.g. uname -a): Linux jack-oneill 6.9.3-arch1-1 #1 SMP PREEMPT_DYNAMIC Fri, 31 May 2024 15:14:45 +0000 x86_64 GNU/Linux
• Others:

Logs:

kubectl describe pod:

Details

Name:             dragonfly-client-bgw5s
Namespace:        dragonfly
Priority:         0
Service Account:  default
Node:             e2e/192.168.39.248
Start Time:       Tue, 25 Jun 2024 23:27:38 +0200
Labels:           app=dragonfly
                  component=client
                  controller-revision-hash=7745678fdd
                  pod-template-generation=3
                  release=dragonfly
Annotations:      checksum/config: ff55a474fbf9a76574ac381a461ce0b797d557fdf76759063600387a8eaf0831
                  kubectl.kubernetes.io/restartedAt: 2024-06-25T23:27:37+02:00
Status:           Pending
IP:               192.168.39.248
IPs:
  IP:           192.168.39.248
Controlled By:  DaemonSet/dragonfly-client
Init Containers:
  update-containerd-remove-registry-mirrors:
    Container ID:  containerd://bc64537fca42caecc1a78c1e9b3ae2e307ef1c9e27ef8876c6c34609367f2d6b
    Image:         python:3.12-slim
    Image ID:      docker.io/library/python@sha256:2fba8e70a87bcc9f6edd20dda0a1d4adb32046d2acbca7361bc61da5a106a914
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -cxe
      apt-get update && apt-get install -y jq
      pip install yq
      if tomlq -e '.plugins."io.containerd.grpc.v1.cri".registry.mirrors' /etc/containerd/config.toml > /dev/null; then
        tomlq -i -t 'del(.plugins."io.containerd.grpc.v1.cri".registry.mirrors)' /etc/containerd/config.toml
        nsenter -t 1 -m -- systemctl try-reload-or-restart containerd.service
        echo "containerd config updated"
      else
        echo "Entry does not exist, no changes made"
      fi
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 25 Jun 2024 23:27:38 +0200
      Finished:     Tue, 25 Jun 2024 23:27:42 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/containerd from containerd-config-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jj67n (ro)
  wait-for-scheduler:
    Container ID:  containerd://e79694fd393fd32ec9d161dbab25e1ff8cc023b5c92d227e096c849016f4fcd5
    Image:         docker.io/busybox:latest
    Image ID:      docker.io/library/busybox@sha256:9ae97d36d26566ff84e8893c64a6dc4fe8ca6d1144bf5b87b2b85a32def253c7
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      until nslookup dragonfly-scheduler.dragonfly.svc.cluster.local && nc -vz dragonfly-scheduler.dragonfly.svc.cluster.local 8002; do echo waiting for scheduler; sleep 2; done;
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 25 Jun 2024 23:27:43 +0200
      Finished:     Tue, 25 Jun 2024 23:27:43 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jj67n (ro)
  dfinit:
    Container ID:  containerd://935e0fe5c37bb824fc553fb717cbf40f80bf588b53fe1e01d1645b21ab1954c4
    Image:         docker.io/dragonflyoss/dfinit:v0.1.82
    Image ID:      docker.io/dragonflyoss/dfinit@sha256:4c793f262a9e1db6f55cedc2a7f322a1a01165fc50480b652637f5f7639b8192
    Port:          <none>
    Host Port:     <none>
    Args:
      --log-level=info
      --verbose
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 25 Jun 2024 23:27:44 +0200
      Finished:     Tue, 25 Jun 2024 23:27:44 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/containerd from containerd-config-dir (rw)
      /etc/dragonfly from dfinit-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jj67n (ro)
  restart-container-runtime:
    Container ID:  containerd://bd622dc89080b0d6d65e09078805f399a94bc7603123feae452cd463991441c9
    Image:         docker.io/busybox:latest
    Image ID:      docker.io/library/busybox@sha256:9ae97d36d26566ff84e8893c64a6dc4fe8ca6d1144bf5b87b2b85a32def253c7
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -cx
      nsenter -t 1 -m -- systemctl restart containerd.service
      echo "restart container"
    State:          Waiting
      Reason:       RunContainerError
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jj67n (ro)
Containers:
  client:
    Container ID:  
    Image:         docker.io/dragonflyoss/client:v0.1.82
    Image ID:      
    Ports:         4000/TCP, 4003/TCP, 4002/TCP, 4004/TCP
    Host Ports:    4000/TCP, 4003/TCP, 4002/TCP, 4004/TCP
    Args:
      --log-level=info
      --verbose
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:        0
      memory:     0
    Liveness:     exec [/bin/grpc_health_probe -addr=:4000] delay=15s timeout=1s period=10s #success=1 #failure=3
    Readiness:    exec [/bin/grpc_health_probe -addr=:4000] delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/dragonfly from config (rw)
      /var/lib/dragonfly/ from storage (rw)
      /var/log/dragonfly/dfdaemon/ from logs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jj67n (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 False 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      dragonfly-client
    Optional:  false
  dfinit-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      dragonfly-dfinit
    Optional:  false
  containerd-config-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/containerd
    HostPathType:  DirectoryOrCreate
  storage:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/dragonfly/
    HostPathType:  DirectoryOrCreate
  logs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-jj67n:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              fal/group=default
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  41m   default-scheduler  Successfully assigned dragonfly/dragonfly-client-bgw5s to e2e
  Normal   Pulled     41m   kubelet            Container image "python:3.12-slim" already present on machine
  Normal   Created    41m   kubelet            Created container update-containerd-remove-registry-mirrors
  Normal   Started    41m   kubelet            Started container update-containerd-remove-registry-mirrors
  Normal   Pulled     41m   kubelet            Container image "docker.io/busybox:latest" already present on machine
  Normal   Created    41m   kubelet            Created container wait-for-scheduler
  Normal   Started    41m   kubelet            Started container wait-for-scheduler
  Normal   Pulled     41m   kubelet            Container image "docker.io/dragonflyoss/dfinit:v0.1.82" already present on machine
  Normal   Created    41m   kubelet            Created container dfinit
  Normal   Started    41m   kubelet            Started container dfinit
  Normal   Pulled     41m   kubelet            Container image "docker.io/busybox:latest" already present on machine
  Normal   Created    41m   kubelet            Created container restart-container-runtime
  Warning  Failed     41m   kubelet            Error: error reading from server: EOF

Harbor configuration preheating, error reported "500 Internal Server Error {"message":"READONLY You can't write against a read only replica."}"

Bug report:

P2P provider is health; Error reported when performing preheating operation

Environment:

• Dragonfly version: CHART VERSIONS 1.1.67
• OS: ubuntu 20.04
• Kernel (e.g. uname -a): Linux harbor.xnwl.com 5.4.0-171-generic #189-Ubuntu SMP Fri Jan 5 14:23:02 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
• Others: containerd containerd.io 1.6.33 d2d58213f83a351ca8f528a95fbd145f5654e957

If Containerd InjectConfigPath is true and config_path has not been configured, dragonfly needs to restart containerd dameon.

• Containerd If injectConfigPath is true, and containerd is not configured with config_path, you need to restart containerd.

https://github.com/dragonflyoss/helm-charts/blob/main/charts/dragonfly/templates/dfdaemon/dfdaemon-daemonset.yaml#L406

spec.serviceName on STS should be the same as metadata.name on correlating Service

Bug report:

When Release.Name != dragonfly the following error is spitted in the Seed Peer logs

2023-07-16T06:48:47.621Z	WARN	config/dynconfig_manager.go:124	scheduler x.x.x.x <.Release.Name>-scheduler-0.scheduler.<.Release.Namespace>.svc.cluster.local 8002 has not reachable addresses

How to reproduce it:

• Install Helm chart with a release name different than dragonfly
• Look in Seed Peer logs

Environment:

• Dragonfly version: v2.1.0-beta.1
• OS: Linux

Wrong version of nydus snapshotter in the README

Bug report:

Wrong version of nydus snapshotter in the README.

Expected behavior:

Use the available version

Redis slave pod can't be backup if master is unavailable

I deployed the dragonfly with helm chart version 0.5.26, when redis master is down, all the slave also failed, enter CrashLoopBackOff stat:

dragonfly-redis-slave-0              0/1     CrashLoopBackOff   7          14m
dragonfly-redis-slave-1              0/1     CrashLoopBackOff   7          15m

So, if redis slave can't be backup, why we need to deploy redis slave?

Also, I noticed that in the configuration of manager, only redis master is configed:

        password: dragonfly
        host: dragonfly-redis-master.dragonfly.svc.cluster.local
        port: 6379

How redis ha work in this situation? does the dragonfly support the redis sentinel ?

没有对外部redis和mysql提供支持

helm-charts/charts/dragonfly/templates/manager/manager-deployment.yaml

Line 70 in 775fe8d

command: ['sh', '-c', 'until nslookup {{ .Release.Name }}-{{ default "redis" .Values.redis.fullname }}-master.{{ $.Release.Namespace }}.svc.{{ $.Values.clusterDomain }} && nc -vz {{ .Release.Name }}-{{ default "redis" .Values.redis.fullname }}-master.{{ $.Release.Namespace }}.svc.{{ $.Values.clusterDomain }} {{ .Values.redis.master.service.ports.redis }}; do echo waiting for redis; sleep 2; done;']

nydus-snapshotter change config.toml of the containerd incorrectly

Bug report:

Set nydus-snapshotter.containerRuntime.containerd.enable true will cause pull image failed, refer to https://github.com/dragonflyoss/helm-charts/blob/main/charts/dragonfly-stack/values.yaml#L1518.

1. Test needs to mount /dev/fuse/.
2. Change initContainer shell to https://github.com/dragonflyoss/client/tree/main/dragonfly-client-init.

Expected behavior:

How to reproduce it:

Logs: https://github.com/dragonflyoss/helm-charts/actions/runs/9659058289/job/26641572724

Environment:

• Dragonfly version:
• OS:
• Kernel (e.g. uname -a):
• Others:

Put passwords in secrets instead of configmaps

Bug report:

Redis & MySQL passwords are stored in Configmaps.

Expected behavior:

Password should be in Secrets.

How to reproduce it:

N/A

Environment:

• Dragonfly version: N/A
• OS: N/A
• Kernel (e.g. uname -a): N/A
• Others: N/A

Fail to enable Ingress in minekube v1.25.2

Env

± % minikube version    --components                                                                                                  !9103
minikube version: v1.25.2
commit: 362d5fdc0a3dbee389b3d3f1034e8023e72bd3a7

buildctl:
buildctl github.com/moby/buildkit v0.9.3 8d2625494a6a3d413e3d875a2ff7dd9b1ed1b1a9

containerd:
containerd containerd.io 1.4.12 7b11cfaabd73bb80907dd23182b9347b4245eb5d

crictl:
crictl version v1.21.0

crio:
crio version 1.22.1

crun:
crun version UNKNOWN

ctr:
ctr containerd.io 1.4.12

docker:
Docker version 20.10.12, build e91ed57

dockerd:
Docker version 20.10.12, build 459d0df

podman:
podman version 3.4.2

runc:
runc version 1.0.2

Step

upgrade dragonfly deployment with the following config to enable ingress of manager

manager:
  ingress:
    # -- Enable ingress
    enable: true
    # -- Ingress annotations
    annotations: {}
    # -- Ingress host path
    path: /
    # -- Manager ingress hosts
    hosts: []
    # -- Ingress TLS configuration
    tls: []

get error information:

Error: UPGRADE FAILED: unable to recognize "": no matches for kind "Ingress" in version "extensions/v1beta1" 

Add support for reading in credentials from env or file, so k8s secret can be used

https://kubernetes.io/docs/concepts/configuration/secret/

v2.0.3 features

Refer to dragonflyoss/d7y.io#74

Feature: values.yaml add extraInitContainers configuration

Like https://github.com/elastic/helm-charts/blob/main/elasticsearch/values.yaml#L160-L163

1. Reduced maintenance / 降低维护工作量 (https://github.com/dragonflyoss/helm-charts/blob/main/charts/dragonfly/templates/dfdaemon/dfdaemon-daemonset.yaml#L294-L410)
2. Easy to custom / 易于定制化

dfdaemon supports configuring direct connection scheduler

Feature request:

dfdaemon has a configuration item scheduler.netAddr, which cannot be set in helm

Use case:

When it is necessary to manually assign the scheduler of the dfdamon connection

UI Example:

Add support for resource specifications on initContainers

Feature request:

I would like to able to specify resource requests for initContainers via the values.yaml file. I'm working with a platform that requires resource limits/request to be specified for all containers.

Use case:

For systems that require resource requests/limits on all containers, providing functionality to do this from the values.yaml would significantly increase quality of life and upgradeability. To get around this right now I have to locally modify each initContainer in the templates and if I want to update the helm chart version I have to manually update everything again. This has the potential to lead to mistakes and makes the project more difficult to maintain.

Inject containerRuntime config failed

Bug report:

The config_path setting within the containerd configuration occasionally fails to be injected properly.

Expected behavior:

The config_path is intended to be successfully injected in the end; nonetheless, if injection fails at any point, it is imperative that the process either retries or throws an error to promptly notify the user.

How to reproduce it:

kind config

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker

use kind to install kubernetes

$ kind create cluster --config kind-config.yaml
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.29.2) 🖼
 ✓ Preparing nodes 📦 📦 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✓ Joining worker nodes 🚜
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a nice day! 👋

kind load image

kind load docker-image dragonflyoss/scheduler:v2.1.43
kind load docker-image dragonflyoss/manager:v2.1.43
kind load docker-image dragonflyoss/dfdaemon:v2.1.43
kind load docker-image busybox:latest
kind load docker-image jaegertracing/all-in-one:1.39.0
kind load docker-image bitnami/redis:7.0.8-debian-11-r0
kind load docker-image bitnami/mysql:8.0.31-debian-11-r30
kind load docker-image dragonflyoss/openssl:latest

add charts-config.yaml

containerRuntime:
  containerd:
    enable: true
    injectConfigPath: true
    registries:
      - 'https://docker.io'
      - 'https://ghcr.io'

scheduler:
  image:
    repository: dragonflyoss/scheduler
    tag: v2.1.43
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066

seedPeer:
  image:
    repository: dragonflyoss/dfdaemon
    tag: v2.1.43
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066

dfdaemon:
  image:
    repository: dragonflyoss/dfdaemon
    tag: v2.1.43
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066

manager:
  image:
    repository: dragonflyoss/manager
    tag: v2.1.43
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066

jaeger:
  enable: true

use helm to install dragonfly

$ helm upgrade --cleanup-on-fail \
  --install dragonfly ./dragonfly \
  --namespace dragonfly-system \
  --create-namespace \
  --values charts-config.yaml

check config.toml of containerd in node, there is nothing about config_path

$ docker exec -it kind-worker2 bash
root@kind-worker2:/# grep "config_path" /etc/containerd/config.toml
root@kind-worker2:/#

check log of container update-containerd, find that there is a create error

k -n dragonfly-system logs dragonfly-dfdaemon-4dx8m -c  update-containerd

...

+ tmp=
+ '[[' -z  ]]
+ echo inject config_path into /host/etc/containerd/config.toml
+ grep -q '\[plugins."io.containerd.grpc.v1.cri".registry\]' /host/etc/containerd/config.toml
inject config_path into /host/etc/containerd/config.toml
/bin/sh: can't create /host/etc/containerd/config.toml: Interrupted system call
+ cat
+ echo 'Registry config_path /host/etc/containerd/certs.d added'
+ need_restart=1
Registry config_path /host/etc/containerd/certs.d added

...

Environment:

• Dragonfly version: v2.1.43
• OS: WSL debian12
• Kernel (e.g. uname -a):
  • node of kind: Linux kind-worker2 5.15.146.1-microsoft-standard-WSL2 #1 SMP Thu Jan 11 04:09:03 UTC 2024 x86_64 GNU/Linux
  • wsl kernel: Linux debian-wsl 5.15.146.1-microsoft-standard-WSL2 #1 SMP Thu Jan 11 04:09:03 UTC 2024 x86_64 GNU/Linux
• Others:
  • version of kind node: kindest/node:v1.29.2

nydus-snapshotter uninstalled，containerd config.toml still used nydus，so the node cannot run container again.

Failed to create pod sandbox: rpc error: code = Unavailable desc = failed to create containerd container: connection error: desc = "transport: Error while dialing dial unix /run/containerd-nydus/containerd-nydus-grpc.sock: connect: connection refused": unavailable

Add nydus charts

• Nydus supports yaml format configuration.

Need support for self-signed certificates in harbor

When we use helm to install dragonfly, harbor's self-signed certificate is not currently supported, and may need to be supported through configmap.

Request to have Postgres as a Database option

Feature request:

The Dragonfly manager container currently supports using either MySQL or PostgreSQL:

• dragonflyoss/Dragonfly2#1423
• dragonflyoss/Dragonfly2#1459

I'd like the helm chart value.yaml to also support deploying with this option.

Use case:

My organization already uses PostgreSQL within our environment. Requiring MySQL would add another component instead of using our existing PostgreSQL environment.

UI Example:

N/A