Kestra will go to > 80% CPU and 100% ram if secret in webhook trigger about kestra

[nidomiro]

I think I just found the cause. SECRET_TRIGGER_DOCKER_SERVER_AUTODEPLOY_WEBHOOK_SECRET in the .env file was not base64 encoded but the original secret.
I encoded the value and now it works as expected.

[loicmathieu]

Hi,
Can you try with the new 0.17.5 release?
It should have a fix for that.

[nidomiro]

Unfortunately I can only confirm that the exception does not appear in the log anymore.
The CPU and RAM usage is still the same. I also still need to hard-reset the system (an lxc-container running in Proxmox)

[loicmathieu]

Can you paste here your full flow YAML and the resources allocated to your container?

[nidomiro]

Sure. The value mySuperSecretKey is just a placeholder.

The flow yaml:

id: trigger-docker-server-autodeploy
namespace: hl443
description: Trigger autodeploy for all Docker servers

labels:
  type: autodeploy

variables:
  servers:
    - fqn: docker01.hl443.de
      user: root
    - fqn: docker02.hl443.de
      user: root
    - fqn: docker03.hl443.de
      user: root
    - fqn: nextcloud.hl443.de
      user: root

tasks:
  - id: parallel
    type: io.kestra.plugin.core.flow.EachParallel
    value: "{{ vars.servers }}"
    tasks:

      - id: debugLog
        type: io.kestra.plugin.core.log.Log
        message:
          - "{{ taskrun.value }}"

      - id: trigger-autodeploy
        type: io.kestra.plugin.fs.ssh.Command
        host: "{{ json(taskrun.value)['fqn'] }}"
        username: "{{ json(taskrun.value)['user'] }}"
        authMethod: PUBLIC_KEY
        privateKey: "{{ secret('SSH_ACCESS_KEY_' + json(taskrun.value)['fqn']  | replace({'.': '_'})) }}"
        warningOnStdErr: false
        commands:
          - "source ~/.profile"
          - "cd $HOMELAB_APPS_ROOT"
          - "git pull"
          - "./autodeploy.mts"

triggers:
  - id: on-git-commit
    type: io.kestra.plugin.core.trigger.Webhook
    key: mySuperSecretKey
    #key: "{{ secret('TRIGGER_DOCKER_SERVER_AUTODEPLOY_WEBHOOK_SECRET') }}"


disabled: false

The docker-compose:

version: "3.4"
services:
  postgres:
    image: postgres:16.3
    restart: unless-stopped
    volumes:
      - ${HOMELAB_APPS_ROOT:?}/kestra/data/postgres-data:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: kestra
      POSTGRES_USER: kestra
      POSTGRES_PASSWORD: ${KESTRA_POSTGRES_PASSWORD:?}
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
      interval: 30s
      timeout: 10s
      retries: 10
    networks:
      - default
      - proxynet

  kestra:
    image: kestra/kestra:v0.17.5-full
    restart: unless-stopped
    pull_policy: always
    # Note that this is meant for development only. Refer to the documentation for production deployments of Kestra which runs without a root user.
    user: "root"
    command: server standalone --worker-thread=128
    volumes:
      - ${HOMELAB_APPS_ROOT:?}/kestra/data/kestra-data:/app/storage
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/kestra-wd:/tmp/kestra-wd
    env_file:
      - ${HOMELAB_APPS_ROOT:?}/kestra/.env
    environment:
      KESTRA_CONFIGURATION: |
        datasources:
          postgres:
            url: jdbc:postgresql://postgres:5432/kestra
            driverClassName: org.postgresql.Driver
            username: kestra
            password: ${KESTRA_POSTGRES_PASSWORD:?}
        kestra:
          server:
            basic-auth:
              enabled: false
              username: "[email protected]" # it must be a valid email address
              password: kestra
          repository:
            type: postgres
          storage:
            type: local
            local:
              base-path: "/app/storage"
          queue:
            type: postgres
          tasks:
            tmp-dir:
              path: /tmp/kestra-wd/tmp
          url: http://kestra.hl443.de/
    labels:
      traefik.enable: true
      # Frontend
      traefik.http.routers.kestra.rule: Host(`kestra.hl443.de`)
      traefik.http.routers.kestra.entrypoints: websecure
      traefik.http.routers.kestra.tls.certresolver: myresolver
      traefik.http.services.kestra.loadbalancer.server.port: 8080
      traefik.http.routers.kestra.service: kestra

      traefik.http.routers.kestra-metrics.rule: Host(`kestra-metrics.hl443.de`)
      traefik.http.routers.kestra-metrics.entrypoints: websecure
      traefik.http.routers.kestra-metrics.tls.certresolver: myresolver
      traefik.http.services.kestra-metrics.loadbalancer.server.port: 8081
      traefik.http.routers.kestra-metrics.service: kestra-metrics
    ports:
      - 127.0.0.1:8080:8080
      - 127.0.0.1:9080:8081
    networks:
      - default
      - proxynet
    depends_on:
      postgres:
        condition: service_started


networks:
  default:
  proxynet:
    external: true

The version of the .env file without values:

KESTRA_POSTGRES_PASSWORD=
SECRET_SSH_ACCESS_KEY_docker01_hl443_de=
SECRET_SSH_ACCESS_KEY_docker02_hl443_de=
SECRET_SSH_ACCESS_KEY_docker03_hl443_de=
SECRET_TRIGGER_DOCKER_SERVER_AUTODEPLOY_WEBHOOK_SECRET=

The LXC-Container config:

• 4 Cores (Intel(R) Xeon(R) CPU E3-1230 v6)
• 4 GiB RAM
• 512 MiB SWAP
• 195,9 GiB Bootdisk
• unpriviliged container
• nesting=1
• Kernel: Linux docker03 6.8.4-3-pve SMP PREEMPT_DYNAMIC PMX 6.8.4-3 (2024-05-02T11:55Z) x86_64 GNU/Linux
• OS: debian 12.5
• docker version:

Client: Docker Engine - Community
Version:           26.1.4
API version:       1.45
Go version:        go1.21.11
Git commit:        5650f9b
Built:             Wed Jun  5 11:29:22 2024
OS/Arch:           linux/amd64
Context:           default

Server: Docker Engine - Community
Engine:
 Version:          26.1.4
 API version:      1.45 (minimum version 1.24)
 Go version:       go1.21.11
 Git commit:       de5c9cf
 Built:            Wed Jun  5 11:29:22 2024
 OS/Arch:          linux/amd64
 Experimental:     false
containerd:
 Version:          1.6.33
 GitCommit:        d2d58213f83a351ca8f528a95fbd145f5654e957
runc:
 Version:          1.1.12
 GitCommit:        v1.1.12-0-g51d5e94
docker-init:
 Version:          0.19.0
 GitCommit:        de40ad0

[anna-geller]

closing since you confirmed you found the issue. if the issue persists, feel free to give update here on the issue

[nidomiro]

I would say the issue cannot be closed. The thing I would expect from not encoding the secret is an error in the log or ui, but never to render it unusable due to the high load.

[anna-geller]

Interesting! can you say more and perhaps open a follow-up issue if needed? We do plan to display Logs for each trigger soon so in terms of visibility as to why some triggers (including webhook trigger) is failing to generate executions, it should get much easier to troubleshoot it via Logs #4245