NPE beim Deployment in Keycloak pod - trying to add subystem microprofile-metrics-smallrye about helm-charts

[bbfeng1979]

如果你查看数据库，会发现也会报错：PSQLException: FATAL: password authentication failed for user "keycloak。

当升级商店以后，如果重启 keycloak 的 POD 或者增加 POD，则连接数据库出错。
原因分析是由于在应答中没有设置数据库密码，则会每次升级随机生成一个，导致新的 POD 无法连接数据库。解决办法就是增加自己的设置。

# 新建一个密文包含数据库用户密码
   persistence: 
      dbVendor: "postgres"
      deployPostgres: true
      existingSecret: "keycloak-db-credential"
      existingSecretPasswordKey: "passwd"
      existingSecretUsernameKey: "dbuser"
      
# 给数据库设置固定的密码
  postgresql:
    postgresqlPassword: "JruQly2WnT"
      
# 设置以后，升级一下 keycloak，然后增加 POD 和重启 POD 看是否会再出错

[mohbadar]

Hello, I know that this issue has been closed a long ago, but, I really need your help to solve this issue.

This is the log info

2021-06-05 20:44:09,541 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 63) WFLYCLINF0003: Stopped client-mappings cache from ejb container
2021-06-05 20:44:09,572 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("subsystem" => "microprofile-metrics-smallrye")]): java.lang.NullPointerException
        at [email protected]//org.wildfly.extension.microprofile.metrics.MicroProfileMetricsSubsystemAdd$2.execute(MicroProfileMetricsSubsystemAdd.java:86)
        at [email protected]//org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
        at [email protected]//org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
        at [email protected]//org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
        at [email protected]//org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1413)
        at [email protected]//org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:521)
        at [email protected]//org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:472)
        at [email protected]//org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:434)
        at [email protected]//org.jboss.as.server.ServerService.boot(ServerService.java:435)
        at [email protected]//org.jboss.as.server.ServerService.boot(ServerService.java:394)
        at [email protected]//org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:374)
        at java.base/java.lang.Thread.run(Thread.java:829)

2021-06-05 20:44:09,583 ERROR [org.jboss.as.server] (ServerService Thread Pool -- 55) WFLYSRV0022: Deploy of deployment "keycloak-server.war" was rolled back with no failure message

Deployment Env: VM, not a Docker Container
DB: H2
OS: Ubuntu, Linux

[unguiculus]

I just tested on my Mac with Docker Desktop and could not reproduce this. I did not deploy a theme using an init container, though, but I doubt that's the problem. Does this cause startup to fail?

[bbhuston]

@unguiculus I actually just hit the same issue at the OP. This only happens with I attempt use postgres though. Not from a default "no persistance" install.

20:45:21,584 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0028: Stopped deployment keycloak-server.war (runtime-name: keycloak-server.war) in 3486ms
20:45:21,593 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("subsystem" => "microprofile-metrics-smallrye")]): java.lang.NullPointerException
        at org.wildfly.extension.microprofile.metrics.MicroProfileMetricsSubsystemAdd$2.execute(MicroProfileMetricsSubsystemAdd.java:86)
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
        at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
        at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:521)
        at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:472)
        at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:434)
        at org.jboss.as.server.ServerService.boot(ServerService.java:435)
        at org.jboss.as.server.ServerService.boot(ServerService.java:394)
        at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:374)
        at java.lang.Thread.run(Thread.java:748)

The precise helm command and values file that is throwing this error is listed below

command to install chart

helm install --name keycloak --namespace keycloak codecentric/keycloak -f values.yaml

the values.yaml file

init:
  image:
    repository: alpine
    tag: 3.9
    pullPolicy: IfNotPresent
  resources: {}
    # limits:
    #   cpu: "10m"
    #   memory: "32Mi"
    # requests:
    #   cpu: "10m"
    #   memory: "32Mi"

clusterDomain: cluster.local

keycloak:
  replicas: 1

  image:
    repository: jboss/keycloak
    tag: 6.0.1
    pullPolicy: IfNotPresent

    ## Optionally specify an array of imagePullSecrets.
    ## Secrets must be manually created in the namespace.
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    ##
    pullSecrets: []
    # - myRegistrKeySecretName

  hostAliases: []
  #  - ip: "1.2.3.4"
  #    hostnames:
  #      - "my.host.com"

  enableServiceLinks: false

  restartPolicy: Always

  serviceAccount:
    # Specifies whether a service account should be created
    create: false
    # The name of the service account to use.
    # If not set and create is true, a name is generated using the fullname template
    name:

  securityContext:
    fsGroup: 1000

  containerSecurityContext:
    runAsUser: 1000
    runAsNonRoot: true

  ## The path keycloak will be served from. To serve keycloak from the root path, use two quotes (e.g. "").
  basepath: auth

  ## Additional init containers, e. g. for providing custom themes
  extraInitContainers: |

  ## Additional sidecar containers, e. g. for a database proxy, such as Google's cloudsql-proxy
  extraContainers: |

  ## lifecycleHooks defines the container lifecycle hooks
  lifecycleHooks: |
    # postStart:
    #   exec:
    #     command: ["/bin/sh", "-c", "ls"]

  ## Additional arguments to start command e.g. -Dkeycloak.import= to load a realm
  extraArgs: ""

  ## Username for the initial Keycloak admin user
  username: keycloak

  ## Password for the initial Keycloak admin user. Applicable only if existingSecret is not set.
  ## If not set, a random 10 characters password will be used
  password: "FakePwd1234!"

  # Specifies an existing secret to be used for the admin password
  existingSecret: ""

  # The key in the existing secret that stores the password
  existingSecretKey: password

  ## Allows the specification of additional environment variables for Keycloak
  extraEnv: |
    - name: PROXY_ADDRESS_FORWARDING
      value: "true"
    # - name: KEYCLOAK_LOGLEVEL
    #   value: DEBUG
    # - name: WILDFLY_LOGLEVEL
    #   value: DEBUG
    # - name: CACHE_OWNERS
    #   value: "2"
    # - name: DB_QUERY_TIMEOUT
    #   value: "60"
    # - name: DB_VALIDATE_ON_MATCH
    #   value: true
    # - name: DB_USE_CAST_FAIL
    #   value: false

  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              {{- include "keycloak.selectorLabels" . | nindent 10 }}
            matchExpressions:
              - key: role
                operator: NotIn
                values:
                  - test
          topologyKey: kubernetes.io/hostname
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchLabels:
                {{- include "keycloak.selectorLabels" . | nindent 12 }}
              matchExpressions:
                - key: role
                  operator: NotIn
                  values:
                    - test
            topologyKey: failure-domain.beta.kubernetes.io/zone

  nodeSelector: {}
  priorityClassName: ""
  tolerations: []

  ## Additional pod labels
  ## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
  podLabels: {}

  ## Extra Annotations to be added to pod
  podAnnotations: {}

  livenessProbe:
    initialDelaySeconds: 120
    timeoutSeconds: 5
  readinessProbe:
    initialDelaySeconds: 30
    timeoutSeconds: 1

  resources:
    limits:
      cpu: "400m"
      memory: "2048Mi"
    requests:
      cpu: "100m"
      memory: "1024Mi"

  ## WildFly CLI configurations. They all end up in the file 'keycloak.cli' configured in the configmap which is
  ## executed on server startup.
  cli:
    enabled: true
    nodeIdentifier: |
      {{ .Files.Get "scripts/node-identifier.cli" }}

    logging: |
      {{ .Files.Get "scripts/logging.cli" }}

    ha: |
      {{ .Files.Get "scripts/ha.cli" }}

    datasource: |
      {{ .Files.Get "scripts/datasource.cli" }}

    # Custom CLI script
    custom: |

  ## Custom startup scripts to run before Keycloak starts up
  startupScripts: {}
    # mystartup.sh: |
    #   #!/bin/sh
    #
    #   echo 'Hello from my custom startup script!'

  ## Add additional volumes and mounts, e. g. for custom themes
  extraVolumes: |
  extraVolumeMounts: |

  ## Add additional ports, eg. for custom admin console
  extraPorts: |

  podDisruptionBudget: {}
    # maxUnavailable: 1
    # minAvailable: 1

  service:
    annotations: {}
    # service.beta.kubernetes.io/aws-load-balancer-internal: "0.0.0.0/0"

    labels: {}
    # key: value

    ## ServiceType
    ## ref: https://kubernetes.io/docs/user-guide/services/#publishing-services---service-types
    type: ClusterIP

    ## Optional static port assignment for service type NodePort.
    # nodePort: 30000

    port: 80

    # Optional: jGroups port for high availability clustering
    jgroupsPort: 7600

  ## Ingress configuration.
  ## ref: https://kubernetes.io/docs/user-guide/ingress/
  ingress:
    enabled: true
    path: /

    annotations:
      kubernetes.io/ingress.class: nginx
      # kubernetes.io/tls-acme: "true"
      # ingress.kubernetes.io/affinity: cookie

    labels: {}
    # key: value

    ## List of hosts for the ingress
    hosts:
      - keycloak.my.domain.io

    ## TLS configuration
    tls: []
    # - hosts:
    #     - keycloak.example.com
    #   secretName: tls-keycloak

  ## Persistence configuration
  persistence:
    # If true, the Postgres chart is deployed
    deployPostgres: true

    # The database vendor. Can be either "postgres", "mysql", "mariadb", or "h2"
    dbVendor: postgres

    ## The following values only apply if "deployPostgres" is set to "false"

    # Specifies an existing secret to be used for the database password
    existingSecret: ""

    # The key in the existing secret that stores the password
    existingSecretKey: password

    dbName: keycloak
    dbHost: mykeycloak
    dbPort: 5432
    dbUser: keycloak

    # Only used if no existing secret is specified. In this case a new secret is created
    dbPassword: ""

postgresql:
  ### PostgreSQL User to create.
  ##
  postgresqlUsername: keycloak

  ## PostgreSQL Password for the new user.
  ## If not set, a random 10 characters password will be used.
  ##
  postgresqlPassword: ""

  ## PostgreSQL Database to create.
  ##
  postgresqlDatabase: keycloak

  ## Persistent Volume Storage configuration.
  ## ref: https://kubernetes.io/docs/user-guide/persistent-volumes
  ##
  persistence:
    ## Enable PostgreSQL persistence using Persistent Volume Claims.
    ##
    enabled: true

test:
  enabled: true
  image:
    repository: unguiculus/docker-python3-phantomjs-selenium
    tag: v1
    pullPolicy: IfNotPresent
  securityContext:
    fsGroup: 1000
  containerSecurityContext:
    runAsUser: 1000
    runAsNonRoot: true

I am attempting the installation on a Kops 12.x cluster running on AWS. Any suggestions on what may be going on or how I should proceed to debug? Thanks!

[bbhuston]

An update to my post above. I followed the advice suggested in the keycloak stable repo and omitted a name for my helm deployment. Keycloak now runs along with postgres and persistance.

Specifically I ran the command helm install --namespace keycloak codecentric/keycloak -f values.yaml with the values.yaml file I pasted above. Note that I dropped the --name flag. I suspect that naming the release "keycloak" may also be part of the problem, not just the <23 byte requirement my linked URL mentions

[bbhuston]

An update to my post above. I followed the advice suggested in the keycloak stable repo and omitted a name for my helm deployment. Keycloak now runs along with postgres and persistance.

Specifically I ran the command helm install --namespace keycloak codecentric/keycloak -f values.yaml with the values.yaml file I pasted above. Note that I dropped the --name flag. I suspect that naming the release "keycloak" may also be part of the problem, not just the <23 byte requirement my linked URL mentions

Important update to my previous post. It turns out that I could not actually get the codecentric keycloak helm chart to work. The chart that works with the no --name flag fix is the upstream stable/keycloak chart. The exact code I ran to get keycloak working is --namespace keycloak stable/keycloak -f values.yaml

[unguiculus]

This is weird. stable/keycloak is deprecated. The chart was moved here.

A solution could be to explicitly set the pod's hostname. I'll try and get to testing this.

[klafbang]

I got the same error with the chart unable to connect to an external Postgres DB; from the YAML files it seems the root cause is different here, though. I created a separate ticket for my issue, but if you're running Postgres in a non-standard port check #126 for a workaround that worked for me.

[peterbosalliandercom]

anyone a fix for this?

[BenCoughlan15]

@bbhuston , any joy with getting the codecentric version to work?

B

[albrechtflo-hg]

Just FYI, I ran into exactly the same issue today without Helm, just "plain" kubernetes. I use MariaDB as Keycloak DB. Seems to be related to JSON file import on startup, but I have no idea what exactly in the JSON (generated with the same Keycloak using the export command) causes this.

[bbhuston]

The only way I got keycloak to work was to use the deprecated helm chart that is still hosted at the charts/stable repo.

…

On Fri, Mar 20, 2020, 6:12 AM albrechtflo-hg ***@***.***> wrote: Just FYI, I ran into exactly the same issue today without Helm, just "plain" kubernetes. I use MariaDB as Keycloak DB. Seems to be related to JSON file import on startup, but I have no idea what exactly in the JSON (generated with the same Keycloak using the export command) causes this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#102 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFYTD6P3NMX5OK4MRAFKNB3RIM6O7ANCNFSM4IOVEEAQ> .

[unguiculus]

Closing this now. Feel free to open a new ticket for the current version of the chart if you still have issues.