prefecthq / prefect-helm

Helm charts for deploying Prefect Services

Prefect Helm Charts

This repository contains the official Prefect Helm charts for installing and configuring Prefect on Kubernetes. These charts support multiple use cases of Prefect on Kubernetes depending on the values provided and chart selected including:

Prefect worker

Workers are lightweight polling services that retrieve scheduled runs from a work pool and execute them.

Workers are similar to agents, but offer greater control over infrastructure configuration and the ability to route work to specific types of execution environments.

Workers each have a type corresponding to the execution environment to which they will submit flow runs. Workers are only able to join work pools that match their type. As a result, when deployments are assigned to a work pool, you know in which execution environment scheduled flow runs for that deployment will run.

Prefect agent

Note: Workers are recommended

Agents are part of the block-based deployment model. Work Pools and Workers simplify the specification of a flow's infrastructure and runtime environment. If you have existing agents, you can upgrade from agents to workers to significantly enhance the experience of deploying flows.

Agent processes are lightweight polling services that get scheduled work from a work pool and deploy the corresponding flow runs.

Agents poll for work every 15 seconds by default. This interval is configurable in your profile settings with the PREFECT_AGENT_QUERY_INTERVAL setting.

It is possible for multiple agent processes to be started for a single work pool. Each agent process sends a unique ID to the server to help disambiguate themselves and let users know how many agents are active.

Prefect server

Prefect server is a self-hosted open source backend that makes it easy to observe and orchestrate your Prefect flows. It is an alternative to using the hosted Prefect Cloud platform. Prefect Cloud provides additional features including automations and user management.

Prometheus Prefect Exporter

The Prometheus Prefect Exporter is a tool to pull relevant Prefect metrics from a hosted Prefect Server instance

Usage

Helm must be installed to use the charts. Please refer to Helm's documentation to get started.

TL;DR

helm repo add prefect https://prefecthq.github.io/prefect-helm
helm search repo prefect
helm install my-release prefect/prefect-<chart>

Installing released versions

Charts are automatically versioned and released together. The appVersion and prefectTag version are pinned at package time to the current release of Prefect 2.

The charts are hosted in a Helm repository deployed to the web courtesy of GitHub Pages.

1. Add the Prefect Helm repository to Helm and list available charts and versions:

   helm repo add prefect https://prefecthq.github.io/prefect-helm
   helm repo update
   helm search repo prefect

2. Install the Helm chart

   Each chart will require some specific configuration values to be provided, see the individual chart README's for more information on configuring and installing the chart.

   If chart installation fails, run the same command with --debug to see additional diagnostic information.

   Refer to the Helm install documentation for all options.

The helm charts are tested against Kubernetes 1.26.0 and newer minor versions.

Installing development versions

Development versions of the Helm chart will always be available directly from this Github repository.

1. Clone repository

2. Change to this directory

3. Download the chart dependencies

   helm dependency update

4. Install the chart

   helm install . --generate-name

Upgrading

1. Look up the name of the last release

   helm list

2. Run the upgrade

   # Choose a version to upgrade to or omit the flag to use the latest version
   helm upgrade prefect-server prefect/prefect-server --version 2023.09.07

   For development versions, make sure your cloned repository is updated (git pull) and reference the local chart

   helm upgrade prefect-server .

3. Upgrades can also be used enable features or change options

   helm upgrade prefect-server prefect/prefect-server --set newValue=true

Important notes about upgrading

• Updates will only update infrastructure that is modified.
• You will need to continue to set any values that you set during the original install (e.g. --set namespaceOverride=prefect or --values path/to/values.yaml).
• If you are using the postgresql subchart with an autogenerated password, it will complain that you have not provided that password for the upgrade. Export the password as the error asks then set it within the subchart using:

  helm upgrade ... --set postgresql.postgresqlPassword=$POSTGRESQL_PASSWORD

Options:

See comments in values.yaml.

Security context

By default, the worker (or agent), and server run as an unprivileged user with a read-only root filesystem. You can customize the security context settings for both the worker and server in the values.yaml file for your use case.

If you need to install system packages or configure other settings at runtime, you can configure a writable filesystem and run as root by configuring the pod and container security context accordingly:

podSecurityContext:
  runAsUser: 0
  runAsNonRoot: false
  fsGroup: 0

containerSecurityContext:
  runAsUser: 0
  # this must be false, since we are running as root
  runAsNonRoot: false
  # make the filesystem writable
  readOnlyRootFilesystem: false
  # this must be false, since we are running as root
  allowPrivilegeEscalation: false

If you are running in OpenShift, the default restricted security context constraint will prevent customization of the user. In this case, explicitly configure the runAsUser settings to null to use OpenShift-assigned settings:

podSecurityContext:
  runAsUser: null
  fsGroup: null

containerSecurityContext:
  runAsUser: null

The other default settings, such as a read-only root filesystem, are suitable for an OpenShift environment.

Additional permissions for Prefect agents

Dask

If you are running flows on your worker’s pod (i.e. with Process infrastructure), and using the Dask task runner to create Dask Kubernetes clusters, you will need to grant the following permissions within values.yaml.

role:
  extraPermissions:
    - apiGroups: [""]
      resources: ["pods", "services"]
      verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
    - apiGroups: ["policy"]
      resources: ["poddisruptionbudgets"]
      verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

Version support policy

Prefect follows the upstream Kubernetes support policy, meaning that we test against the three most recent minor version releases of Kubernetes. The charts may be compatible with older releases of Kubernetes, however, we do not test against those versions and may choose to reject issues or patches to add support.

Troubleshooting

The database deploys correctly but other services fail with "bad password"

If you are using the subchart deployed database with persistence enabled, it is likely the password for the user has persisted between deployments in the PVC for the database but the secret has been regenerated by the Helm chart and does not match. Deploy and set the 'postgresql.auth.existingSecret' option or set a constant password at postgresql.auth.password.

Contributing

Contributions to the Prefect Helm Charts are always welcome! We welcome your help - whether it's adding new functionality, tweaking documentation, or anything in between. In order to successfully contribute, you'll need to fork this repository and commit changes to your local prefect-helm repo. You can then open a PR against this upstream repo that the team will review!

Documentation

Please make sure that your changes have been linted & the chart documentation has been updated. The easiest way to accomplish this is by installing pre-commit.

Testing & validation

Make sure that any new functionality is well tested! You can do this by installing the chart locally, see above for how to do this.

Opening a PR

A helpful PR explains WHAT changed and WHY the change is important. Please take time to make your PR descriptions as helpful as possible. If you are opening a PR from a forked repository - please follow these docs to allow prefect-helm maintainers to push commits to your local branch.

prefect-helm's Issues

`workQueueName` also needs a value for the agent to run

Kind of related, but workQueueName also needs a value for the agent to run, due to the addition of the -q variable here

prefect-helm/charts/prefect-agent/templates/deployment.yaml

Line 53 in 8434d5e

command: ["prefect", "agent", "start", "-q", {{ .Values.agent.config.workQueueName }}]

Originally posted by @padbk in #53 (comment)

Deployments do not reap subprocesses correctly

In our deployment configurations, we set command (which corresponds to a Dockerfile ENTRYPOINT) to prefect server:

prefect-helm/charts/prefect-server/templates/deployment.yaml

Line 55 in 92f1ee7

command: ["prefect", "server", "start", "--host", "0.0.0.0", "--log-level", "WARNING", "--port", {{ .Values.service.port | quote }}]

This means that we override the default behavior, which uses tini to reap zombie processes.

We can resolve this issue by either changing from command to args (thus using the default entrypoint), or configuring the chart to invoke tini and the entrypoint.sh script instead: https://github.com/PrefectHQ/prefect/blob/36656858fdec1f2636920c56a8ee5c31f3e56bd3/Dockerfile#L124-L126

Prefect 2.0 UI not working when ingress enabled

Helm installed Prefect 2.0 with postgresql as backend, and ingress enabled with host address prefect.example.com

prefect config set PREFECT_API_URL="http://prefect.example.com/api"

the flow run states being stored in the database successfully, you can query them in the database table.

But nothing shown in the UI, because all the API calls from the UI page still go to local server: http://127.0.0.1:4200/api/

PREFECT_API_KEY is required in the agent chart

In the previous release, it was possible to omit the prefect API secret key configuration: https://github.com/PrefectHQ/prefect-helm/blob/0.6.1/charts/prefect-agent/templates/agent/deployment.yaml#L38

With the new release, it is now mandatory to provide a secret reference and if we want to use the agent only with the deployed orion API, we have to create a fake k8s secret. Any particular reason for that change? I can do a PR if needed.

Force useSubChart to false if postgres.enabled is false

Summary

If users disable PostgreSQL, the default behavior is to install PostgreSQL from the subchart anyways.

Observed behavior

Run helm install orion prefect/prefect-orion --set=postgresql.enabled=false, and you will have a PostgreSQL installation.

Expected behavior

When suppressing use of PostgreSQL, we should default to SQLite and thus have no need to install the PostgreSQL chart. Thus we should avoid installing PostgreSQL if postgresql.enabled is false.

Debugging

You can observe this behavior by running the following command: helm template orion ./charts/prefect-orion --set=postgresql.enabled=false from the repository root:

# Source: prefect-orion/charts/postgresql/templates/primary/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: orion-postgresql
  namespace: "prefect-orion"
  labels:
    app.kubernetes.io/name: postgresql
    helm.sh/chart: postgresql-11.6.26
    app.kubernetes.io/instance: orion
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: primary

Users have to specify both postgres.enabled=false and postgresql.useSubChart=false to get an installation of Orion backed by SQLite only.

Publish charts on ArtifactHub

Docs here: https://artifacthub.io/docs/topics/repositories/helm-charts/

`existingSecret` config path documentation is wrong

Documented path is postgresql.auth.existingSecret

Path used in source code is postgresql.existingSecret

Either doc or sources should be fixed.

Because of that issue, it currently takes a while to figure out why external db authentication does not work.

semver - prefect-agent helm chart should use semver releases (instead of dates)

The current release numbers being dates don't agree well with dependency getter bots like renovate or the fluxcd auto update feature.

After further investigation, it seems that the change to use a zero trailing for month has broken the dependency autoupdate.

prefect-version does not match image when prefectTag set

The prefect-version label comes from the chart and does not match prefectTag when set.

eg:

helm upgrade --install prefect-orion prefect/prefect-orion --version=2022.11.11 --set orion.image.prefectTag=2.7.0-python3.9
...

kubectl get pods -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image} {.metadata.labels.prefect-version}{"\n"}{end}'

prefect-agent-5bf86d6985-cz2jl	prefecthq/prefect:2.7.0-python3.9 2.6.7
prefect-orion-886f48798-t9ts5	prefecthq/prefect:2.7.0-python3.9 2.6.7

ie: tag is prefecthq/prefect:2.7.0-python3.9 but prefect-version is 2.6.7

Is `publicApiUrl` for `PREFECT_API_URL` or `PREFECT_ORION_UI_API_URL`?

Before I have

my-values.yaml

orion:
  env:
    - name: PREFECT_API_URL
      value: http://prefect-orion.hm-prefect.svc:4200/api
    - name: PREFECT_ORION_UI_API_URL
      value: http://localhost:4200/api

postgresql:
  auth:
    password: passw0rd

and it works.

I found publicApiUrl here.

However, I am confused for the naming.

publicApiUrl seems replacing PREFECT_API_URL.

But based on the comment:

 # -- sets PREFECT_ORION_UI_API_URL; should be publicly accessible API URL; UI will not work unless set 
 publicApiUrl: "" 

publicApiUrl seems replacing PREFECT_ORION_UI_API_URL.

Is publicApiUrl to replace PREFECT_API_URL or PREFECT_ORION_UI_API_URL? Please clarify, thanks! 😃

No way to customize api ingress labels?

Hi - i see you can specify the api ingress' annotations.... but there is no way to customize the labels? Can this functionality please be added?

replace soon-to-be deprecated set-output syntax in GHA

https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

5/31/23 is the cutoff

Helm Tutorial Request

Requests for guided walk through of helm installation of getting a worker running for cloud.

Any step-by-step instructions would be greatly appreciated! Thanks

Prefect server should have liveness and readyness probe

As a DevOps team relying on Kubernetes to give us the health of the system, we need to have Prefect to provide us with the status of the pod using the liveness and readiness probe

At the moment, the Prefect server is missing endpoints like

• ping => used for liveness
• health => used for readiness

I'm also raising an issue on the Prefect repository to get these endpoints added.

`publicApiUrl` does not work, while `PREFECT_ORION_UI_API_URL` works

I found is that if I use publicApiUrl based on

prefect-helm/charts/prefect-orion/values.yaml

Lines 35 to 36 in 5c56dea

	# -- sets PREFECT_ORION_UI_API_URL; should be publicly accessible API URL; UI will not work unless set
	publicApiUrl: ""

This won't work:

my-values.yaml

orion:
  publicApiUrl: http://localhost:4200/api
  env:
    - name: PREFECT_API_URL
      value: http://prefect-orion.hm-prefect.svc:4200/api
postgresql:
  auth:
    # username: prefect
    password: passw0rd

It will give me the error

Can't connect to Orion API at http://prefect-orion.hm-prefect.svc:4200/api. Check that it's accessible from your machine.

When I open http://localhost:4200

Using PREFECT_ORION_UI_API_URL works

orion:
  env:
    - name: PREFECT_API_URL
      value: http://prefect-orion.hm-prefect.svc:4200/api
    - name: PREFECT_ORION_UI_API_URL
      value: http://localhost:4200/api

postgresql:
  auth:
    password: passw0rd

Missing ClusterRole and ClusterRoleBinding definitions for prefect-agent / prefect-worker

Missing cluster role definitions result in 403 errors when trying to deploy on the cluster into a dedicated namespace.

Submission failed. kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'd8c61061-378f-4686-8163-07b21c220c17', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '903a45a7-c309-4b50-9720-817e715a41a4', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a496d18f-252e-4832-8af8-99bab3405eea', 'Date': 'Mon, 17 Apr 2023 18:00:51 GMT', 'Content-Length': '346'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:prefect:prefect-agent\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}

I was able to get this resolved by adding the following cluster role and cluster role binding definitions:

apiVersion: {{ include "common.capabilities.rbac.apiVersion" . }}
kind: ClusterRole
metadata:
  name: {{ include "common.names.fullname" . }}
rules:
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list"]

apiVersion: {{ include "common.capabilities.rbac.apiVersion" . }}
kind: ClusterRoleBinding
metadata:
  name: {{ include "common.names.fullname" . }}
subjects:
  - kind: ServiceAccount
    name: {{ template "agent.serviceAccountName" . }}
    namespace: {{ include "common.names.namespace" . | quote }}
roleRef:
  kind: ClusterRole
  name: {{ template "common.names.fullname" . }}
  apiGroup: rbac.authorization.k8s.io

Prefect Agent Chart: would be useful to tag the prefect version on the deployment

Basically, when I'm looking at k8s objects in the GKE UI, it would be nice to know what version of Prefect the agent is currently on at a glance. Since the best practice for Prefect agents is to be on the latest version, having a label with the current version would help us determine if we need the deployment to pull a new version of the prefect-latest image to be up to date. Also helps with debugging

Each Endpoint is responding with 405 error but there are not error logs.

Update documentation for `PREFECT_API_URL`

Related to #108 and other community messages, the documentation is not clear on what users should set the value of PREFECT_API_URL to.

Braces are missing

prefect-helm/charts/prefect-agent/templates/agent/deployment.yaml

Line 42 in 262165a

value: include "prefect-agent.prefectApiUrl" .

I believe this is supposed to be value: {{ include "prefect-agent.prefectApiUrl" . }}

Job template from Kubernetes configmap

It seems to me that there is currently no option in values.yaml for mounting Kubernetes configmap or any other volume to agent pod. It would be great to allow this for e.g. jobTemplateFilePath.

Add default resources

Users frequently ask for our recommendation on resource requests/limits for prefect agents. As of right now the numbers that I have are:

2048 CPU and 16000(aprox) memory

Invalid semantic version in hpa template

First check

• I added a descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the Prefect documentation for this issue.
• I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

hpa.yaml is specifying a semantic version comparison of '<1.23-0"` here:

https://github.com/PrefectHQ/prefect-helm/blame/abe3dd1ce7979102b0f7209d77d04ad57f5c961a/charts/prefect-server/templates/hpa.yaml#L27-L39

This is an invalid semantic version.

Reproduction

Here's an example of trying to validate the helm charts when you have the following property set to true:

prefect-server:
  server:
    autoscaling:
      # -- enable autoscaling for server
      enabled: true

> helm template . --values deployments/values-prod.yaml --debug
install.go:194: [debug] Original chart version: ""
install.go:211: [debug] CHART PATH: /tools/prefect


Error: template: prefect/charts/prefect-server/templates/hpa.yaml:2:15: executing "prefect/charts/prefect-server/templates/hpa.yaml" at <include "common.capabilities.hpa.apiVersion" .>: error calling include: template: prefect/charts/prefect-server/charts/common/templates/_capabilities.tpl:133:7: executing "common.capabilities.hpa.apiVersion" at <semverCompare "<1.23-0" (include "common.capabilities.kubeVersion" .context)>: error calling semverCompare: Invalid Semantic Version
helm.go:84: [debug] template: prefect/charts/prefect-server/templates/hpa.yaml:2:15: executing "prefect/charts/prefect-server/templates/hpa.yaml" at <include "common.capabilities.hpa.apiVersion" .>: error calling include: template: prefect/charts/prefect-server/charts/common/templates/_capabilities.tpl:133:7: executing "common.capabilities.hpa.apiVersion" at <semverCompare "<1.23-0" (include "common.capabilities.kubeVersion" .context)>: error calling semverCompare: Invalid Semantic Version

### Error

```python3
Error: template: prefect/charts/prefect-server/templates/hpa.yaml:2:15: executing "prefect/charts/prefect-server/templates/hpa.yaml" at <include "common.capabilities.hpa.apiVersion" .>: error calling include: template: prefect/charts/prefect-server/charts/common/templates/_capabilities.tpl:133:7: executing "common.capabilities.hpa.apiVersion" at <semverCompare "<1.23-0" (include "common.capabilities.kubeVersion" .context)>: error calling semverCompare: Invalid Semantic Version
helm.go:84: [debug] template: prefect/charts/prefect-server/templates/hpa.yaml:2:15: executing "prefect/charts/prefect-server/templates/hpa.yaml" at <include "common.capabilities.hpa.apiVersion" .>: error calling include: template: prefect/charts/prefect-server/charts/common/templates/_capabilities.tpl:133:7: executing "common.capabilities.hpa.apiVersion" at <semverCompare "<1.23-0" (include "common.capabilities.kubeVersion" .context)>: error calling semverCompare: Invalid Semantic Version

Versions

2.10.3

Additional context

No response

When attempting to use this chart to deploy I receive an error showing the cloud api URL being a mash of duplicated account and workspace paths

When attempting to deploy a prefect agent there is no secret created. When I did manually create the secret for the cloud it wasn't decoded by the deployment.

[Prefect Orion] Can't connect to Orion API at xxx. Check that it's accessible from your machine.

Experiment 1 (Succeed)

I tried to deploy Prefect Orion in Kubernetes.

1. Create my-values.yaml

   postgresql:
     auth:
       password: passw0rd

2. Run

   helm repo add prefect https://prefecthq.github.io/prefect-helm
   
   helm install \
     prefect-orion \
     prefect/prefect-orion \
     --namespace=hm-prefect \
     --create-namespace \
     --values=my-values.yaml
   
   kubectl port-forward service/prefect-orion --namespace=hm-prefect 4200:4200

When I open http://localhost:4200, everything looks good.

Experiment 2 (Having Issue)

This time only port forwards to a different port.

1. Create my-values.yaml

   postgresql:
     auth:
       password: passw0rd

2. Run

   helm repo add prefect https://prefecthq.github.io/prefect-helm
   
   helm install \
     prefect-orion \
     prefect/prefect-orion \
     --namespace=hm-prefect \
     --create-namespace \
     --values=my-values.yaml
    
   # Note here, only port forwards to a different port, internally it still uses 4200.
   kubectl port-forward service/prefect-orion --namespace=hm-prefect 48838:4200

When I open http://localhost:48838, I got error in the UI

Can't connect to Orion API at http://127.0.0.1:4200/api. Check that it's accessible from your machine.

I feel the UI should use DNS name to contact the server, which is why I did next experiment 3.

Experiment 3 (Having Issue)

This time set the DNS name.

1. Create my-values.yaml

   orion:
     env:
       - name: PREFECT_API_URL
         value: http://prefect-orion.hm-prefect.svc:48838/api
   
   service:
     port: 48838
   
   postgresql:
     auth:
       password: passw0rd

2. Run

   helm repo add prefect https://prefecthq.github.io/prefect-helm
   
   helm install \
     prefect-orion \
     prefect/prefect-orion \
     --namespace=hm-prefect \
     --create-namespace \
     --values=my-values.yaml
   
   kubectl port-forward service/prefect-orion --namespace=hm-prefect 48838:48838

When I open http://localhost:48838 , I got error in the UI

Can't connect to Orion API at http://prefect-orion.hm-prefect.svc:48838/api. Check that it's accessible from your machine.

Any idea why failed to use a different port? Thanks! 😃

Add support for configuring workers

This chart currently installs a Prefect Agent. We need a way to configure it to instead start as a worker:

prefect worker start --pool {work_pool_name} --type kubernetes

Service 'EventsWorker' failed.

I have a very basic test flow which sets cache_result_in_memory=False and fails with python 3.10 and prefect>2.10 but works with prefect==2.9.0. I am testing with a local-only setup so there is no server involved:

@task
def say_hello():
    print("hello!")


@flow(cache_result_in_memory=False)
def testflow() -> None:
    logger = get_run_logger()
    logger.info("START FLOW")
    say_hello()
    logger.info("END FLOW")

$ py -c 'from testflow import testflow; testflow()'

11:14:22.447 | INFO    | prefect.engine - Created flow run 'umber-nyala' for flow 'testflow'
11:14:22.569 | INFO    | Flow run 'umber-nyala' - START FLOW
11:14:22.659 | INFO    | Flow run 'umber-nyala' - Created task run 'say_hello-0' for task 'say_hello'
11:14:22.661 | INFO    | Flow run 'umber-nyala' - Executing 'say_hello-0' immediately...
hello!
11:14:22.920 | INFO    | Task run 'say_hello-0' - Finished in state Completed()
11:14:22.924 | INFO    | Flow run 'umber-nyala' - END FLOW
11:14:24.174 | INFO    | Flow run 'umber-nyala' - Finished in state Completed('All states completed.')
11:14:24.615 | ERROR   | prefect._internal.concurrency.services - Service 'EventsWorker' failed.
Traceback (most recent call last):
  File "/opt/conda/envs/testflow/lib/python3.10/site-packages/prefect/_internal/concurrency/services.py", line 114, in _run
    await self._main_loop()
  File "/opt/conda/envs/testflow/lib/python3.10/site-packages/prefect/_internal/concurrency/services.py", line 127, in _main_loop
    item: T = await self._queue.get()
  File "/opt/conda/envs/testflow/lib/python3.10/asyncio/queues.py", line 159, in get
    await getter
asyncio.exceptions.CancelledError

If I either downgrade to prefect 2.9 or remove cache_result_in_memory=False, it works--everything looks the same except for the EventsWorker service error

Sidecar container option for helm chart

Are we able to add sidecar to agent or server deployments from helm chart? I couldn't see any option like that?

How to use external Postgres when deploy by Helm Chart?

Originally asked at https://discourse.prefect.io/t/how-to-use-external-postgres-when-deploy-by-helm-chart/2501/1
Here is a copy:

---

When deploy Prefect Server into Kubernetes by Helm Chart, I can disable default Postgres by changing this to

postgresql:
  enabled: false

Next step, I am trying to point to a existing Postgres, however, I didn’t find any info in the Helm chart.

Any guide would be appreciate. Thanks! 😃

Add configuration settings inspired by Bitnami's chart templates

Bitnami charts include a number of customization capabilities that may be useful to adopt in this chart as well. They provide a template with some comprehensive values here: https://github.com/bitnami/charts/blob/master/template/CHART_NAME/values.yaml

Open questions:

• Which of these features would be useful to replicate in Prefect's Helm charts?
• Should we use the same value names, or choose different ones?
• How should we handle backward compatibility for users of our existing charts? Or do we make this a breaking change?

Setup permissions so that we can read the kube-system namespace to get its UID.

Prefect uses the kube-system namespace to get the UID for job identification purpose.

See the discussion here: https://prefect-community.slack.com/archives/CL09KU1K7/p1670585894619609?thread_ts=1670582075.035229&cid=CL09KU1K7

password authentication failed for user "prefect"

Following the readme:

$ helm install prefect-orion prefect/prefect-orion                         
....
$ kubectl logs deploy/prefect-orion-api
 ___ ___ ___ ___ ___ ___ _____    ___  ___ ___ ___  _  _
| _ \ _ \ __| __| __/ __|_   _|  / _ \| _ \_ _/ _ \| \| |
|  _/   / _|| _|| _| (__  | |   | (_) |   /| | (_) | .` |
|_| |_|_\___|_| |___\___| |_|    \___/|_|_\___\___/|_|\_|

Configure Prefect to communicate with the server with:

    prefect config set PREFECT_API_URL=http://0.0.0.0:4200/api

View the API reference documentation at http://0.0.0.0:4200/docs

Check out the dashboard at http://0.0.0.0:4200



INFO:     Started server process [8]
INFO:     Waiting for application startup.
ERROR:    Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 635, in lifespan
    async with self.lifespan_context(app):
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 530, in __aenter__
    await self._router.startup()
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 612, in startup
    await handler()
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/api/server.py", line 271, in run_migrations
    await db.create_db()
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/interface.py", line 55, in create_db
    await self.run_migrations_upgrade()
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/interface.py", line 63, in run_migrations_upgrade
    await run_sync_in_worker_thread(alembic_upgrade)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 57, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/alembic_commands.py", line 29, in alembic_upgrade
    alembic.command.upgrade(alembic_config(), revision, sql=dry_run)
  File "/usr/local/lib/python3.10/site-packages/alembic/command.py", line 322, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.10/site-packages/alembic/script/base.py", line 569, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/migrations/env.py", line 147, in <module>
    apply_migrations()
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 208, in wrapper
    return run_async_from_worker_thread(async_fn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 137, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/usr/local/lib/python3.10/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/migrations/env.py", line 140, in apply_migrations
    async with engine.connect() as connection:
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/ext/asyncio/base.py", line 60, in __aenter__
    return await self.start(is_ctxmanager=True)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/ext/asyncio/engine.py", line 157, in start
    await (greenlet_spawn(self.sync_engine.connect))
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 126, in greenlet_spawn
    result = context.throw(*sys.exc_info())
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/future/engine.py", line 406, in connect
    return super(Engine, self).connect()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3315, in connect
    return self._connection_cls(self, close_with_result=close_with_result)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
    else engine.raw_connection()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3394, in raw_connection
    return self._wrap_pool_connect(self.pool.connect, _connection)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3361, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 320, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 884, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 486, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 145, in _do_get
    with util.safe_reraise():
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 266, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 381, in __init__
    self.__connect()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 677, in __connect
    with util.safe_reraise():
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 673, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 578, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 598, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 780, in connect
    await_only(self.asyncpg.connect(*arg, **kw)),
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 68, in await_only
    return current.driver.switch(awaitable)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 121, in greenlet_spawn
    value = await result
  File "/usr/local/lib/python3.10/site-packages/asyncpg/connection.py", line 2093, in connect
    return await connect_utils._connect(
  File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 889, in _connect
    return await _connect_addr(
  File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 781, in _connect_addr
    return await __connect_addr(params, timeout, True, *args)
  File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 839, in __connect_addr
    await compat.wait_for(connected, timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/asyncpg/compat.py", line 66, in wait_for
    return await asyncio.wait_for(fut, timeout)
  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
    return fut.result()
asyncpg.exceptions.InvalidPasswordError: password authentication failed for user "prefect"

ERROR:    Application startup failed. Exiting.
Orion stopped!

Chart version:

$ helm list                                                                
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
prefect-orion   default         1               2022-09-17 22:49:49.142136 +1000 AEST   deployed        prefect-orion-0.6.0     0.6.0      

persistence should under postgresql.primary

prefect-helm/charts/prefect-orion/values.yaml

Line 105 in 4c5cb68

persistence:

And alos I can't find any usage for postgresql.initdbUser, should be primary.initdb.user?

I checked postgresql helm chart here: https://github.com/bitnami/charts/tree/master/bitnami/postgresql

unable to connect local runs to ingress api

I'm able to get the web ui working properly with these custom settings:

server:
  publicApiUrl: "https://example.com/api"
ingress:
  enabled: true
  host:
    hostname: example.com
postgresql:
  auth:
    password: "prefect"

However, when I try to connect to the API from my local laptop I keep getting failed connection errors:

RuntimeError: Cannot create flow run. Failed to reach API at https://example.com/api/.

(after running prefect config set PREFECT_API_URL=https://example.com/api/)

Is there a step that I'm missing here? There's no example yaml file in this repo's README, so it's difficult know if I'm on the right track. Seems like this user had a similar issue but no answer was ever posted.

Thanks in advance for any help 😄

Can't connect to Orion API at

Official Helm Chart for Prefect 2.0, returns me an error while jumping into Orion UI:

Can't connect to Orion API at http://prefect-orion:4200/api. Check that it's accessible from your machine.

Helm values:

## Common parameters
# -- partially overrides common.names.name
nameOverride: ""
# -- fully override common.names.fullname
fullnameOverride: "prefect-orion"
# -- fully override common.names.namespace
namespaceOverride: ""
# -- labels to add to all deployed objects
commonLabels: {}
# -- annotations to add to all deployed objects
commonAnnotations: {}

## Deployment Configuration
orion:
  image:
    # -- orion image repository
    repository: prefecthq/prefect
    ## prefect tag is pinned to the latest available image tag at packaging time.  Update the value here to
    ## override pinned tag
    # -- prefect image tag (immutable tags are recommended)
    prefectTag: 2.7.4-python3.10
    # -- orion image pull policy
    pullPolicy: IfNotPresent
    ## Optionally specify an array of imagePullSecrets.
    ## Secrets must be manually created in the namespace.
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    ## e.g:
    ## pullSecrets:
    ##   - myRegistryKeySecretName
    # -- orion image pull secrets
    pullSecrets: []
    # -- enable orion image debug mode
    debug: false

  # -- array with environment variables to add to orion nodes
  ## env: []
  env:
    - name: PREFECT__SERVER__TELEMETRY__ENABLED
      value: "false"
    - name: PREFECT_API_URL
      value: http://0.0.0.0:4200/api
    - name: PREFECT_ORION_API_HOST
      value: "0.0.0.0"
    - name: PREFECT_ORION_API_PORT
      value: "4200"

  # Autoscaling configuration
  # requests MUST be specified if using an HPA, otherwise the HPA will not know when to trigger a scale event
  autoscaling:
    # -- enable autoscaling for orion
    enabled: false
    # -- minimum number of orion replicas
    minReplicas: 1
    # -- maximum number of orion replicas
    maxReplicas: 5
    # -- target CPU utilization percentage
    targetCPU: 80
    # -- target Memory utilization percentage
    targetMemory: 80

  # -- number of orion replicas to deploy
  replicaCount: 1

  # requests MUST be specified if using an HPA, otherwise the HPA will not know when to trigger a scale event
  resources:
    # -- the requested resources for the orion container
    requests: {}
    # -- the requested limits for the orion container
    limits: {}

  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod
  podSecurityContext:
    # -- set orion pod's security context runAsUser
    runAsUser: 1001
    # -- set orion pod's security context runAsNonRoot
    runAsNonRoot: true
    # -- set orion pod's security context fsGroup
    fsGroup: 1001

  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container
  containerSecurityContext:
    # -- set orion containers' security context runAsUser
    runAsUser: 1001
    # -- set orion containers' security context runAsNonRoot
    runAsNonRoot: true
    # -- set orion containers' security context readOnlyRootFilesystem
    readOnlyRootFilesystem: true
    # -- set orion containers' security context allowPrivilegeEscalation
    allowPrivilegeEscalation: false

  ## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
  # -- extra labels for orion pod
  podLabels: {}

  ## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
  # -- extra annotations for orion pod
  podAnnotations: {}

  ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
  # -- affinity for orion pods assignment
  affinity: {}

  ## ref: https://kubernetes.io/docs/user-guide/node-selection/
  # -- node labels for orion pods assignment
  nodeSelector: {}

  ## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
  # -- tolerations for orion pods assignment
  tolerations: []

  # -- name of existing ConfigMap containing extra env vars to add to orion nodes
  extraEnvVarsCM: ""

  # -- name of existing Secret containing extra env vars to add to orion nodes
  extraEnvVarsSecret: ""

  # -- array with extra volumes for the orion pod
  extraVolumes: []

  # -- array with extra volumeMounts for the orion pod
  extraVolumeMounts: []

## ServiceAccount configuration
serviceAccount:
  # -- specifies whether a ServiceAccount should be created
  create: true
  # -- the name of the ServiceAccount to use. if not set and create is true, a name is generated using the common.names.fullname template
  name: ""
  # -- additional service account annotations (evaluated as a template)
  annotations: {}

## Service configuration
service:
  # -- service type
  type: NodePort
  # -- service port
  port: 4200
  # -- service Cluster IP
  clusterIP: ""
  # -- service port if defining service as type nodeport
  nodePort: ""

  ## ref http://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip
  # -- service external traffic policy
  externalTrafficPolicy: Cluster
  ## -- additional custom annotations for orion service
  annotations: {}

# Ingress configuration
ingress:
  # -- enable ingress record generation for orion
  enabled: false
  ## This is supported in Kubernetes 1.18+ and required if you have more than one IngressClass marked as the default for your cluster .
  ## ref: https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/
  # -- IngressClass that will be be used to implement the Ingress (Kubernetes 1.18+)
  className: ""
  host:
    # -- default host for the ingress record
    hostname: prefect.local
    # -- default path for the ingress record
    path: /
    # -- ingress path type
    pathType: ImplementationSpecific

  ## Use this parameter to set the required annotations for cert-manager, see
  ## ref: https://cert-manager.io/docs/usage/ingress/#supported-annotations
  # -- additional annotations for the Ingress resource. To enable certificate autogeneration, place here your cert-manager annotations.
  annotations: {}
  ## annotations:
  ##   kubernetes.io/ingress.class: nginx
  ##   cert-manager.io/cluster-issuer: cluster-issuer-name

  ## TLS certificates will be retrieved from a TLS secret with name: `{{- printf "%s-tls" .Values.ingress.host.hostname }}`
  ## You can:
  ##   - Create this secret manually within your cluster
  ##   - Rely on cert-manager to create it by setting the corresponding annotations
  ##   - Rely on Helm to create self-signed certificates by setting `ingress.selfSigned=true`
  # -- enable TLS configuration for the host defined at `ingress.host.hostname` parameter
  tls: false
  # -- create a TLS secret for this ingress record using self-signed certificates generated by Helm
  selfSigned: false

  # -- an array with additional hostname(s) to be covered with the ingress record
  extraHosts: []
  ## extraHosts:
  ##   - name: orion.local
  ##     path: /

  # -- an array with additional arbitrary paths that may need to be added to the ingress under the main host
  extraPaths: []
  ## extraPaths:
  ## - path: /*
  ##   backend:
  ##     serviceName: ssl-redirect
  ##     servicePort: use-annotation

  ## ref: https://kubernetes.io/docs/concepts/services-networking/ingress/#tls
  # -- an array with additional tls configuration to be added to the ingress record
  extraTls: []
  ## extraTls:
  ## - hosts:
  ##     - orion.local
  ##   secretName: orion.local-tls

  ## ref: https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-rules
  # -- additional rules to be covered with this ingress record
  extraRules: []
  ## extraRules:
  ## - host: example.local
  ##     http:
  ##       path: /
  ##       backend:
  ##         service:
  ##           name: example-svc
  ##           port:
  ##             name: http

# Postgresql configuration
postgresql:
  enabled: true
  auth:
    # -- determines whether an admin user is created within postgres
    enablePostgresUser: false
    # -- name for a custom database to create
    database: orion
    # -- name for a custom user to create
    username: prefect
    ## This is the password for `username` and will be set within the secret `{fullnameOverride}-postgresql` at the key `password`.
    ## This argument is only relevant when using the Postgres database included in the chart.
    ## For an external postgres connection, you must create and use `existingSecret` instead.
    # -- password for the custom user to create. Ignored if `auth.existingSecret` with key `password` is provided
    password: "prefect"

    ## This secret must contain two key-value pairs where the first key is `connection-string` and the value is the
    ## connection string containing your password (e.g. postgresql+asyncpg://{username}:{password}@{hostname}/{database}).
    ## The second key-value pair has the key `password` and the value is the {password} used in the connection string
    # -- Name of existing secret to use for PostgreSQL credentials.
    existingSecret: null

  # -- PostgreSQL container port
  containerPorts:
    postgresql: 5432

  # externalHostname defines the address to contact an externally
  # managed postgres database instance at. This is not required if
  # `internalPostgres` is `true`
  externalHostname: ""

  # -- enable use of bitnami/postgresql subchart
  useSubChart: true

  ## postgresql configuration below here is only used if using the subchart

  ## Initdb configuration
  ## ref: https://github.com/bitnami/containers/tree/main/bitnami/postgresql#specifying-initdb-arguments
  primary:
    initdb:
      # -- specify the PostgreSQL username to execute the initdb scripts
      user: postgres

    ## persistence enables a PVC that stores the database between deployments. If making changes to the database deployment, this
    ## PVC will need to be deleted for database changes to take effect. This is especially notable when the authentication password
    ## changes on redeploys. This is disabled by default because we do not recommend using the subchart deployment for production deployments.
    persistence:
      # -- enable PostgreSQL Primary data persistence using PVC
      enabled: false
      # -- PVC Storage Request for PostgreSQL volume
      size: 8Gi

  image:
    # -- Version tag, corresponds to tags at https://hub.docker.com/r/bitnami/postgresql/
    tag: 14.3.0

K8s Deployment:

apiVersion: v1
items:
- apiVersion: apps/v1
  kind: Deployment
  metadata:
    annotations:
      deployment.kubernetes.io/revision: "1"
      meta.helm.sh/release-name: prefect
      meta.helm.sh/release-namespace: prefect
    creationTimestamp: "2023-01-15T21:09:34Z"
    generation: 1
    labels:
      app.kubernetes.io/component: orion
      app.kubernetes.io/instance: prefect
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: prefect-orion
      helm.sh/chart: prefect-orion-2022.12.22
      prefect-version: 2.7.4-python3.10
    name: prefect-orion
    namespace: prefect
    resourceVersion: "4241507"
    uid: eadbe031-b390-4307-bafe-4699c674edb1
  spec:
    progressDeadlineSeconds: 600
    replicas: 1
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app.kubernetes.io/component: orion
        app.kubernetes.io/instance: prefect
        app.kubernetes.io/name: prefect-orion
    strategy:
      rollingUpdate:
        maxSurge: 25%
        maxUnavailable: 25%
      type: RollingUpdate
    template:
      metadata:
        creationTimestamp: null
        labels:
          app.kubernetes.io/component: orion
          app.kubernetes.io/instance: prefect
          app.kubernetes.io/managed-by: Helm
          app.kubernetes.io/name: prefect-orion
          helm.sh/chart: prefect-orion-2022.12.22
          prefect-version: 2.7.4-python3.10
      spec:
        containers:
        - command:
          - prefect
          - orion
          - start
          - --host
          - 0.0.0.0
          - --log-level
          - WARNING
          - --port
          - "4200"
          env:
          - name: HOME
            value: /home/prefect
          - name: PREFECT_DEBUG_MODE
            value: "false"
          - name: PREFECT_ORION_DATABASE_CONNECTION_URL
            valueFrom:
              secretKeyRef:
                key: connection-string
                name: prefect-orion-postgresql-connection
          - name: PREFECT__SERVER__TELEMETRY__ENABLED
            value: "false"
          - name: PREFECT_API_URL
            value: http://0.0.0.0:4200/api
          - name: PREFECT_ORION_API_HOST
            value: 0.0.0.0
          - name: PREFECT_ORION_API_PORT
            value: "4200"
          image: prefecthq/prefect:2.7.4-python3.10
          imagePullPolicy: IfNotPresent
          name: prefect-orion
          ports:
          - containerPort: 4200
            protocol: TCP
          resources: {}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1001
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /home/prefect
            name: scratch
            subPathExpr: home
          - mountPath: /tmp
            name: scratch
            subPathExpr: tmp
          workingDir: /home/prefect
        dnsPolicy: ClusterFirst
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext:
          fsGroup: 1001
          runAsNonRoot: true
          runAsUser: 1001
        serviceAccount: prefect-orion
        serviceAccountName: prefect-orion
        terminationGracePeriodSeconds: 30
        volumes:
        - emptyDir: {}
          name: scratch
  status:
    availableReplicas: 1
    conditions:
    - lastTransitionTime: "2023-01-15T21:09:34Z"
      lastUpdateTime: "2023-01-15T21:09:35Z"
      message: ReplicaSet "prefect-orion-77d7c4c97" has successfully progressed.
      reason: NewReplicaSetAvailable
      status: "True"
      type: Progressing
    - lastTransitionTime: "2023-01-15T21:10:30Z"
      lastUpdateTime: "2023-01-15T21:10:30Z"
      message: Deployment has minimum availability.
      reason: MinimumReplicasAvailable
      status: "True"
      type: Available
    observedGeneration: 1
    readyReplicas: 1
    replicas: 1
    updatedReplicas: 1
kind: List
metadata:
  resourceVersion: ""

Automate chart releases when an OS release happens

Currently, when an open source release happens [of prefect], we have to manually generate a release in the prefect-helm repo. Instead, it would be ideal to automate a release of both charts, packaged with the latest prefect version.

Fix update workflow

Our updatecli workflow is failing due to a permissions issue. I think I've seen this before, and the problem is that we're trying to update a workflow from a running workflow.

This is the error:

 ! [remote rejected] helm-version-2023-06-01 -> helm-version-2023-06-01 (refusing to allow a GitHub App to create or update workflow `.github/workflows/agent-lint-and-test.yaml` without `workflows` permission)

We saw this previously when trying to update the gcloud sdk in our build: https://github.com/PrefectHQ/nebula/pull/4403 - this seems to have been fixed in https://github.com/PrefectHQ/nebula/pull/4401

Example of failing run: https://github.com/PrefectHQ/prefect-helm/actions/runs/5149664089

Typo in tl;dr - chart name

The tl;dr says to install the chart with helm install my-release prefecthq/prefect-<chart>. The prefecthq in that command should really be prefect

Default installation stuck in CrashLoopBackOff

Summary

When installing prefect-orion from the Helm chart, without supplying any configuration options, the orion service gets stuck in CrashLoopBackOff.

Observed behavior

The PostgreSQL subchart installs and starts up correctly, but orion has an "invalid password" error and repeatedly crashloops indefinitely. I see the same behavior with both the latest release version (2022.12.6) as well as installing from source.

The problem seems to be that the subchart generates and stores a random password, but orion does not read it.

Expected behavior

I expected the chart to install and start with default settings.

Note: It may be preferable to default to SQLite instead of PostgreSQL.

Steps to reproduce

When installing from the chart repository:

$ helm repo add prefect https://prefecthq.github.io/prefect-helm/
"prefect" has been added to your repositories

$ helm search repo prefect
NAME                    CHART VERSION   APP VERSION     DESCRIPTION                     
prefect/prefect-agent   2022.12.6       2.7.0           Prefect Agent application bundle
prefect/prefect-orion   2022.12.6       2.7.0           Prefect orion application bundle

$ kubectl create namespace prefect-orion
namespace/prefect-orion created

$ kubens prefect-orion
Active namespace is "prefect-orion".

$ helm install orion prefect/prefect-orion

orion has the following logs:

$ k logs prefect-orion-8c8cf59c9-k6bjw

 ___ ___ ___ ___ ___ ___ _____    ___  ___ ___ ___  _  _
| _ \ _ \ __| __| __/ __|_   _|  / _ \| _ \_ _/ _ \| \| |
|  _/   / _|| _|| _| (__  | |   | (_) |   /| | (_) | .` |
|_| |_|_\___|_| |___\___| |_|    \___/|_|_\___\___/|_|\_|

Configure Prefect to communicate with the server with:

    prefect config set PREFECT_API_URL=http://0.0.0.0:4200/api

View the API reference documentation at http://0.0.0.0:4200/docs

Check out the dashboard at http://0.0.0.0:4200



Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 671, in lifespan
    async with self.lifespan_context(app):
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 566, in __aenter__
    await self._router.startup()
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 648, in startup
    await handler()
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/api/server.py", line 345, in run_migrations
    await db.create_db()
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/interface.py", line 55, in create_db
    await self.run_migrations_upgrade()
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/interface.py", line 63, in run_migrations_upgrade
    await run_sync_in_worker_thread(alembic_upgrade)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 69, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/alembic_commands.py", line 29, in alembic_upgrade
    alembic.command.upgrade(alembic_config(), revision, sql=dry_run)
  File "/usr/local/lib/python3.10/site-packages/alembic/command.py", line 322, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.10/site-packages/alembic/script/base.py", line 569, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/migrations/env.py", line 147, in <module>
    apply_migrations()
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 201, in coroutine_wrapper
    return run_async_from_worker_thread(async_fn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 152, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/usr/local/lib/python3.10/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/migrations/env.py", line 140, in apply_migrations
    async with engine.connect() as connection:
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/ext/asyncio/base.py", line 66, in __aenter__
    return await self.start(is_ctxmanager=True)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/ext/asyncio/engine.py", line 157, in start
    await (greenlet_spawn(self.sync_engine.connect))
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 126, in greenlet_spawn
    result = context.throw(*sys.exc_info())
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/future/engine.py", line 406, in connect
    return super(Engine, self).connect()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3315, in connect
    return self._connection_cls(self, close_with_result=close_with_result)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
    else engine.raw_connection()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3394, in raw_connection
    return self._wrap_pool_connect(self.pool.connect, _connection)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3361, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 325, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 888, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 491, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 145, in _do_get
    with util.safe_reraise():
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 210, in raise_
    raise exception
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 271, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 386, in __init__
    self.__connect()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 684, in __connect
    with util.safe_reraise():
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 210, in raise_
    raise exception
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 680, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 578, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 598, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 780, in connect
    await_only(self.asyncpg.connect(*arg, **kw)),
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 68, in await_only
    return current.driver.switch(awaitable)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 121, in greenlet_spawn
    value = await result
  File "/usr/local/lib/python3.10/site-packages/asyncpg/connection.py", line 2092, in connect
    return await connect_utils._connect(
  File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 881, in _connect
    return await _connect_addr(
  File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 773, in _connect_addr
    return await __connect_addr(params, timeout, True, *args)
  File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 831, in __connect_addr
    await compat.wait_for(connected, timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/asyncpg/compat.py", line 56, in wait_for
    return await asyncio.wait_for(fut, timeout)
  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
    return fut.result()
asyncpg.exceptions.InvalidPasswordError: password authentication failed for user "prefect"

The database starts up correctly, and shows the failed connection attempts:

$ kubectl logs orion-postgresql-0
postgresql 00:34:52.94 
postgresql 00:34:52.94 Welcome to the Bitnami postgresql container
postgresql 00:34:52.94 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql
postgresql 00:34:52.94 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues
postgresql 00:34:52.94 
postgresql 00:34:52.96 INFO  ==> ** Starting PostgreSQL setup **
postgresql 00:34:52.98 INFO  ==> Validating settings in POSTGRESQL_* env vars..
postgresql 00:34:52.98 INFO  ==> Loading custom pre-init scripts...
postgresql 00:34:52.98 INFO  ==> Initializing PostgreSQL database...
postgresql 00:34:53.00 INFO  ==> pg_hba.conf file not detected. Generating it...
postgresql 00:34:53.00 INFO  ==> Generating local authentication configuration
postgresql 00:34:53.86 INFO  ==> Starting PostgreSQL in background...
postgresql 00:34:54.09 INFO  ==> Changing password of postgres
postgresql 00:34:54.11 INFO  ==> Creating user prefect
postgresql 00:34:54.12 INFO  ==> Granting access to "prefect" to the database "orion"
postgresql 00:34:54.15 INFO  ==> Setting ownership for the 'public' schema database "orion" to "prefect"
postgresql 00:34:54.18 INFO  ==> Configuring replication parameters
postgresql 00:34:54.27 INFO  ==> Configuring synchronous_replication
postgresql 00:34:54.27 INFO  ==> Configuring fsync
postgresql 00:34:54.32 INFO  ==> Loading custom scripts...
postgresql 00:34:54.32 INFO  ==> Enabling remote connections
postgresql 00:34:54.33 INFO  ==> Stopping PostgreSQL...
waiting for server to shut down.... done
server stopped
postgresql 00:34:54.44 INFO  ==> ** PostgreSQL setup finished! **

postgresql 00:34:54.47 INFO  ==> ** Starting PostgreSQL **
2022-12-10 00:34:54.489 GMT [1] LOG:  pgaudit extension initialized
2022-12-10 00:34:54.495 GMT [1] LOG:  starting PostgreSQL 14.3 on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2022-12-10 00:34:54.496 GMT [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2022-12-10 00:34:54.496 GMT [1] LOG:  listening on IPv6 address "::", port 5432
2022-12-10 00:34:54.501 GMT [1] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2022-12-10 00:34:54.507 GMT [155] LOG:  database system was shut down at 2022-12-10 00:34:54 GMT
2022-12-10 00:34:54.517 GMT [1] LOG:  database system is ready to accept connections
2022-12-10 00:35:13.984 GMT [176] FATAL:  password authentication failed for user "prefect"
2022-12-10 00:35:13.984 GMT [176] DETAIL:  Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"
2022-12-10 00:35:29.941 GMT [193] FATAL:  password authentication failed for user "prefect"
2022-12-10 00:35:29.941 GMT [193] DETAIL:  Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"
2022-12-10 00:35:57.881 GMT [234] FATAL:  password authentication failed for user "prefect"
2022-12-10 00:35:57.881 GMT [234] DETAIL:  Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"

We have the same error when installing from source. There's a secret in this namespace:

$ kubectl get secret orion-postgresql -o yaml
apiVersion: v1
data:
  password: djNlN0FNMFV5Vw==
  postgres-password: VW00dGxXUmlnMw==
kind: Secret
metadata:
  annotations:
    meta.helm.sh/release-name: orion
    meta.helm.sh/release-namespace: prefect-orion
  creationTimestamp: "2022-12-10T00:34:51Z"
  labels:
    app.kubernetes.io/instance: orion
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: postgresql
    helm.sh/chart: postgresql-11.9.13
  name: orion-postgresql
  namespace: prefect-orion
  resourceVersion: "2812118"
  uid: 51487add-2bbf-4f3e-8c8d-eca100138a3a
type: Opaque

The passwords appear randomly generated (good) but orion does not seem to have access to them.

Decoding the base64, password is v3e7AM0UyW and postgres-password is Um4tlWRig3.

There is also a generated secret prefect-orion-postgresql-connection with the connection-string of postgresql+asyncpg://prefect:@orion-postgresql.prefect-orion:5432/orion

Default rolebinding points to `default` service account instead of the created service account

after helm install prefecthq/prefect-agent --values values.yaml --version 0.3.1 --generate-name

kubectl get sa yields

NAME                       SECRETS   AGE
default                    1         3d3h
prefect-agent-1660256443   1         32m

where default already existed, but then kubectl get rolebinding -n my_ns -o yaml yields:

...
  subjects:
  - kind: ServiceAccount
    name: default
...

which means I had to manually apply a modified rolebinding pointing at the correct, newly-created service account in order to create jobs.

It seems to me that if serviceAccount.create == True then the created service account should be the subject of the rolebinding

License?

Can you please add a license for repository?

Wildcard host not valid in ingress template

The ingress template for Orion here will result in helm throwing an error if the value ingress.host.hostname is specified with a wildcard as described here
https://kubernetes.io/docs/concepts/services-networking/ingress/#hostname-wildcards

This can be fixed by updating the line to quote the input, such as

host: {{ .Values.ingress.host.hostname | quote }}

Cannot create resource pods when using Dask

If we define a flow with a Dask task runner and dask_kubernetes.KubeCluster as its cluster_class, the agent cannot create pods in the cluster with the error:

"message":"pods is forbidden: User \"system:serviceaccount:myserviceaccount\" cannot create resource \"pods\" in API group \"\" in the namespace \"mynamespace\"","reason":"Forbidden","details":{"kind":"pods"},"code":403

I can obviously add my own role and rolebinding for the service account created in the prefect-agent helm chart but maybe you were planning to support this?

Can you deploy behind OIDC authentication?

We are trying to deploy prefect via helm behind okta and the server is deployed fine but trying to get everything working together is returning 405s from the API trying to hit any endpoint.

Is this possible or are there patterns in place somewhere I could grok?

[Prefect Agent] Error from server (BadRequest): container "prefect-agent" in pod "prefect-agent-xxx" is waiting to start: CreateContainerConfigError

I tried to deploy Prefect Agent to the Kubernetes. Here is my steps:

helm install \
  prefect-agent \
  prefect/prefect-agent \
  --namespace=hm-prefect \
  --create-namespace

After running above command, it prints

NAME: prefect-agent
LAST DEPLOYED: Thu Dec 29 00:09:13 2022
NAMESPACE: hm-prefect
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Check Prefect agent connections in the prefect UI at "https://api.prefect.cloud/api/accounts//workspaces/"

However, when I check the pod log by

kubectl logs --namespace=hm-prefect prefect-agent-768c7844b4-b2frc

It shows

Error from server (BadRequest): container "prefect-agent" in pod "prefect-agent-768c7844b4-b2frc" is waiting to start: CreateContainerConfigError

Errors caused by new default security settings

Mainly finding these on the agent. Will update this issue as I come across them.

1. Error when executing a flow on the agent:
   FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/opt/prefect']
   Solved by changing the following from true to false:
   containerSecurityContext:
   readOnlyRootFilesystem: false

2. Error when installing additional python packages on the agent:
   ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/.local'
Check the permissions.
   Currently unsolved, but think it is due to the user id 1001 not existing in /etc/passwd

Admin DB user vs Orion DB user distinction / documentation

I have found the comments in the values.yml regarding database users a little confusing. I would have expected that you launch the chart providing, potentially, admin credentials to a database server, and the chart creates a database, a user, and a password to access this database.

However, when looking at the comments https://github.com/PrefectHQ/prefect-helm/blob/main/charts/prefect-orion/values.yaml#L219, https://github.com/PrefectHQ/prefect-helm/blob/main/charts/prefect-orion/values.yaml#L227, I am not fully clear whether these values are for the admin user or for the "application" user

Thank you for your help

Information about `prefectConfig` is missing in the README for `prefect-orion`

Setting prefectConfig is useful to modify prefect's environment variables. However, a documentation, how to use it is missing from the README.

The Chart README seems to be semi-generated. What is a good place to document this?

Unable to use sqlite as database

In version 2022.09.22 of the orion chart is it no longer possible to use sqlite, eg:

helm upgrade --install prefect-orion prefect/prefect-orion --version=2022.09.22 --set postgresql.enabled=false --set postgresql.useSubChart=false

kubectl logs svc/prefect-orion         
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'

 ___ ___ ___ ___ ___ ___ _____    ___  ___ ___ ___  _  _
| _ \ _ \ __| __| __/ __|_   _|  / _ \| _ \_ _/ _ \| \| |
|  _/   / _|| _|| _| (__  | |   | (_) |   /| | (_) | .` |
|_| |_|_\___|_| |___\___| |_|    \___/|_|_\___\___/|_|\_|

Configure Prefect to communicate with the server with:

    prefect config set PREFECT_API_URL=http://0.0.0.0:4200/api

View the API reference documentation at http://0.0.0.0:4200/docs

Check out the dashboard at http://0.0.0.0:4200



Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
Failed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
INFO:     Started server process [8]
INFO:     Waiting for application startup.
ERROR:    Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3361, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 320, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 884, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 486, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 266, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 381, in __init__
    self.__connect()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 677, in __connect
    with util.safe_reraise():
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 673, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 578, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 598, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 291, in connect
    await_only(connection),
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 68, in await_only
    return current.driver.switch(awaitable)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 121, in greenlet_spawn
    value = await result
  File "/usr/local/lib/python3.10/site-packages/aiosqlite/core.py", line 137, in _connect
    self._connection = await future
  File "/usr/local/lib/python3.10/site-packages/aiosqlite/core.py", line 102, in run
    result = function()
  File "/usr/local/lib/python3.10/site-packages/aiosqlite/core.py", line 397, in connector
    return sqlite3.connect(loc, **kwargs)
sqlite3.OperationalError: unable to open database file

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 645, in lifespan
    async with self.lifespan_context(app):
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 540, in __aenter__
    await self._router.startup()
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 622, in startup
    await handler()
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/api/server.py", line 325, in run_migrations
    await db.create_db()
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/interface.py", line 55, in create_db
    await self.run_migrations_upgrade()
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/interface.py", line 63, in run_migrations_upgrade
    await run_sync_in_worker_thread(alembic_upgrade)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 57, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/alembic_commands.py", line 29, in alembic_upgrade
    alembic.command.upgrade(alembic_config(), revision, sql=dry_run)
  File "/usr/local/lib/python3.10/site-packages/alembic/command.py", line 322, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.10/site-packages/alembic/script/base.py", line 569, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/migrations/env.py", line 147, in <module>
    apply_migrations()
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 208, in wrapper
    return run_async_from_worker_thread(async_fn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 137, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/usr/local/lib/python3.10/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/migrations/env.py", line 140, in apply_migrations
    async with engine.connect() as connection:
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/ext/asyncio/base.py", line 60, in __aenter__
    return await self.start(is_ctxmanager=True)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/ext/asyncio/engine.py", line 157, in start
    await (greenlet_spawn(self.sync_engine.connect))
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 126, in greenlet_spawn
    result = context.throw(*sys.exc_info())
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/future/engine.py", line 406, in connect
    return super(Engine, self).connect()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3315, in connect
    return self._connection_cls(self, close_with_result=close_with_result)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
    else engine.raw_connection()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3394, in raw_connection
    return self._wrap_pool_connect(self.pool.connect, _connection)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3364, in _wrap_pool_connect
    Connection._handle_dbapi_exception_noconnection(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2198, in _handle_dbapi_exception_noconnection
    util.raise_(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3361, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 320, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 884, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 486, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 266, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 381, in __init__
    self.__connect()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 677, in __connect
    with util.safe_reraise():
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 673, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 578, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 598, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 291, in connect
    await_only(connection),
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 68, in await_only
    return current.driver.switch(awaitable)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 121, in greenlet_spawn
    value = await result
  File "/usr/local/lib/python3.10/site-packages/aiosqlite/core.py", line 137, in _connect
    self._connection = await future
  File "/usr/local/lib/python3.10/site-packages/aiosqlite/core.py", line 102, in run
    result = function()
  File "/usr/local/lib/python3.10/site-packages/aiosqlite/core.py", line 397, in connector
    return sqlite3.connect(loc, **kwargs)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
(Background on this error at: https://sqlalche.me/e/14/e3q8)

ERROR:    Application startup failed. Exiting.
Orion stopped!

This worked in version 0.6.0

Specifying list of queues with agent's chart has unexpected behaviour

In the agent's helm chart, it's currently not possible to specify several queues as the list of queue names will just be concatenated, with a comma in between due to

prefect-helm/charts/prefect-agent/templates/deployment.yaml

Line 55 in f111427

command: ["prefect", "agent", "start", "-q", '{{ join "," .Values.agent.config.workQueues }}']

Is this known and will it be addressed soon? Thanks