A simple guide to understand and make Prefect work with your own docker-compose configuration.
This allows you to package your Prefect instance for Kubernetes or offline use.
Open and edit the server/.env
file.
All PREFECT_SERVER_*
options are explained in the official documentation and listed in the config.toml
file.
Then you can run :
docker-compose -f server/docker-compose.yml up -d
Insert the following content in file ~/.prefect/config.toml
:
# ~/.prefect/config.toml
debug = true
# base configuration directory (typically you won't change this!)
home_dir = "~/.prefect"
backend = "server"
[server]
host = "http://172.17.0.1"
port = "4200"
host_port = "4200"
endpoint = "${server.host}:${server.port}"
Finally, we need to create a tenant. Execute on your host :
pip3 install prefect
prefect backend server
prefect server create-tenant --name default --slug default
Access the UI at localhost:8080
Agents are services that run your scheduled flows.
Open and edit the agent/config.toml
file.
ℹ️ In each
config.toml
, you will find the172.17.0.1
IP address. This is the IP of the Docker daemon on which are exposed all exposed ports of your containers. This allows containers launched from different docker-compose networks to communicate. Change it if yours is different (check your daemon IP by typingip a | grep docker0
).Here, mine is
192.168.254.1
but the default is generally to172.17.0.1
.
Then you can run :
docker-compose -f agent/docker-compose.yml up -d
ℹ️ You can run the agent on another machine than the one with the Prefect server. Edit the
agent/config.toml
file for that.
Maybe you want to instanciate multiple agents automatically ?
docker-compose -f agent/docker-compose.yml up -d --scale agent=3 agent
This means the Prefect server never stores your code. It just orchestrates the running (optionally the scheduling) of it.
-
When coding a flow, you need first to register it to the Prefect server through a script. In that script, you may ask the server to run your flow 3 times a day, for example.
-
Your code never lies on the Prefect server : this means the code has to be stored somewhere accessible to the agents in order to be executed.
Prefect has a lot of storage options but the most famous are : Local, S3 and Docker.
- Local : saves the flows to be run on disk. So the volume where you save the flows must be shared among your client and your agent(s). Requires your agent to have the same environment than your client (Python modules, packages installed etc... (the same Dockerfile if your agent and client are containers))
- S3 : similar to local, but saves the flows to be run in S3 objects.
- Docker : saves the flows to be run as Docker images to your Docker Registry so your agents can easily run the code.
ℹ️ If your agents are installed among multiple machines, I recommend you to mount a shared directory with SSHFS.
Open the client/config.toml
file and edit the IP to match your Prefect instance. Then you can run :
docker-compose -f client/docker-compose.yml up # Executes weather.py
Now your flow is registered. You can access the UI to run it.
Tutorial for S3 Storage
We will use MinIO as our S3 server.
docker-compose -f client_s3/docker-compose.yml up -d minio # Starts MinIO
-
Go to localhost:9000 create a new bucket named
prefect
by clicking the red (+) button bottom right. -
Open the
client/config.toml
file and edit the IP to match your Prefect instance and S3 server endpoint. Then you can run :
docker-compose -f client_s3/docker-compose.yml up weather # Executes weather.py
Now your flow is registered. You can access the UI to run it.
This method requires our client AND agent containers to have access to Docker so they can package or load the image in which the flow will be executed. We use Docker in Docker for that.
Tutorial for (secure) Docker Storage
A Docker Registry is needed in order to save images that are going to be used by our agents.
-
Open the
client_docker/config.toml
client_docker/docker-compose.yml
files and edit the IP to match your Prefect instance. -
Generate the authentication credentials for our registry
sudo apt install apache2-utils # required to generate basic_auth credentials
cd client_docker/registry/auth && htpasswd -B -c .htpasswd myusername && cd -
To add more users, re-run the previous command without the -c option
- Start the registry
docker-compose -f client_docker/docker-compose.yml up -d registry
- Login to the registry
You need to allow your Docker daemon to push to this registry. Insert this in your /etc/docker/daemon.json
(create if needed) :
{
"insecure-registries": ["172.17.0.1:5000"]
}
Then, run :
docker login http://172.17.0.1:5000 # with myusername and the password you typed
You should see : Login Succeeded
Edit registry credentials in ./agent_docker/docker-compose.yml
and run :
docker-compose -f agent_docker/docker-compose.yml up -d
We're going to push our Docker image with Python dependencies and register our flow.
- Build, tag and push the image
docker build . -f ./client_docker/execution.Dockerfile -t 172.17.0.1:5000/weather/base_image
You must prefix your image by the registry URI
172.17.0.1
docker push 172.17.0.1:5000/weather/base_image
- Register the flow
Edit registry credentials in ./client_docker/docker-compose.yml
and run :
docker-compose -f ./client_docker/docker-compose.yml up weather
Now your flow is registered. You can access the UI to run it.