This post contains code and commands you can use to deploy Prefect agents to Google Cloud’s Google Kubernetes Engine. The agents stand ready to execute workflows triggered by Prefect projects. One agent can run tasks from multiple projects.

The example here demonstrates how to create a single agent with minimal customization. It is configured with a Dockerfile, which installs necessary dependencies, and a k8s.cfg file, which connects the system to a Prefect account.

Agents are deployed via the gcloud command-line utility and its kubectl extension. Proper permissions within a project on Google Cloud are required.

Getting started

Before you can begin you will need to have several tools installed and configured.

If you don’t have a project on Google Cloud, you’ll need to start there. You will need its unique identifer before you can continue.

Also, you should install the gcloud command-line tool by following Google’s instructions. Then initialize the tool and connect it with your Google Cloud account. When asked, set your project’s id as the default.

gcloud init

Install the kubectl extensions that allows you to communicate with Google Kubernetes Engine.

gcloud components install kubectl

Install the Prefect Python library with the Kubernetes and Google extensions.

pipenv install "prefect[kubernetes,google]"

Install docker and docker-compose.

Creating a new cluster

Set the region where the cluster will be made. I’ll use the one closest to me. You can pick another from Google’s roster, if you’d like.

export COMPUTE_REGION=us-west2

Come up with a name for your cluster.

export CLUSTER=prefect-cluster

Create the cluster.

gcloud container clusters create-auto $CLUSTER \
  --region=$COMPUTE_REGION;

After the creation process completes, ask for credentials to access it.

gcloud container clusters get-credentials $CLUSTER \
  --region $COMPUTE_REGION;

Verify it exists.

kubectl get nodes;

Creating a new agent

Agents are created by booting and configuring an image within a cluster. The contents of the image are configured by a Dockerfile. Its connection with Prefect is configured by a separate file that is then applied to the machine.

The first step is to come up with a slug for your agent that we’ll use in naming conventions. For this example, I will use etl, which is a common acronym for the extract, transform and load processes that Prefect specializes in supporting.

Create a Dockerfile in the current directory. At a minimum, it should start by inheriting Prefect’s default image. Nothing else is necessary to get started. But if you agent will require serverside dependencies beyond the basics, this is where you include them.

FROM prefecthq/prefect:latest

The Dockerfile is then built for Google Artifact Registry. First thing to do is login there.

gcloud auth configure-docker us-west2-docker.pkg.dev

Before you can push the image, you should create a new Docker repository in the artifact registry with a name like prefect-agents.

gcloud artifacts repositories create prefect-agents \
    --repository-format=docker \
    --location=us-west2
    --description="Prefect agents"

You need to come up with another name for the image. I recommend adding the slug of your directory as a suffix to prefect-agents-. Be sure to put in your project id. The first release should be tagged as 0.0.1.

docker build -t us-west2-docker.pkg.dev/your-project-id-here/prefect-agents/prefect-agents-etl:0.0.1 .

Push the image.

docker push us-west2-docker.pkg.dev/your-project-id-here/prefect-agents/prefect-agents-etl:0.0.1

The image should also be aliased to the latest tag.

docker build -t us-west2-docker.pkg.dev/your-project-id-here/prefect-agents/prefect-agents-etl:latest .

Push that too.

docker push us-west2-docker.pkg.dev/your-project-id-here/prefect-agents/prefect-agents-etl:latest

You should now create a Kubernetes configuration file that will be applied to the image. It will need include a Prefect API key that allows it to speak to your cloud dashboard. If you don’t have a key already, go make one inside your Prefect account.

export API_KEY=your-api-key-here
pipenv run prefect agent kubernetes install --rbac -k $API_KEY > k8s.cfg

Open the configuration file. In the metadata section, give your agent a slugged name. I again recommend using your project slug as a suffix.

metadata:
  labels:
    app: prefect-agent
  name: prefect-agent-etl

Your agent will need labels. These are used to mark your agent as available for the Prefect workflows that carry the same tag. Again I recommend using your agent’s slug.

- name: PREFECT__CLOUD__AGENT__LABELS
  value: '["etl"]'

Your API key will need to be included.

- name: PREFECT__CLOUD__API_KEY
  value: your-api-key-here

Point the configuration file to the latest version of your Docker image in Google Artifact registry.

image: us-west2-docker.pkg.dev/your-project-id-here/prefect-agents/prefect-agents-etl:latest

Save those edits. Now ask kubectl to connect to our cluster where you would like to deploy this agent.

gcloud container clusters get-credentials prefect-cluster --zone $COMPUTE_REGION;

Ask for a new deployment in your Kubernetes cluster using the configuration file.

kubectl apply -f k8s.cfg

Once created you should be able to see it running in your cluster.

kubectl get deploy

Check the Agents page in your Prefect account. After a few minutes have gone by the new agent should appear.

Updating an agent

Deployment of updates is done with the following steps

Up the version number on the the Dockerfile and push it to Google Artifact Registry. Here I’m moving it up to version number 0.0.2. You want to iterate the image number up with each release.

docker build -t us-west2-docker.pkg.dev/your-project-id-here/prefect-agents/prefect-agents-etl:0.0.2 .

Connect to Google Artifact Registry.

gcloud auth configure-docker us-west2-docker.pkg.dev

Push the image.

docker push us-west2-docker.pkg.dev/your-project-id-here/prefect-agents/prefect-agents-etl:0.0.2

We also need to overwrite the latest tag with your update.

docker build -t us-west2-docker.pkg.dev/your-project-id-here/prefect-agents/prefect-agents-etl:latest .

Push it too.

docker push us-west2-docker.pkg.dev/your-project-id-here/prefect-agents/prefect-agents-etl:latest

Save any changes to the configuration file, then apply it.

kubectl apply -f k8s.cfg

Wait a few minutes and the agent should update.