Allow environment variables and jinja variables to be set across the whole prefect.yml file

[walterdolce]

Describe the current behavior

Hi folks.

Currently it appears as though environment variables can only be defined under job_variables. Additionally, it is possible to define reusable blocks of YAML to be used inside the prefect.yaml as definitions (as documented here), but that's only possible for certain configuration areas (job_variables, work_pools, etc)

Describe the proposed behavior

I believe that giving users the ability to define both environment variables and Jinja variables will enable users to have truly dynamic prefect configuration files.

I'd love to be able to do the following:

---
# Generic metadata about this project
name: foobar
prefect-version: 3.4.7

###
# The custom bit
###
definitions:
   PROJECT_NAME: foobar
   GIT_BRANCH: "{{ env['GIT_BRANCH'] | default('main') }}"   # If I define $GIT_BRANCH as env var, use it. Otherwise, default to main
  GIT_REPO: "{{ env['GIT_REPO'] | default('/path/to/git-repo') }}"
  BASE_DIRECTORY: "{{ env['BASE_DIRECTORY'] | default('/foo/bar') }}"  # Same as above
  REPO_DIR: "{{ BASE_DIRECTORY + '/' + PROJECT_NAME }} "


# pull section allows you to provide instructions for cloning this project in remote locations
pull:
  - prefect.deployments.steps.set_working_directory:
      directory: "{{ BASE_DIRECTORY }}"
  - prefect.deployments.steps.run_shell_script:
      script: powershell.exe -Command "if (Test-Path -Path {{ GIT_REPO }} -PathType Container) { Set-Location {{ REPO_DIR }}; git switch -f {{ GIT_BRANCH }}; git pull } else { git clone --branch {{ GIT_BRANCH }} {{ GIT_REPO }} {{ PROJECT_NAME }} }"
  - prefect.deployments.steps.set_working_directory:
      directory: "{{ REPO_DIR }}"
  - prefect.deployments.steps.run_shell_script:
      script: uv sync

As you can see above, I'm not even using deployments definitions but the concept is the same. Basically:

• Add the ability for users to define env vars at source (i.e. FOO=bla prefect deploy --all), and have those env vars be read and accessible within the parsed YAML (everywhere across the YAML model).
• Add the ability for users to use Jinja-style capabilities. So every piece of configuration could potentially be reusable.

Thanks

Example Use

No response

Additional context

No response

[desertaxle]

Hi @walterdolce, thanks for the detailed write-up! I think most of what you're looking for is actually already supported today — it might just be a matter of adjusting the syntax. Let me walk through it.

Environment variables across the whole YAML file

Prefect already supports referencing environment variables anywhere in prefect.yaml using the {{ $ENV_VAR }} syntax (note the $ prefix). This works in pull:, build:, push:, deployments:, and any other section. So your example of wanting to pass GIT_BRANCH=main prefect deploy --all already works — you'd write it as:

pull:
  - prefect.deployments.steps.set_working_directory:
      directory: "{{ $BASE_DIRECTORY }}"
  - prefect.deployments.steps.run_shell_script:
      script: >
        powershell.exe -Command "if (Test-Path -Path {{ $GIT_REPO }} -PathType Container)
        { Set-Location {{ $REPO_DIR }}; git switch -f {{ $GIT_BRANCH }}; git pull }
        else { git clone --branch {{ $GIT_BRANCH }} {{ $GIT_REPO }} {{ $PROJECT_NAME }} }"
  - prefect.deployments.steps.set_working_directory:
      directory: "{{ $REPO_DIR }}"
  - prefect.deployments.steps.run_shell_script:
      script: uv sync

Then invoke with:
GIT_BRANCH=main BASE_DIRECTORY=/foo/bar GIT_REPO=/path/to/git-repo PROJECT_NAME=foobar prefect deploy --all

See the templating options docs for more details.

Reusing values across the file

For reusing configuration values (your definitions concept), this is already supported via standard YAML anchors and aliases. Since prefect.yaml is parsed with a standard YAML parser, &anchor / *alias syntax works natively. We recommend putting these in a definitions: section for organization (Prefect ignores unknown top-level keys), but you can put them anywhere.

Here's a full example close to what you described:

definitions:
  directories:
    base: &base_directory /foo/bar
    project: &project_name foobar

pull:
  - prefect.deployments.steps.set_working_directory:
      directory: *base_directory
  - prefect.deployments.steps.run_shell_script:
      script: uv sync

deployments:
  - name: my-deployment
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: my-pool

See the reuse configuration docs for a more detailed example showing reuse of work pools, schedules, and build actions.

What's isn't supported

The one thing from your proposal that isn't possible today is Jinja2 expression syntax like {{ env['GIT_BRANCH'] | default('main') }} or string concatenation like {{ BASE_DIRECTORY + '/' + PROJECT_NAME }}.

Prefect uses a lightweight placeholder system, not a full Jinja2 engine, so filters (| default(...)) and expressions aren't available.

We could explore adopting Jinja2 as our templater instead, but we'd want to do our best to maintain backward compatibility.

[walterdolce]

Hey @desertaxle. Thanks for the detailed answer! This is definitely helpful.

Yes I did try to use $FOOBAR directly but it didn't work (probably because I was mixing it with ... | default(foo).

I think this guidance alone is a nice step forward and I will try to test this asap. I would still love to see the capability proposed as I think that would be a massive enabler (i.e. var of vars being the most useful use case).

Thanks