Skip to content

Latest commit

 

History

History
253 lines (196 loc) · 9.42 KB

File metadata and controls

253 lines (196 loc) · 9.42 KB

Task Type Definitions

Task types describe how to run a unit of work — what command to execute, what environment variables it needs, what timeout to enforce, and which workers are eligible to claim it. They live as TOML files under any directory listed in the tasks.typesPaths config option. The filename stem is the type name: echo_task.tomlecho_task.

The only hard requirements are:

  • the filename must end in .toml
  • the file must be in one of the locations listed in the tasks.typesPaths variable in the server config
  • the command field is present (it is currently the only required field)

The command is rendered through Go's text/template so it can substitute environment variables at submit time.

Field names

description

A one-line summary of what the task type does. Shown on the task types list and on the new-task form.

documentation

Multi-line free text for setup, prerequisites, and gotchas — the things a description is too short to hold. Rendered on the task type's detail page (/ui/task-types/:name). Optional; use a TOML triple-quoted string for multi-line content.

description = "Run a dbt build against the warehouse"

documentation = '''
Requires `dbt` on $PATH and a populated ~/.dbt/profiles.yml on the
worker host. Writes artifacts to the task result dir.
'''

tags

A list of strings. Defines the capabilities required of any worker that wants to execute this task. A worker only claims a task whose tags it satisfies — a tag is a constraint, not a label. See tag_ontology.md for the namespaced namespace:value convention (os:linux, exec:bash, team:data-eng, ...) and how to extend it.

timeout

Max duration of the task in seconds. Default is 3600 (one hour). Tasks that exceed this are killed and marked TIMEDOUT.

command

The command to execute when the task runs. Supports Go template substitution against the task's environment map (e.g. {{.NAME}} is replaced with the value of the NAME env var at submit time). Environment variables are also available at exec time as $NAME — the difference matters when the value is set by the caller versus inherited from the shell.

executor

The shell or interpreter that runs command. Supported values:

executor how it runs typical platforms
bash (default) bash -c <command> Linux, macOS, WSL
cmd cmd /c <command> Windows
powershell powershell -Command <command> Windows, macOS, Linux
any other binary <executor> -c <command> depends on the binary

The executor binary must be on the worker's $PATH. Run blanket task-validate to check that all configured task types have their executor available on the current host.

result_file

The name of a file the task writes, whose contents callers get back as structured data. Optional; a type that declares none simply reports result: null.

name = "lookup_user"
command = "./lookup.sh {{.USER_ID}} > result.json"
result_file = "result.json"

The path is relative to the task's result directory — the same directory the task runs in, so a task that writes result.json in its working directory needs no path at all. It is read once the task reaches a terminal state, parsed as JSON, and returned as the result field of a synchronous submission's completion payload.

  • A missing file yields result: null and is not an error — a task that failed before writing its result is a normal outcome.
  • A file that exists but is unparseable, unreadable, or larger than tasks.sync.maxResultBytes (default 1 MiB) yields result: null plus a resultError message, so a malformed result never looks like an absent one.
  • The path must stay inside the result directory. Absolute paths (/etc/passwd, C:\...), UNC paths, and anything that escapes via .. are rejected when the task type is loaded — a type declaring one doesn't load at all, and blanket task-validate reports it as check 009. The same rule is applied again when the file is read.
  • Subdirectories are fine: result_file = "out/result.json".

environment

A map of environment variables with three sections: default, required, and optional.

  • default: present by default, can be overridden by the caller
  • required: must be sent when a new task instance is created
  • optional: may be set but is not required; no default value (primarily for documentation and discoverability)

Each entry takes a name and description. default entries also take a value. When submitting a task, you can always add additional env variables that are not part of the type definition.

Environment variables are the main unit of configurability for tasks, so this is where most of the complexity ends up. As a rule of thumb, 2-5 inputs is comfortable to use; more than 10 is a sign the type should probably be split (see check 008 below).

Validation

blanket task-validate [type-name] checks every configured task type against a set of coded rules and prints a per-type status plus the individual findings. Codes are stable once assigned.

Code Check Level
001 command is present and non-empty error
002 executor resolves on $PATH error
003 command parses as a Go template error
004 every {{.VAR}} reference is a declared input warn
005 a required input is never referenced by command warn
006 description is present and non-empty warn
007 documentation is present and non-empty warn
008 declared input count is in the healthy range (2-5) warn
009 result_file is a relative path contained in the result dir error

004 is deliberately a warning, not an error — a {{.VAR}} reference can legitimately resolve to a variable inherited from the worker's own environment rather than one declared in this type's environment table.

009 is an error because a type that trips it is not servable at all: the loader rejects the file rather than serving a type whose declared result path points outside the task's own directory.

Tag lint (codes 010-014)

These check tags against the resolved vocabulary from tag_ontology.md. Extension stays frictionless by default — 010/011 nudge toward the convention, 012-014 are opt-in for deployments that want stricter enforcement.

Code Check Default
010 tag is a near-miss (edit distance ≤2) of a known tag warn
011 unnamespaced tag has a namespaced value-match (bashexec:bash) warn
012 tag is new — not declared anywhere, not used by any other type off (--warn-new-tag)
013 tag isn't declared in the known-tags vocabulary, even if used elsewhere off (--warn-undeclared-tag)
014 no registered worker advertises a superset of this type's tags off (--check-workers)

Introducing a well-formed, novel namespaced tag (e.g. team:platform for the first time) never triggers 010 or 011 — those only fire on a tag that looks like a typo of something that already exists. 012 and 013 have matching tasks.warnNewTag / tasks.warnUndeclaredTag config keys, so a deployment can make either the default without passing the flag every time. 014 needs a running server (GET /worker/) to know what's registered; if it can't be reached, --check-workers degrades to a single "skipped" finding instead of failing the whole run.

Flags:

  • --json — print findings as a JSON array ({type, code, level, message, suggestion}) instead of the table. This is what an authoring tool should drive against.
  • --strict — exit non-zero on warnings too, not just errors. Default behavior exits non-zero only on errors, so it stays usable as a pre-flight check without failing on style nits.
  • --dump-known-tags — print the resolved tag vocabulary instead of validating. See tag_ontology.md.
  • --no-builtin-tags — exclude the built-in seed vocabulary when resolving known tags.
  • --warn-new-tag — enable code 012.
  • --warn-undeclared-tag — enable code 013.
  • --check-workers — enable code 014.

Examples

See examples/types/ for the full set of copy-paste-ready starters: echo_task (minimal), bash_task (arbitrary command via env var), python_hello, windows_echo (uses cmd, no bash needed), and windows_powershell (same, via powershell).

A simple bash task that runs a user-supplied command

tags = ["exec:bash", "os:unix"]

# timeout in seconds
timeout = 200

# The command to execute
command='''
{{.DEFAULT_COMMAND}}
'''

executor="bash"

    # Environment variables are injected into the process environment

    [[environment.default]]
    name = "ANIMAL"
    value = "giraffe"

    [[environment.default]]
    name = "SECOND_ANIMAL"
    value = "hippo"

    # Remember, everything is interpreted as a string when passed as an env variable
    [[environment.default]]
    name = "NUM_FROGS"
    value = "3"

    [[environment.required]]
    name = "DEFAULT_COMMAND"
    description = "The bash command to run. E.g. `echo $(date)`"

A Windows-native task using cmd

tags = ["os:windows"]
executor = "cmd"
command = "echo hello from blanket"
timeout = 10

No bash or WSL required — runs anywhere cmd.exe is available.