Filter which affected files are passed to task commands

[moltar]

When using options.affectedFiles, moon filters changed files against the task's inputs and passes matching files as arguments (or via the MOON_AFFECTED_FILES env var) to the command. The problem is that inputs serves a dual purpose: cache invalidation and affected file filtering. This means config files like .eslintrc.json or tsconfig.json that are listed in inputs for cache invalidation also get passed as command arguments — causing tools like ESLint to try to lint those config files.

For example, given this task:

tasks:
  lint:
    command: eslint
    inputs:
      - 'src/**/*.ts'
      - '.eslintrc.json'
    options:
      affectedFiles: args

If .eslintrc.json changes, it correctly invalidates the cache, but it also gets passed to ESLint as eslint .eslintrc.json, which is not the intended behavior.

It would be great to have a way to control which affected files actually get passed to the command, independent of cache invalidation inputs. Something like a glob filter on affectedFiles:

tasks:
  lint:
    command: eslint
    inputs:
      - 'src/**/*.ts'
      - '.eslintrc.json'
    options:
      affectedFiles:
        args: true
        filter: 'src/**/*.ts'

This way, inputs continues to define what invalidates the cache, while affectedFiles.filter controls which of those files actually get forwarded to the command. Only files matching the filter pattern would be passed as arguments.

[milesj]

Now that affectedFiles supports an object of params, this is much easier to implement. I can look into this for the next release.

[dudicoco]

I've been facing the same issue, tried to workaround it in different ways for a week, this is my final workaround:

  check-yaml:
    command: .config/moon/scripts/check.sh
    inputs:
    - .config/moon/scripts/check.sh
    - '@group(yaml)'
    - .yamlfmt.yaml
    args:
    - yamlfmt
    env:
      ALL_FILES: '@files(yaml)'
    options:
      affectedFiles:
        pass: env

#!/usr/bin/env bash
# Usage: check.sh <tool> [--arg=<arg>] ...
# Example: check.sh yamlfmt --arg=--lint
#
# Wrapper script that handles all-files/affected-files behavior.
#
# How it works:
# - Reads $MOON_PROJECT_SNAPSHOT to detect config files
# - In order for this functionality to work, the task inputs for files to check must be defined with the `@group` token
# - If any affected file matches a config file, runs in all-files mode
# - Reads files from ALL_FILES or MOON_AFFECTED_FILES depending on mode
# - Filters out deleted files
# - Executes tool in parallel batches

set -euo pipefail

tool=""
tool_args=()
declare -A config_file_map

while [[ $# -gt 0 ]]; do
  case "$1" in
  --arg=*)
    arg_value="${1#*=}"
    if [[ -n "$arg_value" ]]; then
      tool_args+=("$arg_value")
    fi
    shift
    ;;
  *)
    if [[ -z "$tool" ]]; then
      tool="$1"
    fi
    shift
    ;;
  esac
done

task_id="${MOON_TARGET#*:}"

while IFS= read -r input; do
  config_file_map["$input"]=1
done < <(jq -r --arg task "$task_id" '.tasks[$task].inputFiles | keys[]' "$MOON_PROJECT_SNAPSHOT" 2>/dev/null || true)

IFS=',' read -ra all_files <<<"$ALL_FILES"
IFS=':' read -ra affected_files <<<"$MOON_AFFECTED_FILES"

mode="affected"
existing_affected_files=()

for file in "${affected_files[@]}"; do
  if [[ -f "$file" ]]; then
    if [[ -n "${config_file_map[$file]:-}" ]]; then
      mode="all"
      echo "Detected affected config file: $file, switching to all-files mode"
      break
    fi
    existing_affected_files+=("$file")
  fi
done

if [[ "$mode" == "all" ]]; then
  files_to_process=("${all_files[@]}")
  echo "Running $tool ${tool_args[*]} on ${#files_to_process[@]} files in all-files mode"
else
  files_to_process=("${existing_affected_files[@]}")
  echo "Running $tool ${tool_args[*]} on ${#files_to_process[@]} files in affected-files mode"
fi

if [[ ${#files_to_process[@]} -eq 0 ]]; then
  echo "No files to process, exiting"
  exit 0
fi

printf "%s\n" "${files_to_process[@]}" | xargs --max-procs 0 --max-args 1000 "$tool" "${tool_args[@]}"

Note that I'm also running on all files in case a config file has changed - you have to do that in order to ensure all of the files in the repo still pass the checks.
I'm also passing the script as input since changes to it should also invalidate all files, and moon passes the task definitions + moon config files by default which would also invalidate all files.
It would be great if moon incorporated this out of the box.

[milesj]

Will support this here #2401

[dudicoco]

Thanks @milesj.
Will we support also passing all source files when config files change per my workaround above?

Maybe something like this:

tasks:
  lint:
    command: eslint
    inputs:
      - 'src/**/*.ts'
      - '.eslintrc.json'
    options:
      affectedFiles:
        args: true
        inputs:
        - 'src/**/*.ts' # will pass all of these inputs when .eslintrc changes since all inputs must be revalidated 

[milesj]

Probably more like:

tasks:
  lint:
    command: eslint
    inputs:
      - 'src/**/*.ts'
      - '.eslintrc.json'
    options:
      affectedFiles:
        pass: 'args'
        passInputsWhenNoMatch: true
        filters:
        - 'src/**/*.ts'

[dudicoco]

@milesj I believe passInputsWhenNoMatch is insufficient here.
For example, if I change a file within src/**/*.ts and the .eslintrc.json config file, I would like all of the filtered inputs to be passed for validation.

[milesj]

I don't know, that goes beyond what this setting is for, which is affected files specifically.

[dudicoco]

@milesj how would you suggest to accomplish this otherwise?
I've tried two tasks as one of my workarounds - one for affected-files and one for all-files, but that was very cumbersome + I had to find a way to skip the affected-files task if the all-files tasks inputs changed, which is messy.

[milesj]

If you're always just going to lint the same files, regardless of affected status, why not just pass that files list as args?

[dudicoco]

@milesj i'm not always going to lint the same files.
Let's say that I have 1000 ts files:

• If one of them changed I want to lint only that file
• If a config file changed (.eslintrc.json) I want to lint all 1000 files.

[milesj]

Hrmm, this option wasn't really meant to do that.

[dudicoco]

@milesj so what is your suggestion?
Do you see the value in running on all files if the config file changed? I think it's a very important validation - otherwise you don't know if you've broken the repo.
Without a native implementation within Moon we have to resort to hacks in order to accomplish this.

[milesj]

This option only exists to pass affected files to task, as either args or env vars, so that you can handle it however you like. What you are trying to do goes way beyond the scope of what this option is meant for, and definitely falls into "solve in user land" territory.