Skip to content

Conversation

andtrott
Copy link

@andtrott andtrott commented Aug 20, 2025

Summary

This PR addresses the issue where test failure tables in parallel execution environments (e.g., hourly/daily Airflow DAGs) overwrite each other due to non-unique table names. It adds support for unique suffixes on test failure tables to ensure proper isolation during parallel runs.

Problem

When running dbt tests with store_failures: true in parallel environments:

  • Multiple DAG runs can overwrite the same test failure table
  • Debugging becomes difficult as failure data gets replaced
  • Log messages show table names without suffixes, causing confusion

Solution

This PR implements a complete solution in dbt-core that:

  1. Adds new configuration options to TestConfig:

    • store_failures_unique: Boolean to enable unique table suffixes
    • store_failures_suffix: Strategy for generating suffixes
  2. Supports multiple suffix strategies:

    • invocation_id: Uses first 8 chars of invocation ID (default)
    • timestamp: Full timestamp (YYYYMMDD_HHMMSS)
    • date: Date only (YYYYMMDD)
    • hour: Date and hour (YYYYMMDD_HH)
    • Custom string literals
  3. Updates compilation process:

    • Generates suffix during test compilation
    • Modifies node.alias with suffix before relation creation
    • Ensures logging shows the actual table name with suffix
    • Works for all adapters without adapter-specific changes

Key Changes

  • Modified core/dbt/artifacts/resources/v1/config.py to add new config fields
  • Updated core/dbt/compilation.py to generate and apply suffixes to node.alias
  • Added unit tests for suffix generation logic
  • Added changelog entry

Why This Solution Works

The key insight is that test materializations use model['alias'] to create the table identifier, not model['relation_name']. By modifying node.alias during compilation (before create_from() is called), the suffix is automatically included in both:

  • The table identifier used by the materialization
  • The relation_name used for logging

This keeps all suffix logic in dbt-core, providing a single source of truth that works for all adapters.

Testing

  • Unit tests added for various suffix strategies
  • Verified suffix generation logic works correctly
  • Ensures backward compatibility (no suffix when disabled)

Related Work

Breaking Changes

None. The feature is opt-in and backward compatible.

Checklist

- Added store_failures_unique and store_failures_suffix config options to TestConfig
- Implemented suffix generation in compilation.py for test nodes
- Supports multiple suffix strategies: invocation_id, timestamp, date, hour, or custom
- Updates relation_name with suffix when store_failures_unique is enabled

This ensures test failure tables have unique names in parallel execution environments
Added unit tests to verify suffix generation behavior for different strategies
@andtrott andtrott requested a review from a team as a code owner August 20, 2025 00:24
@cla-bot cla-bot bot added the cla:yes label Aug 20, 2025
Copy link
Contributor

Additional Artifact Review Required

Changes to artifact directory files requires at least 2 approvals from core team members.

Copy link
Contributor

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

1 similar comment
Copy link
Contributor

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

The suffix must be applied to the node.alias field, not just relation_name,
because materializations use model['alias'] to create the table identifier.
This ensures the suffix is properly used throughout the entire flow.
Addresses PR feedback about missing changelog entry
@andtrott
Copy link
Author

Added changelog entry in commit ad483f9. The entry has been added to following the project's changie format.

Copy link

@arnab-sarkar-ant arnab-sarkar-ant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla:yes community This PR is from a community member
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Add optional unique suffixes to test failure table names for parallel execution support
2 participants