-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Fix test failure table naming with unique suffixes #11939
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
- Added store_failures_unique and store_failures_suffix config options to TestConfig - Implemented suffix generation in compilation.py for test nodes - Supports multiple suffix strategies: invocation_id, timestamp, date, hour, or custom - Updates relation_name with suffix when store_failures_unique is enabled This ensures test failure tables have unique names in parallel execution environments
Added unit tests to verify suffix generation behavior for different strategies
Additional Artifact Review RequiredChanges to artifact directory files requires at least 2 approvals from core team members. |
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide. |
1 similar comment
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide. |
The suffix must be applied to the node.alias field, not just relation_name, because materializations use model['alias'] to create the table identifier. This ensures the suffix is properly used throughout the entire flow.
Addresses PR feedback about missing changelog entry
Added changelog entry in commit ad483f9. The entry has been added to following the project's changie format. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Summary
This PR addresses the issue where test failure tables in parallel execution environments (e.g., hourly/daily Airflow DAGs) overwrite each other due to non-unique table names. It adds support for unique suffixes on test failure tables to ensure proper isolation during parallel runs.
Problem
When running dbt tests with
store_failures: true
in parallel environments:Solution
This PR implements a complete solution in dbt-core that:
Adds new configuration options to
TestConfig
:store_failures_unique
: Boolean to enable unique table suffixesstore_failures_suffix
: Strategy for generating suffixesSupports multiple suffix strategies:
invocation_id
: Uses first 8 chars of invocation ID (default)timestamp
: Full timestamp (YYYYMMDD_HHMMSS)date
: Date only (YYYYMMDD)hour
: Date and hour (YYYYMMDD_HH)Updates compilation process:
node.alias
with suffix before relation creationKey Changes
core/dbt/artifacts/resources/v1/config.py
to add new config fieldscore/dbt/compilation.py
to generate and apply suffixes tonode.alias
Why This Solution Works
The key insight is that test materializations use
model['alias']
to create the table identifier, notmodel['relation_name']
. By modifyingnode.alias
during compilation (beforecreate_from()
is called), the suffix is automatically included in both:This keeps all suffix logic in dbt-core, providing a single source of truth that works for all adapters.
Testing
Related Work
Companion PR for dbt-adapters: feat: add optional unique suffixes to test failure table names dbt-adapters#1277UPDATE: No longer needed - solution is complete in dbt-coreBreaking Changes
None. The feature is opt-in and backward compatible.
Checklist