[Bug] Handle 409 Conflict as success during BigQuery job creation retries

[winiar93]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Description

When dbt retries a BigQuery job creation (e.g. due to a timeout or transient network spike), BigQuery may return a 409 Conflict response, indicating the job was already registered. dbt currently treats this as a fatal error and fails, instead of recognizing that the job was successfully submitted and continuing normally.

The root cause is not the timeout itself — subsequent runs complete without issues, suggesting this is an edge case where the job confirmation did not return to dbt in time. The real problem is that dbt's retry logic does not handle 409 gracefully.

Actual Behavior

dbt raises an error on 409 Conflict, causing the run (and any orchestration flow, e.g. Prefect) to fail.

Expected Behavior

When dbt retries a job and receives 409 Conflict from BigQuery, it should recognize that the job already exists and treat it as a successful submission — then continue polling for the result as normal.

Steps To Reproduce

1. Run a dbt model against BigQuery
2. Simulate a transient delay (network spike, VM slowdown) that causes the job confirmation to not return within job_creation_timeout_seconds
3. dbt retries with the same job ID
4. BigQuery returns 409 Conflict
5. dbt fails instead of continuing with the existing job

Relevant log output

ValueError: Failures detected during invocation of dbt command 'test--selectpath:models/xxx/xxx':
Test relationships_context_arrival_map_site_id__site_id__ref_context_all_sites_ errored with message: "Database Error in test relationships_xxxxx_ (models/xxx/xxx/xxx.yaml)
  409 POST https://bigquery.googleapis.com/xxxx: Already Exists: Job xxx-xxx:US.xxxx-xxxx-xxxx-xxxx-xxxx

Environment

- OS: Debian 12
- Python: 3.12
- dbt: 1.10.3

Which database adapter are you using with dbt?

bigquery

Additional Context

No response

[dbeatty10]

Thanks for reaching out @winiar93 !

Which version of dbt-bigquery are you using?

[dbeatty10]

Potential fix

Update this:

dbt-adapters/dbt-bigquery/src/dbt/adapters/bigquery/connections.py

Lines 614 to 620 in bbbb5c0

	query_job = client.query(
	query=sql,
	job_config=query_job_config,
	job_id=job_id,
	job_retry=None,
	timeout=self._retry.create_job_creation_timeout(),
	)

to be this instead:

from google.api_core.exceptions import Conflict

        # ...

        try:
            query_job = client.query(
                query=sql,
                job_config=query_job_config,
                job_id=job_id,
                job_retry=None,
                timeout=self._retry.create_job_creation_timeout(),
            )
        except Conflict:
            logger.info("Caught 409 Conflict for job_id=%s, recovering via get_job()", job_id)
            query_job = client.get_job(job_id)
            logger.info("Job recovered, state: %s", query_job.state)

[dbeatty10]

Transferred from https://github.com/dbt-labs/dbt-core/issues/12636 to #1746.

[winiar93]

Thank you for your response.

Just wanted to flag that the workaround you suggested ( wrapping code into try/except ) is unfortunately not something we can use in our setup.

We are using dlt which uses dbt-bigquery under the hood. So from our side, we just call pipeline.run() and have no way to get in between dbt's interal client.query() call to add that error handling.

The 409 http error just crashes the whole pipeline, and there's really nothing we can do about it at our level.
From our perspective, this really needs to be fixed inside dbt-bigquery itself, specially when a library does its own retries and generates its own job IDs, it should really deal with what comes back from external resource - not leave it up to user to figure out.