Skip to content

[Bug] fix(athena): create empty target table when no partition batches found #1748

@dtaniwaki

Description

@dtaniwaki

Is this a new bug?

  • I believe this is a new bug
  • I have searched the existing issues, and I could not find an existing issue for this bug

Which packages are affected?

  • dbt-adapters
  • dbt-tests-adapter
  • dbt-athena
  • dbt-athena-community
  • dbt-bigquery
  • dbt-postgres
  • dbt-redshift
  • dbt-snowflake
  • dbt-spark

Current Behavior

When create_table_as_with_partitions is invoked (e.g. after a TOO_MANY_OPEN_PARTITIONS fallback in safe_create_table_as) and the source query returns zero rows, get_partition_batches returns an empty list. The batch for loop is skipped entirely, so the target table is never created.

Any post-hook referencing the relation (e.g. OPTIMIZE or VACUUM) then fails:

TABLE_NOT_FOUND: Table ... does not exist

Expected Behavior

The target table should always be created after create_table_as_with_partitions completes, even when the source data is empty. An empty table is a valid and expected result for a zero-row run.

Steps To Reproduce

  1. Configure a partitioned Iceberg model with a post-hook:
    config:
      materialized: incremental
      incremental_strategy: insert_overwrite
      partitioned_by: [date_col]
      post_hook: "OPTIMIZE {{ this }} REWRITE DATA USING BIN_PACK"
  2. Run the model when the source query returns zero rows (e.g. a date-filtered incremental run on an empty window).
  3. The run falls back to create_table_as_with_partitions via safe_create_table_as.
  4. get_partition_batches returns [], the batch loop is skipped, and no target table is created.
  5. The post-hook runs and fails with TABLE_NOT_FOUND.

Relevant log output

Runtime Error
  TABLE_NOT_FOUND: Table <schema>.<model> does not exist
  > in post-hook: "OPTIMIZE <schema>.<model> REWRITE DATA USING BIN_PACK"

Environment

- OS: macOS
- Python: 3.12.8
- dbt-adapters: 1.22.8
- dbt-athena: 1.9.4

Additional Context

Root cause is in create_table_as_with_partitions (macros/materializations/models/table/create_table_as.sql):

{% set partitions_batches = get_partition_batches(sql=tmp_relation, as_subquery=False) %}

{%- for batch in partitions_batches -%}
    {%- if loop.index == 1 -%}
        {# CREATE TABLE AS SELECT ... WHERE batch — only runs if batches exist #}
    {%- else -%}
        {# INSERT INTO ... WHERE batch #}
    {%- endif -%}
{%- endfor -%}

{# If partitions_batches is empty, the loop never runs and no table is created #}

{%- do drop_relation(tmp_relation) -%}

Metadata

Metadata

Assignees

No one assigned

    Labels

    triage:productIn Product's queuetype:bugSomething isn't working as documented

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions