Skip to content

fix(athena): handle unpartitioned models in create_table_as_with_partitions#1743

Open
dtaniwaki wants to merge 2 commits intodbt-labs:mainfrom
dtaniwaki:fix/athena-unpartitioned-too-many-open-partitions
Open

fix(athena): handle unpartitioned models in create_table_as_with_partitions#1743
dtaniwaki wants to merge 2 commits intodbt-labs:mainfrom
dtaniwaki:fix/athena-unpartitioned-too-many-open-partitions

Conversation

@dtaniwaki
Copy link
Contributor

resolves #1742
docs N/A

Thank you for maintaining this project! I'd appreciate your review on this bug fix for the Athena adapter.

Problem

When a model with no partitioned_by config reads from a highly-partitioned Hive source table, Athena may return TOO_MANY_OPEN_PARTITIONS due to the source table's partition count. safe_create_table_as falls back to create_table_as_with_partitions, which unconditionally calls get_partition_batches. This calls adapter.format_partition_keys(None) and crashes with TypeError: 'NoneType' object is not iterable.

Compilation Error in model my_model (models/my_model.sql)
  'NoneType' object is not iterable
  > in macro create_table_as_with_partitions (macros/materializations/models/table/create_table_as.sql)
  > called by macro safe_create_table_as (macros/materializations/models/table/create_table_as.sql)

This affects both temporary=False (full-refresh) and temporary=True (incremental merge's __dbt_tmp), the latter of which regressed after the temporary branch unification in #1711.

Solution

In create_table_as_with_partitions, check partitioned_by before calling get_partition_batches. If the model is unpartitioned, skip batch INSERT and create the target table directly from the staging table via simple CTAS. The staging table is created with skip_partitioning=True, so it has no partitions and the CTAS from it will not trigger TOO_MANY_OPEN_PARTITIONS.

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

…itions

Signed-off-by: Daisuke Taniwaki <daisuketaniwaki@gmail.com>
@cla-bot cla-bot bot added the cla:yes The PR author has signed the CLA label Mar 11, 2026
@dtaniwaki dtaniwaki marked this pull request as ready for review March 11, 2026 04:38
@dtaniwaki dtaniwaki requested a review from a team as a code owner March 11, 2026 04:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla:yes The PR author has signed the CLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] dbt-athena: TOO_MANY_OPEN_PARTITIONS crashes with TypeError for unpartitioned models

1 participant