fix(athena): handle unpartitioned models in create_table_as_with_partitions#1743
Open
dtaniwaki wants to merge 2 commits intodbt-labs:mainfrom
Open
fix(athena): handle unpartitioned models in create_table_as_with_partitions#1743dtaniwaki wants to merge 2 commits intodbt-labs:mainfrom
dtaniwaki wants to merge 2 commits intodbt-labs:mainfrom
Conversation
…itions Signed-off-by: Daisuke Taniwaki <daisuketaniwaki@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
resolves #1742
docs N/A
Thank you for maintaining this project! I'd appreciate your review on this bug fix for the Athena adapter.
Problem
When a model with no
partitioned_byconfig reads from a highly-partitioned Hive source table, Athena may returnTOO_MANY_OPEN_PARTITIONSdue to the source table's partition count.safe_create_table_asfalls back tocreate_table_as_with_partitions, which unconditionally callsget_partition_batches. This callsadapter.format_partition_keys(None)and crashes withTypeError: 'NoneType' object is not iterable.This affects both
temporary=False(full-refresh) andtemporary=True(incremental merge's__dbt_tmp), the latter of which regressed after thetemporarybranch unification in #1711.Solution
In
create_table_as_with_partitions, checkpartitioned_bybefore callingget_partition_batches. If the model is unpartitioned, skip batch INSERT and create the target table directly from the staging table via simple CTAS. The staging table is created withskip_partitioning=True, so it has no partitions and the CTAS from it will not triggerTOO_MANY_OPEN_PARTITIONS.Checklist