fix(athena): partition tmp table in incremental to reduce batch scan cost

[dtaniwaki]

resolves #1744
docs dbt-labs/docs.getdbt.com/#

Thank you for maintaining this project! I'd appreciate your review on this fix to the Athena adapter's incremental materialization.

Problem

In safe_create_table_as, the temporary=True branch always created the tmp table (__dbt_tmp) without partitioning (skip_partitioning=True), bypassing TOO_MANY_OPEN_PARTITIONS handling entirely.

This caused an O(N) scan cost for batch incremental inserts: since the tmp table had no partitions, every batch in batch_incremental_insert performed a full scan of __dbt_tmp. With N batches, the total cost was (N+1) × full scan.

Solution

Unified the temporary=True and temporary=False code paths in safe_create_table_as so that tmp tables now go through run_query_with_partitions_limit_catching and fall back to create_table_as_with_partitions on TOO_MANY_OPEN_PARTITIONS.

With this change, __dbt_tmp is created with partitions (when partitioned_by is configured), so each batch benefits from partition pruning. Total scan cost drops from (N+1) × full scan to roughly 2 × full scan.

Models without partitioned_by are unaffected — they produce an unpartitioned CTAS as before.

Checklist

• I have read the contributing guide and understand what's expected of me
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

[nicor88]

@dtaniwaki, what you proposed here is against the original implementation decision - when we create a temporary table, we always write unpartitioned data, to then finally write to the final target location in a partitioned table.
We are aware that the first unpartititioned write lead to a full scan - this is because as you wrote, we want to bypass the TOO_MANY_OPEN_PARTITIONS entirely - therefore, I discurage to proceed with your approach as it can lead to other hidden issues.

[dtaniwaki]

@nicor88 I see. Then, how can we avoid massive full scan queries against huge data with massive number of partitions? I mistakenly created a model of this situation and waisted lots of money...

[nicor88]

@dtaniwaki how about introducing the approach that you suggested, but being able to control it via a "config" flag? The flag can be false by default, and properly documented, but then in your case you can set to "true".

doing so, we avoid any regretion, and you and other users are covered for those edge cases - I believe that that's the best compromise. Think about a good descriptive name for such configuration flag.