fix(athena): exclude ICEBERG_FILESYSTEM_ERROR from outer retry#1740
Open
dtaniwaki wants to merge 3 commits intodbt-labs:mainfrom
Open
fix(athena): exclude ICEBERG_FILESYSTEM_ERROR from outer retry#1740dtaniwaki wants to merge 3 commits intodbt-labs:mainfrom
dtaniwaki wants to merge 3 commits intodbt-labs:mainfrom
Conversation
Signed-off-by: Daisuke Taniwaki <daisuketaniwaki@gmail.com>
1f5c493 to
9b54cd0
Compare
Contributor
|
I believe this will be indirectly fixed by #1637 (but the change would still be good to apply to the legacy mode introduced by that PR). In the new retry logic, only ICEBERG_COMMIT_ERROR is retried, all other Iceberg errors are assumed to be non-retryable. Iceberg errors are also not retried by the throttling retry logic anymore. |
Signed-off-by: Daisuke Taniwaki <daisuketaniwaki@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
resolves #1739
docs N/A
Thank you for maintaining this project! I'd appreciate your review on this small bug fix for the Athena adapter.
Problem
The outer retry decorator in
AthenaCursor.execute()does not excludeICEBERG_FILESYSTEM_ERRORfrom retries. When a transient error causes a retry of an Iceberg CTAS query, partial data is left at the same hardcoded S3 UUID location (Iceberg CTAS is non-atomic), anddelete_from_s3is only invoked during Jinja evaluation (first attempt only). All subsequent retries therefore hitICEBERG_FILESYSTEM_ERROR: Cannot create a table on a non-empty location— and since this error was not excluded from the outer retry, it gets retried up tonum_retriestimes against the same S3 path, always failing.The existing comment already explains why
TOO_MANY_OPEN_PARTITIONSis excluded for the same reason, butICEBERG_FILESYSTEM_ERRORwas never added to the exclusion list.Note: this is distinct from the
TOO_MANY_OPEN_PARTITIONS→ batch fallback path handled byrun_query_with_partitions_limit_catching. That path already works correctly. This issue surfaces when a different transient error triggers the outer retry on an Iceberg CTAS.Solution
Add
ICEBERG_FILESYSTEM_ERRORto the no-retry condition alongsideTOO_MANY_OPEN_PARTITIONS. Since retrying the same SQL with the same S3 UUID location will always fail, failing fast avoids unnecessary retries and surfaces the original error more clearly.Checklist