fix: preserve column names with spaces in wr.redshift.copy()#3298
Conversation
Passes flavor=None to internal s3.to_parquet call to prevent pyarrow spark flavor from sanitizing column names (spaces → underscores). Fixes aws#3293
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
@kukushking Could you confirm this failure is pre-existing and unrelated to the fix? Happy to address any other feedback! The |
@kukushking Could you confirm this failure is pre-existing and unrelated to the fix? Happy to address any other feedback! The GitHubCodeBuild (non-distributed) pipeline passed successfully. |
Problem
wr.redshift.copy()silently renames columns with spaces (e.g. "my col" → "my_col")because the internal
s3.to_parquetcall defaults to pyarrowflavor='spark',which sanitizes column names.
Fix
Explicitly pass
pyarrow_additional_kwargs={"flavor": None}in the internals3.to_parquetcall to preserve original column names.Fixes #3293