Skip to content

Fix compatibility with Pandas 2#63

Open
QuLogic wants to merge 1 commit intohpi-dhc:mainfrom
QuLogic:pandas2
Open

Fix compatibility with Pandas 2#63
QuLogic wants to merge 1 commit intohpi-dhc:mainfrom
QuLogic:pandas2

Conversation

@QuLogic
Copy link
Copy Markdown

@QuLogic QuLogic commented Feb 24, 2024

The line_terminator parameter of read_csv was changed to lineterminator in Pandas 1.5, and the old name was dropped entirely in Pandas 2.

This adds compatibility shims across both, though if you prefer to bump minimum Pandas version to 1.5, then the new name could be used everywhere instead.

The `line_terminator` parameter of `read_csv` was changed to
`lineterminator` in Pandas 1.5, and the old name was dropped entirely in
Pandas 2.
Comment thread devicely/shimmer_plus.py
f.write(f'"sep={self.delimiter}"\n')
write_df.to_csv(f, index=False, sep=self.delimiter, line_terminator=f"{self.delimiter}\n")
write_df.to_csv(f, index=False, sep=self.delimiter,
**{_to_csv_line_terminator: f"{self.delimiter}\n"})
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. This raises a ValueError with Python 3.13.

________________________ ShimmerPlusTestCase.test_write ________________________
self = <test_shimmer.ShimmerPlusTestCase testMethod=test_write>
    def test_write(self):
>       self.reader.write(self.WRITE_PATH)
tests/test_shimmer.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../BUILDROOT/usr/lib/python3.13/site-packages/devicely/shimmer_plus.py:74: in write
    write_df.to_csv(f, index=False, sep=self.delimiter,
/usr/lib64/python3.13/site-packages/pandas/util/_decorators.py:333: in wrapper
    return func(*args, **kwargs)
/usr/lib64/python3.13/site-packages/pandas/core/generic.py:3964: in to_csv
    return DataFrameRenderer(formatter).to_csv(
/usr/lib64/python3.13/site-packages/pandas/io/formats/format.py:1014: in to_csv
    csv_formatter.save()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <pandas.io.formats.csvs.CSVFormatter object at 0x7f2269a70a50>
    def save(self) -> None:
        """
        Create the writer & save.
        """
        # apply compression and byte/text conversion
        with get_handle(
            self.filepath_or_buffer,
            self.mode,
            encoding=self.encoding,
            errors=self.errors,
            compression=self.compression,
            storage_options=self.storage_options,
        ) as handles:
            # Note: self.encoding is irrelevant here
>           self.writer = csvlib.writer(
                handles.handle,
                lineterminator=self.lineterminator,
                delimiter=self.sep,
                quoting=self.quoting,
                doublequote=self.doublequote,
                escapechar=self.escapechar,
                quotechar=self.quotechar,
            )
E           ValueError: bad delimiter or lineterminator value
/usr/lib64/python3.13/site-packages/pandas/io/formats/csvs.py:260: ValueError

So far, I haven't got a clue as to why?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems unrelated to this change. The existing call (slightly) abuses lineterminator to always insert a trailing separator. This must have been made more strict in Python 3.13, and I see a few changelog entries that could be related, most likely "gh-113796: Add more validation checks in the csv.Dialect constructor. ValueError is now raised if the same character is used in different roles."

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants