Dataset.data property with ArrowTable (aka. MemoryMappedTable) not updated after filter?

[alvarobartt]

Hi to however is reading this! 🤗

I just wanted to know whether there's any reason why the _data attribute under the data property of a datasets.Dataset is not being updated after filter?

I've seen that the indices are updated and the magic methods implemented within datasets.arrow_dataset.Dataset don't directly use the data property, so there's no issue when using those, but if we do something like dataset.filter(...) to apply a filter to go from 10 to 5 samples/rows, then if we do len(dataset) the output will be 5, but when doing len(dataset.data) we get 10, as the data is not updated.

I understand this is not a big issue in most of the scenarios, but at the end what I want is to be able to filter or map and then keep the ArrowTable either to manipulate it or to do something like Dataset(arrow_table=dataset.data), as I found that more efficient than e.g. Dataset.from_dict(dataset.to_dict()).

So my other question would be, is there any efficient way to generate a new dataset after a filter without having to serialize to Python dict and then read from Python dict?

Thanks in advance! cc @lhoestq @mariosasko

[lhoestq]

Hi !

I just wanted to know whether there's any reason why the _data attribute under the data property of a datasets.Dataset is not being updated after filter?

Because we only need to update the indices to keep and recreating a new pyarrow Table can take time and disk space.

So my other question would be, is there any efficient way to generate a new dataset after a filter without having to serialize to Python dict and then read from Python dict?

You can remove the indices on top of the arrow table using ds = ds.flatten_indices()