[Python] Add rename_columns to DataSet

[aersam]

Describe the enhancement requested

Dataset has fewer methods than Table, which is fine, of course. However we often use rename_columns on Table and it would be really handy for us to have it on Dataset, too. I think it could easily be implemented using replace_schema

https://arrow.apache.org/docs/python/generated/pyarrow.dataset.Dataset.html#pyarrow.dataset.Dataset

Component(s)

Python

[aersam]

Seems using replace_schema does not work. The dataset always uses those column names to query the parquet, meaning the column names must match the ones in physical files. What really is needed is a separation between physical column name and logical column name. This would be really great, especially since parquet is a bit limited in what column names are allowed.
The best would be to have a "column mapping" in the fragment which would map the schema column names to physical column names. This would allow making queries with parquets with different physical column for the same logical column name. I guess that's a bit complex regarding the filters... but still would be great.

If we'd want to abstract Apache Iceberg oder Delta Lake Tables with the dataset, this would be needed (both support such column mapping stuff)

[davlee1972]

Shouldn't dataset() just take the same parameters as to_table()? The difference is that to_table() produces a materialized view while dataset() creates a logical view.. This would include columns whether they are a subset or computed/renamed from the original dataset..

I'm not sure why there is a dataset.scanner class.. It looks logical, but you can't perform a join with it against another dataset.

[ion-elgreco]

@wjones127 do you think this is something that can be added?