Handle Parquet Files With Inconsistent Timestamp Units

[anliakho2]

Describe the bug
If parquet is written with timestamps with time unit other than ns reading such file would produce incorrect dates, whereas pandas is reading the dates correctly

To Reproduce
Generate parquet file as follows:
`
import pandas as pd
import numpy as np

np.random.seed(0)

create an array of 5 dates starting at '2015-02-24', one per minute

rng = pd.date_range('2020-01-01', periods=5, freq='H')
df = pd.DataFrame({ 'Date': rng, 'Val': np.random.randn(len(rng)) })
df.to_parquet('data/myfile.parquet', coerce_timestamps='ms', allow_truncated_timestamps=True)
`

Expected behavior
Data is not corrupted and dates are read back correctly.

Additional context
_

[jmahenriques]

We experience the same issue (using datafusion), parquet metadata states microsecond but it is inferred as nanosecond, completely distorting the timestamps.

[tustvold]

So digging into this the issue is that pandas is attaching an arrow schema that specifies nanosecond precision, whilst specifying the following as the parquet column description.

converted_type: TIMESTAMP_MILLIS,
logical_type: Some(
    TIMESTAMP(
        TimestampType {
            is_adjusted_to_u_t_c: false,
            unit: MILLIS(
                MilliSeconds,
            ),
        },
    ),
),

This is pretty wild because not only are the two different schemas, but TIMESTAMP_MILLIS could overflow if converted to an arrow TimestampNanosecondArray which uses an i64 to store its values. I'm not really sure why it does this.

The issue doesn't occur with fastparquet which uses the LogicalType support for nanosecond precision i64 timestamps, but it also doesn't write an arrow schema so...

I think we can work around this, I need to work out exactly how, but imo this is a bug in pyarrow.

Edit: Adding flavor='spark' makes this work, likely because it stores the timestamps as Int96

Edit Edit: It would appear that pyarrow simply ignores the embedded schema - https://issues.apache.org/jira/browse/ARROW-2429

[tustvold]

I've filed https://issues.apache.org/jira/browse/ARROW-16184 as this feels like a bug in pyarrow, I will now work on adapting our schema inference logic to handle this particular case

[tustvold]

I've created #1558 which adds an option to skip decoding the garbled arrow metadata, this seemed like the safest option for allowing people to read these files, without this coming back to bite us in a strange way down the line.

I would recommend using flavor='spark', engine=fastparquet or version="2.4" when writing these files as a better workaround, but I appreciate rewriting files may not be an option

[jorisvandenbossche]

Note that you need version="2.6" (not "2.4) for nanosecond support (I would personally recommend this over the flavor="spark", because the int96 timestamps are deprecated in the Parquet spec)

[tustvold]

Following on from #1663 and in particular #1682 these files can be read without issue, as the embedded arrow schema is no longer treated as authoritative. This has been released in arrow 15