Support writing object columns with np.nan values

[theroggy]

Closes #60

[jorisvandenbossche]

Thanks for working on this!

[jorisvandenbossche]

Can we combine this with the field_value is None check above to set FieldNull in a single place?
(or move that check here, as currently the only way to get a None is through object dtype)

[brendan-ward]

It seems like the check for isnan should be in the elif field_type == OFTReal: block? The field type for the incoming column should be float32 / float64 rather than object (though I haven't verified this)

[jorisvandenbossche]

@brendan-ward note that the specific case that is being fixed here is if you have an object dtype column (but which contains NaN, pandas doesn't really distinguish None vs NaN in object columns)

[theroggy]

Sure... if you prefer that... I moved the "is None" check as it is redundant for column types other than string/object.

[theroggy]

@brendan-ward in addition to what @jorisvandenbossche wrote: np.nan values in an OFTReal (float) column are automatically treated correctly (as null) by GDAL (OGR_F_SetFieldDouble), so no need to explicitly call OGR_F_SetFieldNull for OFTReal columns. Only for object columns this is needed.

[jorisvandenbossche]

np.nan values in an OFTReal (float) column are automatically treated correctly (as null) by GDAL (OGR_F_SetFieldDouble), so no need to explicitly call OGR_F_SetFieldNull for OFTReal columns. Only for object columns this is needed.

Actually, that's not fully correct I think. They are written as NaN and not as Null. Since numpy/pandas only support NaN in float arrays, that's not an issue for correct rountrip for geopandas->gdal->geopandas, though.

Small illustrations:

# write file with GDAL + pyogrio
import geopandas
import pyogrio
gdf = geopandas.GeoDataFrame({'col': [0.1, np.nan]}, geometry=geopandas.points_from_xy([0, 1], [0, 1]))
pyogrio.write_dataframe(gdf, "test_nulls_pyogrio.arrow", driver="Arrow")

# write file with pyarrow that includes both NaN and Null (Arrow distinguishes both)
import pyarrow as pa
from pyarrow import feather
feather.write_feather(pa.table({"col": [0.1, np.nan, None]}), "test_nulls_pyarrow.arrow")

And check both using GDAL's ogrinfo:

(gdal-dev) $ ogrinfo test_nulls_pyogrio.arrow -al
INFO: Open of `test_nulls_pyogrio.arrow'
      using driver `Arrow' successful.

Layer name: test_nulls_pyogrio
Geometry: Point
Feature Count: 2
Extent: (0.000000, 0.000000) - (1.000000, 1.000000)
Layer SRS WKT:
(unknown)
Geometry Column = geometry
col: Real (0.0)
OGRFeature(test_nulls_pyogrio):0
  col (Real) = 0.1
  POINT (0 0)

OGRFeature(test_nulls_pyogrio):1
  col (Real) = nan
  POINT (1 1)

(gdal-dev) $ ogrinfo test_nulls_pyarrow.arrow -al
INFO: Open of `test_nulls_pyarrow.arrow'
      using driver `Arrow' successful.

Layer name: test_nulls_pyarrow
Geometry: None
Feature Count: 3
Layer SRS WKT:
(unknown)
col: Real (0.0)
OGRFeature(test_nulls_pyarrow):0
  col (Real) = 0.1

OGRFeature(test_nulls_pyarrow):1
  col (Real) = nan

OGRFeature(test_nulls_pyarrow):2
  col (Real) = (null)

[jorisvandenbossche]

Opened #122 to further track this

[jorisvandenbossche]

This doesn't have any effect in practice, as that will get converted to twice a NaN by pandas:

In [10]: pd.Series([1.0, None, np.nan])
Out[10]: 
0    1.0
1    NaN
2    NaN
dtype: float64

[theroggy]

Yes, that's indeed the case. Nonetheless I think it is transparant that it is explicitly in the test?

But obviously if you think that's better I can put 2 times np.nan or whatever...

[brendan-ward]

Thanks for working on this @theroggy ; a few comments to add

[brendan-ward]

It seems like the check for isnan should be in the elif field_type == OFTReal: block? The field type for the incoming column should be float32 / float64 rather than object (though I haven't verified this)

[brendan-ward]

Thanks @theroggy !