Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support writing object columns with np.nan values #118

Merged
merged 6 commits into from
Jun 7, 2022
Merged

Support writing object columns with np.nan values #118

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
da48362
Add support to write nan values in object columns
theroggy Jun 6, 2022
ce1b68a
Add to changelog
theroggy Jun 6, 2022
530b5d4
Consistent newlines
theroggy Jun 6, 2022
744a031
Move is None check within string/object handling
theroggy Jun 6, 2022
6c38520
Only test on gpkg
theroggy Jun 6, 2022
cb28a2e
Update CHANGES.md
theroggy Jun 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
- Use certifi to set `GDAL_CURL_CA_BUNDLE` / `PROJ_CURL_CA_BUNDLE` defaults (#97)
- automatically detect driver for `.geojson`, `.geojsonl` and `.geojsons` files (#101)
- read DateTime fields with millisecond accuracy (#111)
- support writing object columns with np.nan values (#118)

### Breaking changes

Expand Down
33 changes: 18 additions & 15 deletions pyogrio/_io.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import warnings
from libc.stdint cimport uint8_t
from libc.stdlib cimport malloc, free
from libc.string cimport strlen
from libc.math cimport isnan

cimport cython
import numpy as np
Expand Down Expand Up @@ -1359,22 +1360,24 @@ def ogr_write(str path, str layer, str driver, geometry, field_data, fields,
field_value = field_data[field_idx][i]
field_type = field_types[field_idx][0]

if field_value is None:
OGR_F_SetFieldNull(ogr_feature, field_idx)

elif field_type == OFTString:
if field_type == OFTString:
# TODO: encode string using approach from _get_internal_encoding which checks layer capabilities
try:
# this will fail for strings mixed with nans
value_b = field_value.encode("UTF-8")

except AttributeError:
raise ValueError(f"Could not encode value '{field_value}' in field '{fields[field_idx]}' to string")

except Exception:
raise

OGR_F_SetFieldString(ogr_feature, field_idx, value_b)
if (
field_value is None
or (isinstance(field_value, float) and isnan(field_value))
):
OGR_F_SetFieldNull(ogr_feature, field_idx)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we combine this with the field_value is None check above to set FieldNull in a single place?
(or move that check here, as currently the only way to get a None is through object dtype)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like the check for isnan should be in the elif field_type == OFTReal: block? The field type for the incoming column should be float32 / float64 rather than object (though I haven't verified this)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brendan-ward note that the specific case that is being fixed here is if you have an object dtype column (but which contains NaN, pandas doesn't really distinguish None vs NaN in object columns)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure... if you prefer that... I moved the "is None" check as it is redundant for column types other than string/object.

Copy link
Member Author

@theroggy theroggy Jun 6, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brendan-ward in addition to what @jorisvandenbossche wrote: np.nan values in an OFTReal (float) column are automatically treated correctly (as null) by GDAL (OGR_F_SetFieldDouble), so no need to explicitly call OGR_F_SetFieldNull for OFTReal columns. Only for object columns this is needed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

np.nan values in an OFTReal (float) column are automatically treated correctly (as null) by GDAL (OGR_F_SetFieldDouble), so no need to explicitly call OGR_F_SetFieldNull for OFTReal columns. Only for object columns this is needed.

Actually, that's not fully correct I think. They are written as NaN and not as Null. Since numpy/pandas only support NaN in float arrays, that's not an issue for correct rountrip for geopandas->gdal->geopandas, though.

Small illustrations:

# write file with GDAL + pyogrio
import geopandas
import pyogrio
gdf = geopandas.GeoDataFrame({'col': [0.1, np.nan]}, geometry=geopandas.points_from_xy([0, 1], [0, 1]))
pyogrio.write_dataframe(gdf, "test_nulls_pyogrio.arrow", driver="Arrow")

# write file with pyarrow that includes both NaN and Null (Arrow distinguishes both)
import pyarrow as pa
from pyarrow import feather
feather.write_feather(pa.table({"col": [0.1, np.nan, None]}), "test_nulls_pyarrow.arrow")

And check both using GDAL's ogrinfo:

(gdal-dev) $ ogrinfo test_nulls_pyogrio.arrow -al
INFO: Open of `test_nulls_pyogrio.arrow'
      using driver `Arrow' successful.

Layer name: test_nulls_pyogrio
Geometry: Point
Feature Count: 2
Extent: (0.000000, 0.000000) - (1.000000, 1.000000)
Layer SRS WKT:
(unknown)
Geometry Column = geometry
col: Real (0.0)
OGRFeature(test_nulls_pyogrio):0
  col (Real) = 0.1
  POINT (0 0)

OGRFeature(test_nulls_pyogrio):1
  col (Real) = nan
  POINT (1 1)

(gdal-dev) $ ogrinfo test_nulls_pyarrow.arrow -al
INFO: Open of `test_nulls_pyarrow.arrow'
      using driver `Arrow' successful.

Layer name: test_nulls_pyarrow
Geometry: None
Feature Count: 3
Layer SRS WKT:
(unknown)
col: Real (0.0)
OGRFeature(test_nulls_pyarrow):0
  col (Real) = 0.1

OGRFeature(test_nulls_pyarrow):1
  col (Real) = nan

OGRFeature(test_nulls_pyarrow):2
  col (Real) = (null)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened #122 to further track this


else:
try:
value_b = field_value.encode("UTF-8")
OGR_F_SetFieldString(ogr_feature, field_idx, value_b)

except AttributeError:
raise ValueError(f"Could not encode value '{field_value}' in field '{fields[field_idx]}' to string")

except Exception:
raise

elif field_type == OFTInteger:
OGR_F_SetFieldInteger(ogr_feature, field_idx, field_value)
Expand Down
22 changes: 22 additions & 0 deletions pyogrio/tests/test_geopandas_io.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -704,3 +704,25 @@ def test_custom_crs_io(tmpdir, naturalearth_lowres_all_ext):
assert crs["lat_2"] == 51.5
assert crs["lon_0"] == 4.3
assert df.crs.equals(expected.crs)


def test_write_read_null(tmp_path):
from shapely.geometry import Point

output_path = tmp_path / f"test_write_nan.gpkg"
geom = Point(0, 0)
test_data = {
"geometry": [geom, geom, geom],
"float64": [1.0, None, np.nan],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't have any effect in practice, as that will get converted to twice a NaN by pandas:

In [10]: pd.Series([1.0, None, np.nan])
Out[10]: 
0    1.0
1    NaN
2    NaN
dtype: float64

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's indeed the case. Nonetheless I think it is transparant that it is explicitly in the test?

But obviously if you think that's better I can put 2 times np.nan or whatever...

"object_str": ["test", None, np.nan],
}
test_gdf = gp.GeoDataFrame(test_data, crs="epsg:31370")
write_dataframe(test_gdf, output_path)
result_gdf = read_dataframe(output_path)
assert len(test_gdf) == len(result_gdf)
assert result_gdf["float64"][0] == 1.0
assert pd.isna(result_gdf["float64"][1])
assert pd.isna(result_gdf["float64"][2])
assert result_gdf["object_str"][0] == "test"
assert result_gdf["object_str"][1] is None
assert result_gdf["object_str"][2] is None