Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor attribute definition and location #101

Merged
merged 6 commits into from
Apr 21, 2022
+327 −274
Merged

Refactor attribute definition and location #101

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d537f33
Use XML names not STAC ones in datasets
alexamici Apr 20, 2022
32f86f7
Check existing product
alexamici Apr 20, 2022
6ae2df0
Drop more STAC like naming
alexamici Apr 21, 2022
6686ce9
Duplicate attributes to subgroups and arrays
alexamici Apr 21, 2022
4c42bc4
Add more attrs to data variables
alexamici Apr 21, 2022
90ef68e
Support writing partial products
alexamici Apr 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Duplicate attributes to subgroups and arrays
@alexamici
alexamici committed Apr 21, 2022

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
commit 6686ce9d5ef780dd6d7b5daa7c95d7819b276b51
224 changes: 112 additions & 112 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -74,19 +74,19 @@ Dimensions: ()
Data variables:
*empty*
Attributes: ...
constellation: sentinel-1
platform: sentinel-1a
instrument: ['c-sar']
sat:orbit_state: ascending
sat:absolute_orbit: 37258
sat:relative_orbit: 86
... ...
sar:product_type: SLC
xs:instrument_mode_swaths: ['S3']
group: /
subgroups: ['S3', 'S3/VH', 'S3/VH/orbit', 'S3/VH/attitud...
Conventions: CF-1.8
history: created by xarray_sentinel-...
family_name: SENTINEL-1
number: A
mode: SM
swaths: ['S3']
orbit_number: 37258
relative_orbit_number: 86
... ...
start_time: 2021-04-01T15:28:55.111501
stop_time: 2021-04-01T15:29:14.277650
group: /
subgroups: ['S3', 'S3/VH', 'S3/VH/orbit', 'S3/V...
Conventions: CF-1.8
history: created by xarray_sentinel-...
```

@@ -114,19 +114,19 @@ Coordinates:
Data variables:
measurement (azimuth_time, slant_range_time) complex64 ...
Attributes: ...
sar:center_frequency: 5.40500045433435
sar:pixel_spacing_azimuth: 3.55338
sar:pixel_spacing_range: 2.246363
azimuth_time_interval: 0.0005194923129469381
slant_range_time_interval: 1.498612395219899e-08
incidence_angle_mid_swath: 32.03479766845703
... ...
sar:product_type: SLC
xs:instrument_mode_swaths: ['S3']
group: /S3/VH
subgroups: ['orbit', 'attitude', 'azimuth_fm_rate', 'dc_...
Conventions: CF-1.8
history: created by xarray_sentinel-...
family_name: SENTINEL-1
number: A
mode: SM
swaths: ['S3']
orbit_number: 37258
relative_orbit_number: 86
... ...
range_sampling_rate: 66728395.09333333
incidence_angle_mid_swath: 32.03479766845703
group: /S3/VH
subgroups: ['orbit', 'attitude', 'azimuth_fm_ra...
Conventions: CF-1.8
history: created by xarray_sentinel-...
```

@@ -173,19 +173,19 @@ Data variables:
gamma (line, pixel) float32 ...
dn (line, pixel) float32 ...
Attributes: ...
constellation: sentinel-1
platform: sentinel-1a
instrument: ['c-sar']
sat:orbit_state: ascending
sat:absolute_orbit: 37258
sat:relative_orbit: 86
... ...
xs:instrument_mode_swaths: ['S3']
group: /S3/VH/calibration
Conventions: CF-1.8
title: Calibration coefficients
comment: The dataset contains calibration information ...
history: created by xarray_sentinel-...
family_name: SENTINEL-1
number: A
mode: SM
swaths: ['S3']
orbit_number: 37258
relative_orbit_number: 86
... ...
stop_time: 2021-04-01T15:29:14.277650
group: /S3/VH/calibration
Conventions: CF-1.8
title: Calibration coefficients
comment: The dataset contains calibration inf...
history: created by xarray_sentinel-...
```

@@ -247,20 +247,20 @@ Coordinates:
slant_range_time (pixel) float64 ...
Data variables:
measurement (line, pixel) complex64 ...
Attributes: (12/26)
sar:center_frequency: 5.40500045433435
sar:pixel_spacing_azimuth: 13.9283
sar:pixel_spacing_range: 2.329562
azimuth_time_interval: 0.002055556299999998
slant_range_time_interval: 1.554116558005821e-08
incidence_angle_mid_swath: 33.63858785673874
... ...
sar:product_type: SLC
xs:instrument_mode_swaths: ['IW1', 'IW2', 'IW3']
group: /IW1/HH
subgroups: ['orbit', 'attitude', 'azimuth_fm_rate', 'dc_...
Conventions: CF-1.8
history: created by xarray_sentinel-...
Attributes: ...
family_name: SENTINEL-1
number: A
mode: IW
swaths: ['IW1', 'IW2', 'IW3']
orbit_number: 42768
relative_orbit_number: 171
... ...
number_of_bursts: 9
lines_per_burst: 1500
group: /IW1/HH
subgroups: ['orbit', 'attitude', 'azimuth_fm_ra...
Conventions: CF-1.8
history: created by xarray_sentinel-...
```

@@ -281,20 +281,20 @@ Coordinates:
* slant_range_time (slant_range_time) float64 0.005348 0.005349 ... 0.005677
Data variables:
measurement (azimuth_time, slant_range_time) complex64 ...
Attributes: (12/27)
sar:center_frequency: 5.40500045433435
sar:pixel_spacing_azimuth: 13.9283
sar:pixel_spacing_range: 2.329562
azimuth_time_interval: 0.002055556299999998
slant_range_time_interval: 1.554116558005821e-08
incidence_angle_mid_swath: 33.63858785673874
Attributes: ...
family_name: SENTINEL-1
number: A
mode: IW
swaths: ['IW1', 'IW2', 'IW3']
orbit_number: 42768
relative_orbit_number: 171
... ...
group: /IW1/HH
Conventions: CF-1.8
history: created by xarray_sentinel-...
azimuth_anx_time: 2136.774327
burst_index: 8
burst_id: 365923
group: /IW1/HH
Conventions: CF-1.8
history: created by xarray_sentinel-...
azimuth_anx_time: 2136.774327
burst_index: 8
burst_id: 365923
```

@@ -312,20 +312,20 @@ Coordinates:
* slant_range_time (slant_range_time) float64 0.005348 0.005349 ... 0.005677
Data variables:
measurement (azimuth_time, slant_range_time) complex64 ...
Attributes: (12/27)
sar:center_frequency: 5.40500045433435
sar:pixel_spacing_azimuth: 13.9283
sar:pixel_spacing_range: 2.329562
azimuth_time_interval: 0.002055556299999998
slant_range_time_interval: 1.554116558005821e-08
incidence_angle_mid_swath: 33.63858785673874
Attributes: ...
family_name: SENTINEL-1
number: A
mode: IW
swaths: ['IW1', 'IW2', 'IW3']
orbit_number: 42768
relative_orbit_number: 171
... ...
group: /IW1/HH
Conventions: CF-1.8
history: created by xarray_sentinel-...
azimuth_anx_time: 2136.774327
burst_index: 8
burst_id: 365923
group: /IW1/HH
Conventions: CF-1.8
history: created by xarray_sentinel-...
azimuth_anx_time: 2136.774327
burst_index: 8
burst_id: 365923
```

@@ -349,19 +349,19 @@ Coordinates:
Data variables:
measurement (azimuth_time, slant_range_time) complex64 ...
Attributes: ...
sar:center_frequency: 5.40500045433435
sar:pixel_spacing_azimuth: 13.94053
sar:pixel_spacing_range: 2.329562
azimuth_time_interval: 0.002055556299999998
slant_range_time_interval: 1.554116558005821e-08
incidence_angle_mid_swath: 33.87494380774521
... ...
xs:instrument_mode_swaths: ['IW1', 'IW2', 'IW3']
group: /IW1/VH
azimuth_anx_time: 2210.634453
burst_index: 8
Conventions: CF-1.8
history: created by xarray_sentinel-...
family_name: SENTINEL-1
number: B
mode: IW
swaths: ['IW1', 'IW2', 'IW3']
orbit_number: 26269
relative_orbit_number: 168
... ...
lines_per_burst: 1501
group: /IW1/VH
azimuth_anx_time: 2210.634453
burst_index: 8
Conventions: CF-1.8
history: created by xarray_sentinel-...
```

@@ -380,26 +380,26 @@ Coordinates:
* azimuth_time (azimuth_time) datetime64[ns] 2021-04-01T15:28:55.11150...
* slant_range_time (slant_range_time) float64 0.005273 0.005273 ... 0.005303
Attributes:
sar:center_frequency: 5.40500045433435
sar:pixel_spacing_azimuth: 3.55338
sar:pixel_spacing_range: 2.246363
azimuth_time_interval: 0.0005194923129469381
slant_range_time_interval: 1.498612395219899e-08
incidence_angle_mid_swath: 32.03479766845703
sat:anx_datetime: 2021-04-01T13:53:42.874198Z
constellation: sentinel-1
platform: sentinel-1a
instrument: ['c-sar']
sat:orbit_state: ascending
sat:absolute_orbit: 37258
sat:relative_orbit: 86
sar:frequency_band: C
sar:instrument_mode: SM
sar:polarizations: ['VV', 'VH']
sar:product_type: SLC
xs:instrument_mode_swaths: ['S3']
units: m2 m-2
long_name: gamma
family_name: SENTINEL-1
number: A
mode: SM
swaths: ['S3']
orbit_number: 37258
relative_orbit_number: 86
pass: ASCENDING
ascending_node_time: 2021-04-01T13:53:42.874198
transmitter_receiver_polarisations: ['VV', 'VH']
product_type: SLC
start_time: 2021-04-01T15:28:55.111501
stop_time: 2021-04-01T15:29:14.277650
radar_frequency: 5.40500045433435
azimuth_pixel_spacing: 3.55338
range_pixel_spacing: 2.246363
azimuth_time_interval: 0.0005194923129469381
range_sampling_rate: 66728395.09333333
incidence_angle_mid_swath: 32.03479766845703
units: m2 m-2
long_name: gamma
```

79 changes: 49 additions & 30 deletions xarray_sentinel/sentinel1.py
View file
Original file line number Diff line number Diff line change
@@ -63,7 +63,9 @@ def normalise_group(group: T.Optional[str]) -> T.Tuple[str, T.Optional[int]]:
return group, burst_index


def open_calibration_dataset(calibration: esa_safe.PathType) -> xr.Dataset:
def open_calibration_dataset(
calibration: esa_safe.PathType, attrs: T.Dict[str, T.Any] = {}
) -> xr.Dataset:
calibration_vectors = esa_safe.parse_tag_as_list(
calibration, ".//calibrationVector", "calibration"
)
@@ -103,10 +105,12 @@ def open_calibration_dataset(calibration: esa_safe.PathType) -> xr.Dataset:
}
coords = {"line": line_list, "pixel": pixel_list[0]}

return xr.Dataset(data_vars=data_vars, coords=coords)
return xr.Dataset(data_vars=data_vars, coords=coords, attrs=attrs)


def open_noise_range_dataset(noise: esa_safe.PathType) -> xr.Dataset:
def open_noise_range_dataset(
noise: esa_safe.PathType, attrs: T.Dict[str, T.Any] = {}
) -> xr.Dataset:
noise_vectors = esa_safe.parse_tag_as_list(noise, ".//noiseRangeVector", "noise")

azimuth_time_list = []
@@ -132,10 +136,12 @@ def open_noise_range_dataset(noise: esa_safe.PathType) -> xr.Dataset:
}
coords = {"line": line_list, "pixel": pixel_list[0]}

return xr.Dataset(data_vars=data_vars, coords=coords)
return xr.Dataset(data_vars=data_vars, coords=coords, attrs=attrs)


def open_noise_azimuth_dataset(noise: esa_safe.PathType) -> xr.Dataset:
def open_noise_azimuth_dataset(
noise: esa_safe.PathType, attrs: T.Dict[str, T.Any] = {}
) -> xr.Dataset:
noise_vectors = esa_safe.parse_tag_as_list(noise, ".//noiseAzimuthVector", "noise")

first_range_sample = []
@@ -155,11 +161,11 @@ def open_noise_azimuth_dataset(noise: esa_safe.PathType) -> xr.Dataset:
data_vars["noiseAzimuthLut"] = ("line", noiseAzimuthLut_list[0])
coords["line"] = line_list[0]

return xr.Dataset(data_vars=data_vars, coords=coords)
return xr.Dataset(data_vars=data_vars, coords=coords, attrs=attrs)


def open_coordinate_conversion_dataset(
annotation_path: esa_safe.PathType,
annotation_path: esa_safe.PathType, attrs: T.Dict[str, T.Any] = {}
) -> xr.Dataset:
coordinate_conversion = esa_safe.parse_tag_as_list(
annotation_path, ".//coordinateConversionList/coordinateConversion"
@@ -195,10 +201,12 @@ def open_coordinate_conversion_dataset(
data_vars["srgrCoefficients"] = (("azimuth_time", "degree"), srgrCoefficients)
data_vars["grsrCoefficients"] = (("azimuth_time", "degree"), grsrCoefficients)

return xr.Dataset(data_vars=data_vars, coords=coords)
return xr.Dataset(data_vars=data_vars, coords=coords, attrs=attrs)


def open_gcp_dataset(annotation: esa_safe.PathOrFileType) -> xr.Dataset:
def open_gcp_dataset(
annotation: esa_safe.PathOrFileType, attrs: T.Dict[str, T.Any] = {}
) -> xr.Dataset:
geolocation_grid_points = esa_safe.parse_tag_as_list(
annotation, ".//geolocationGridPoint"
)
@@ -217,11 +225,11 @@ def open_gcp_dataset(annotation: esa_safe.PathOrFileType) -> xr.Dataset:
shape = (len(azimuth_time), len(slant_range_time))
dims = ("azimuth_time", "slant_range_time")
data_vars = {
"latitude": (dims, np.full(shape, np.nan)),
"longitude": (dims, np.full(shape, np.nan)),
"height": (dims, np.full(shape, np.nan)),
"incidenceAngle": (dims, np.full(shape, np.nan)),
"elevationAngle": (dims, np.full(shape, np.nan)),
"latitude": (dims, np.full(shape, np.nan), attrs),
"longitude": (dims, np.full(shape, np.nan), attrs),
"height": (dims, np.full(shape, np.nan), attrs),
"incidenceAngle": (dims, np.full(shape, np.nan), attrs),
"elevationAngle": (dims, np.full(shape, np.nan), attrs),
}
line = sorted(line_set)
pixel = sorted(pixel_set)
@@ -239,11 +247,14 @@ def open_gcp_dataset(annotation: esa_safe.PathOrFileType) -> xr.Dataset:
"line": ("azimuth_time", line),
"pixel": ("slant_range_time", pixel),
},
attrs=attrs,
)
return ds


def open_attitude_dataset(annotation: esa_safe.PathOrFileType) -> xr.Dataset:
def open_attitude_dataset(
annotation: esa_safe.PathOrFileType, attrs: T.Dict[str, T.Any] = {}
) -> xr.Dataset:
attitudes = esa_safe.parse_tag_as_list(annotation, ".//attitude")

variables = ["q0", "q1", "q2", "q3", "wx", "wy", "wz", "pitch", "roll", "yaw"]
@@ -257,12 +268,15 @@ def open_attitude_dataset(annotation: esa_safe.PathOrFileType) -> xr.Dataset:
ds = xr.Dataset(
data_vars=data_vars,
coords={"azimuth_time": [np.datetime64(dt) for dt in azimuth_time]},
attrs=attrs,
)

return ds


def open_orbit_dataset(annotation: esa_safe.PathOrFileType) -> xr.Dataset:
def open_orbit_dataset(
annotation: esa_safe.PathOrFileType, attrs: T.Dict[str, T.Any] = {}
) -> xr.Dataset:
orbits = esa_safe.parse_tag_as_list(annotation, ".//orbit")

reference_system = orbits[0]["frame"]
@@ -286,7 +300,7 @@ def open_orbit_dataset(annotation: esa_safe.PathOrFileType) -> xr.Dataset:
position = xr.Variable(data=data["position"], dims=("axis", "azimuth_time")) # type: ignore
velocity = xr.Variable(data=data["velocity"], dims=("axis", "azimuth_time")) # type: ignore

attrs = {}
attrs = attrs | {}
if reference_system is not None:
attrs.update({"reference_system": reference_system})

@@ -302,7 +316,9 @@ def open_orbit_dataset(annotation: esa_safe.PathOrFileType) -> xr.Dataset:
return ds


def open_dc_estimate_dataset(annotation: esa_safe.PathOrFileType) -> xr.Dataset:
def open_dc_estimate_dataset(
annotation: esa_safe.PathOrFileType, attrs: T.Dict[str, T.Any] = {}
) -> xr.Dataset:
dc_estimates = esa_safe.parse_tag_as_list(annotation, ".//dcEstimate")

azimuth_time = []
@@ -324,11 +340,14 @@ def open_dc_estimate_dataset(annotation: esa_safe.PathOrFileType) -> xr.Dataset:
"azimuth_time": [np.datetime64(at) for at in azimuth_time],
"degree": list(range(len(data_dc_poly[0]))),
},
attrs=attrs,
)
return ds


def open_azimuth_fm_rate_dataset(annotation: esa_safe.PathOrFileType) -> xr.Dataset:
def open_azimuth_fm_rate_dataset(
annotation: esa_safe.PathOrFileType, attrs: T.Dict[str, T.Any] = {}
) -> xr.Dataset:
azimuth_fm_rates = esa_safe.parse_tag_as_list(annotation, ".//azimuthFmRate")

azimuth_time = []
@@ -353,6 +372,7 @@ def open_azimuth_fm_rate_dataset(annotation: esa_safe.PathOrFileType) -> xr.Data
"azimuth_time": [np.datetime64(at) for at in azimuth_time],
"degree": list(range(len(azimuth_fm_rate_poly[0]))),
},
attrs=attrs,
)
return ds

@@ -397,6 +417,7 @@ def open_pol_dataset(
measurement: esa_safe.PathOrFileType,
annotation: esa_safe.PathOrFileType,
fs: T.Optional[fsspec.AbstractFileSystem] = None,
attrs: T.Dict[str, T.Any] = {},
) -> xr.Dataset:

product_information = esa_safe.parse_tag(annotation, ".//productInformation")
@@ -411,7 +432,7 @@ def open_pol_dataset(
number_of_bursts = swath_timing["burstList"]["@count"]
range_pixel_spacing = image_information["rangePixelSpacing"]

attrs = {
attrs = attrs | {
"radar_frequency": product_information["radarFrequency"] / 10**9,
"azimuth_pixel_spacing": image_information["azimuthPixelSpacing"],
"range_pixel_spacing": range_pixel_spacing,
@@ -748,7 +769,7 @@ def open_sentinel1_dataset(
product_path = os.path.dirname(manifest_path)

with fs.open(manifest_path) as file:
product_attrs, product_files = esa_safe.parse_manifest_sentinel1(file)
common_attrs, product_files = esa_safe.parse_manifest_sentinel1(file)

if override_product_files:
product_files = do_override_product_files(override_product_files, product_files)
@@ -767,7 +788,7 @@ def open_sentinel1_dataset(

metadata = ""

ds = xr.Dataset()
ds = xr.Dataset(attrs=common_attrs)
if group == "":
subgroups = list(groups)
else:
@@ -777,19 +798,17 @@ def open_sentinel1_dataset(

if group.count("/") == 1:
with fs.open(groups[group][1]) as annotation:
ds = open_pol_dataset(groups[group][0], annotation, fs=fs)
ds = open_pol_dataset(
groups[group][0], annotation, fs=fs, attrs=common_attrs
)
elif group.count("/") == 2:
_, _, metadata = group.split("/", 2)
with fs.open(groups[group][0]) as file:
ds = METADATA_OPENERS[metadata](file)

for data_var in ds.data_vars:
ds.data_vars[data_var].attrs.update(product_attrs)
ds = METADATA_OPENERS[metadata](file, attrs=common_attrs)

product_attrs["group"] = absgroup
ds.attrs["group"] = absgroup
if len(subgroups):
product_attrs["subgroups"] = subgroups
ds.attrs.update(product_attrs) # type: ignore
ds.attrs["subgroups"] = subgroups

if group.count("/") == 1 and burst_index is not None:
ds = crop_burst_dataset(ds, burst_index=burst_index)