diff --git a/docs/MetObs_documentation_flat.rst b/docs/MetObs_documentation_flat.rst index 25e74286..8c69cc34 100644 --- a/docs/MetObs_documentation_flat.rst +++ b/docs/MetObs_documentation_flat.rst @@ -13,7 +13,7 @@ the MetObs toolkit to be used by a user and developer. Station Obstype Analysis - Modeldata + Gee modeldata Template Special functions diff --git a/docs/intro.rst b/docs/intro.rst index 78aa90be..a5dd2dd3 100644 --- a/docs/intro.rst +++ b/docs/intro.rst @@ -123,19 +123,28 @@ The Analysis methods are based on aggregating the observations to get insight in -Modeldata() -------------- -The :ref:`Modeldata ` holds time-series of data from a source other than observations (i.g. a model). The time-series are taken at the same coordinates as the stations and the -names of the stations are used as well. +GEE Modeldata classes +---------------------- -This class is used for comparing other sources to observations and for filling in missing observations and gaps in the observations. +Two classes are designed to interact with a GEE (Google Earth Engine) dataset: + +* `GeeStaticModelData`: This class handles GEE Datasets that do not have a time dimension (static). This class is used to extract GEE dataset values at the location of the station (or buffers arround them). +* `GeeDynamicModelData`: This class handles GEE Dataset that have a time dimension. This class is used to extract timeseries of GEE dataset values at the stations locations. + +Both classes can hold metadata (=Coordinates of the stations), and the ´GeeDynamicModelData` class can hold timeseries data. +These classes are used for extracting extra metadata (landcover, altitude, soil properties, ...) and for comparing +observations with modelled data (plotting, filling gaps, ...). + +There are default modeldata classes prepared, and they are stored in the `Dataset.gee_datasets`. .. code-block:: python - ERA5_timeseries = your_dataset.get_modeldata(modelname='ERA5_hourly', - obstype='temp') + ERA5_timeseries = your_dataset.get_modeldata( + Model=your_datast.gee_datasets['ERA5-land'], + obstype='temp') +See the API documentiontion :ref:`Geemodeldata ` for more details. The toolkit makes use of the Google Earth Engine (GEE), to extract these time-series. To use the GEE API, follow these steps on :ref:`Using Google Earth Engine`. diff --git a/docs/reference/geemodeldata.rst b/docs/reference/geemodeldata.rst new file mode 100644 index 00000000..b58c8f96 --- /dev/null +++ b/docs/reference/geemodeldata.rst @@ -0,0 +1,58 @@ +.. _Geemodeldata api: + +=============== +GEE Modeldata +=============== + +The Gee (Google Earth Engine) Modeldata is the bridge between the Metobs-toolkit +and the GEE services. There are two classes: + + * `GeeStaticModelData`: This class handles GEE Datasets that do not have a time dimension (static). This class is used to extract GEE dataset values at the location of the station (or buffers arround them). + * `GeeDynamicModelData`: This class handles GEE Dataset that have a time dimension. This class is used to extract timeseries of GEE dataset values at the stations locations. + +Both classes can hold metadata (=Coordinates of the stations), and the ´GeeDynamicModelData` class can hold timeseries data. + +.. note:: + Extracting data from a GEE dataset, can be done directly from a `GeeStaticModelData` + or a `GeeDynamicModelData`. In addition, one can call extractions also directly from + a `Dataset`, see the *Extracting data* section in the `Dataset` API documentation. + + +.. note:: + Keep in mind that the Modeldata does not hold gridded data, but timeseries of + point extractions at the locations of the stations. Therefore a Modeldata + instance is linked to a collection of stations defined by the `metadf` attribute. + +Demo examples on the Modeldata class can be found here: :ref:`Extracting ERA5 timeseries`_ . + + +GeeStaticModelData +-------------------- + +.. currentmodule:: metobs_toolkit + +.. autosummary:: + :toctree: api/ + + GeeStaticModelData + GeeStaticModelData.get_info + GeeStaticModelData.extract_static_point_data + GeeStaticModelData.extract_static_buffer_frac_data + GeeStaticModelData.make_gee_plot + + +GeeDynamicModelData +-------------------- + +.. autosummary:: + :toctree: api/ + + GeeDynamicModelData + GeeDynamicModelData.get_info + GeeDynamicModelData.add_modelobstype + GeeDynamicModelData.extract_timeseries_data + GeeDynamicModelData.save_modeldata + GeeDynamicModelData.set_modeldata_from_csv + GeeDynamicModelData.import_modeldata_from_pkl + GeeDynamicModelData.make_plot + GeeDynamicModelData.make_gee_plot diff --git a/docs/reference/modeldata.rst b/docs/reference/modeldata.rst deleted file mode 100644 index ac6fc59f..00000000 --- a/docs/reference/modeldata.rst +++ /dev/null @@ -1,53 +0,0 @@ -.. _Modeldata api: - -============ -Modeldata -============ - -The Modeldata holds timeseries data which is the output of a model and methods -to interact with the Google Earth Engine to extract timeseries. - -.. note:: - Keep in mind that the Modeldata does not hold gridded data, but timeseries of - point extractions at the locations of the stations. Therefore a Modeldata - instance is linked to a collection of stations defined by the `metadf` attribute. - -Demo examples on the Modeldata class can be found here: :ref:`Extracting ERA5 timeseries`_ . - - -.. currentmodule:: metobs_toolkit - -Constructor ------------ - -.. autosummary:: - :toctree: api/ - - Modeldata - - - -Common methods --------------------- - -.. autosummary:: - :toctree: api/ - - Modeldata.get_info - Modeldata.add_obstype - Modeldata.save_modeldata - Modeldata.import_modeldata - Modeldata.make_plot - - -GEE interactions ------------------- - -.. autosummary:: - :toctree: api/ - - Modeldata.add_gee_dataset - Modeldata.list_gee_datasets - Modeldata.get_gee_dataset_data - Modeldata.get_ERA5_data - Modeldata.set_model_from_csv diff --git a/docs/reference/obstype.rst b/docs/reference/obstype.rst index eb20e538..e92fc80a 100644 --- a/docs/reference/obstype.rst +++ b/docs/reference/obstype.rst @@ -46,87 +46,41 @@ Developers methods and attributes for Obstype .. _ModelObstype api: -============== -ModelObstype -============== +============================== +Obstypes used for GEE datasets +=============================== -A child of :ref:`Obstype api` that adds info on how this observationtype is represented in modeloutput. -All methods of Obstype() are inhereted. +A child of :ref:`Obstype api` that adds info on how this observationtype is represented in a GEE dataset. -Constructor for ModelObstype ------------------------------ +There are two classes: + * `ModelObstype` : Represent a scalar Obstype for which there exists a band in a GEE dataset. + * `ModelObstype_Vectorfield` : Represent a vectorfield, for which the *u* and *v* components exists in bands of a GEE dataset. -.. autosummary:: - :toctree: api/ - ModelObstype +ModelObstype +----------------------------- -General methods and attributes for ModelObstype ------------------------------------------------- +All methods of `Obstype` are inhereted. .. autosummary:: :toctree: api/ - ModelObstype.get_mapped_datasets - ModelObstype.get_bandname + ModelObstype ModelObstype.get_modelunit - ModelObstype.add_new_band + ModelObstype.get_modelband -Developers methods and attributes for ModelObstype ---------------------------------------------------- -.. autosummary:: - :toctree: api/ - - ModelObstype.get_bandname_mapper - ModelObstype.has_mapped_band - - -.. _ModelObstype_Vectorfield api: - -========================== ModelObstype_Vectorfield -========================== - -A child of :ref:`Obstype api`, similar to :ref:`ModelObstype api`, that adds info on how this handle 2D vectorfields in the modeloutput. -A vectorfield in the modeloutput is defined by its components. - -All methods of Obstype() are inhereted. +----------------------------- -Constructor for ModelObstype_Vectorfield ------------------------------------------ +All methods of `Obstype` are inhereted. .. autosummary:: :toctree: api/ ModelObstype_Vectorfield - -General methods and attributes for ModelObstype_Vectorfield -------------------------------------------------------------- - -.. autosummary:: - :toctree: api/ - - ModelObstype_Vectorfield.get_mapped_datasets ModelObstype_Vectorfield.get_modelunit - ModelObstype_Vectorfield.add_new_band - -Developers methods and attributes ------------------------------------- - -.. autosummary:: - :toctree: api/ - - ModelObstype_Vectorfield.get_bandname_mapper - ModelObstype_Vectorfield.has_mapped_band - - -Developers vectorfield conversion functions ---------------------------------------------- -These functions are used by the :ref:`ModelObstype_Vectorfield api` to convert components to amplitudes and angles. - -.. autosummary:: - :toctree: api/ - - obstype_modeldata.compute_amplitude - obstype_modeldata.compute_angle + ModelObstype_Vectorfield.get_modelband_u + ModelObstype_Vectorfield.get_modelband_v + ModelObstype_Vectorfield.compute_angle + ModelObstype_Vectorfield.compute_amplitude diff --git a/metobs_toolkit/dataset_visuals.py b/metobs_toolkit/dataset_visuals.py index b606a4bd..626722a7 100644 --- a/metobs_toolkit/dataset_visuals.py +++ b/metobs_toolkit/dataset_visuals.py @@ -949,158 +949,6 @@ def make_gee_dynamic_spatialplot( overwrite=overwrite, ) - # def make_gee_plot( - # self, gee_map="worldcover", show_stations=True, save=False, outputfile=None - # ): - # """Make an interactive plot of a google earth dataset. - - # The location of the stations can be plotted on top of it. - - # Parameters - # ---------- - # gee_map : str, optional - # The name of the dataset to use. This name should be present in the - # settings.gee['gee_dataset_info']. If aggregat is True, an aggregation - # scheme should included as well. The default is 'worldcover' - # show_stations : bool, optional - # If True, the stations will be plotted as markers. The default is True. - # save : bool, optional - # If True, the map will be saved as an html file in the output_folder - # as defined in the settings if the outputfile is not set. The - # default is False. - # outputfile : str, optional - # Specify the path of the html file if save is True. If None, and save - # is true, the html file will be saved in the output_folder. The - # default is None. - - # Returns - # ------- - # Map : geemap.foliumap.Map - # The folium Map instance. - - # See Also - # ----------- - # Dataset.make_plot: plot timeseries. - # Dataset.make_geo_plot: geospatial plot. - # Dataset.make_interactive_plot: Interactive geospatial plot. - - # Warning - # --------- - # To display the interactive map a graphical backend is required, which - # is often missing on (free) cloud platforms. Therefore it is better to - # set save=True, and open the .html in your browser - - # Examples - # -------- - # We start by creating a Dataset, and importing data. - - # >>> import metobs_toolkit - # >>> - # >>> #Create your Dataset - # >>> dataset = metobs_toolkit.Dataset() #empty Dataset - # >>> dataset.import_data_from_file( - # ... input_data_file=metobs_toolkit.demo_datafile, - # ... input_metadata_file=metobs_toolkit.demo_metadatafile, - # ... template_file=metobs_toolkit.demo_template, - # ... ) - # >>> print(dataset) - # Dataset instance containing: - # *28 stations - # *['humidity', 'temp', 'wind_direction', 'wind_speed'] observation types present - # *483828 observation records (not Nan's) - # *0 records labeled as outliers - # *8 gaps - # *records range: 2022-09-01 00:00:00+00:00 --> 2022-09-15 23:55:00+00:00 (total duration: 14 days 23:55:00) - # *time zone of the records: UTC - # *Coordinates are available for all stations. - - # To create an interactive Google earth engine plot, we use the `Dataset.make_gee_plot()` - # method. - - # >>> dataset.make_gee_plot(gee_map='worldcover') - # >> import os - # >>> target_file = os.path.join(os.getcwd(), 'interactive_worldcover_plot.html') - # >>> dataset.make_gee_plot(gee_map='worldcover', - # ... save=True, - # ... outputfile=target_file) - # Gee Map will be save at ... - - # """ - # # Connect to GEE - # connect_to_gee() - - # # get the mapinfo - # mapinfo = self.settings.gee["gee_dataset_info"][gee_map] - - # # Read in covers, numbers and labels - # covernum = list(mapinfo["colorscheme"].keys()) - # colors = list(mapinfo["colorscheme"].values()) - # covername = [mapinfo["categorical_mapper"][covnum] for covnum in covernum] - - # # create visparams - # vis_params = { - # "min": min(covernum), - # "max": max(covernum), - # "palette": colors, # hex colors! - # } - - # if "band_of_use" in mapinfo: - # band = mapinfo["band_of_use"] - # else: - # band = None - - # Map = folium_plot( - # mapinfo=mapinfo, - # band=band, - # vis_params=vis_params, - # labelnames=covername, - # layername=gee_map, - # legendname=f"{gee_map} covers", - # # showmap = show, - # ) - - # if show_stations: - # if not _validate_metadf(self.metadf): - # logger.warning( - # "Not enough coordinates information is provided to plot the stations." - # ) - # else: - # Map = add_stations_to_folium_map(Map=Map, metadf=self.metadf) - - # # Save if needed - # if save: - # if outputfile is None: - # # Try to save in the output folder - # if self.settings.IO["output_folder"] is None: - # logger.warning( - # "The outputfolder is not set up, use the update_settings to specify the output_folder." - # ) - - # else: - # filename = f"gee_{gee_map}_figure.html" - # filepath = os.path.join(self.settings.IO["output_folder"], filename) - # else: - # # outputfile is specified - # # 1. check extension - # if not outputfile.endswith(".html"): - # outputfile = outputfile + ".html" - - # filepath = outputfile - - # print(f"Gee Map will be save at {filepath}") - # logger.info(f"Gee Map will be save at {filepath}") - # Map.save(filepath) - - # return Map - class MetobsDatasetVisualisationError(Exception): """Exception raised for errors in the template.""" diff --git a/metobs_toolkit/modeldata.py b/metobs_toolkit/modeldata.py index 9d4284fa..6aea62f6 100644 --- a/metobs_toolkit/modeldata.py +++ b/metobs_toolkit/modeldata.py @@ -524,6 +524,7 @@ def extract_static_point_data(self): See Also ----------- + metobs_toolkit.connect_to_gee: Establish connection with GEE services. GeeStaticModelData.set_metadf: Set metadata (station locations). GeeStaticModelData.get_info: Print out detailed info method. GeeStaticModelData.extract_static_buffer_frac_data: Extract buffer fractions. @@ -678,6 +679,7 @@ def extract_static_buffer_frac_data(self, bufferradius, agg_bool=False): See Also ----------- + metobs_toolkit.connect_to_gee: Establish connection with GEE services. GeeStaticModelData.set_metadf: Set metadata (station locations). GeeStaticModelData.get_info: Print out detailed info method. GeeStaticModelData.extract_static_point_data: Extract point values. @@ -911,6 +913,7 @@ def make_gee_plot( See Also ----------- + metobs_toolkit.connect_to_gee: Establish connection with GEE services. GeeStaticModelData.set_metadf: Set metadata (station locations). GeeStaticModelData.get_info: Print out detailed info method. @@ -1601,6 +1604,7 @@ def make_gee_plot( See Also ----------- + metobs_toolkit.connect_to_gee: Establish connection with GEE services. GeeDynamicModelData.set_metadf: Set metadata (station locations). GeeDynamicModelData.get_info: Print out detailed info method. GeeDynamicModelData.make_plot: Make a timeseries plot. @@ -1896,6 +1900,7 @@ def make_plot( See Also ----------- + metobs_toolkit.connect_to_gee: Establish connection with GEE services. GeeDynamicModelData.set_metadf: Set metadata (station locations). GeeDynamicModelData.extract_timeseries_data: Extract data from GEE. GeeDynamicModelData.get_info: Print out detailed info method. @@ -2134,6 +2139,7 @@ def extract_timeseries_data( See Also ----------- + metobs_toolkit.connect_to_gee: Establish connection with GEE services. GeeDynamicModelData.set_metadf: Set metadata (station locations). GeeDynamicModelData.get_info: Print out detailed info method. GeeDynamicModelData.make_plot: Make a timeseries plot. diff --git a/metobs_toolkit/obstype_modeldata.py b/metobs_toolkit/obstype_modeldata.py index ce1a62d9..00e0d0bc 100644 --- a/metobs_toolkit/obstype_modeldata.py +++ b/metobs_toolkit/obstype_modeldata.py @@ -18,6 +18,76 @@ class ModelObstype(Obstype): def __init__(self, obstype, model_unit, model_band): + """Initiate an observation type, to link with a GEE dataset band. + + A ModelObstype is specific to a GEE Dataset, and is therefore added + to a `GeeDynamicModelData` (that facilitates the link with a GEE dataset). + + All methods and attributes are inherited of the `Obstype` class. + + Parameters + ---------- + obstype : metobs_toolkit.Obstype + The Obstype that represents the band in the GEE dataset. + model_unit : str + The units of the GEE band. This can be found in the details of + the corresponding GEE dataset. This unit must be known by the obstype ( + add it if it is not known). + model_band : str + The name of the band representing the obstype. This can be found in + the details of the GEE dataset. + + Returns + ------- + None + + See also + ---------- + Obstype: A regular observation type + ModelObstype_Vectorfield: A vector representation of a ModelObstype. + + Examples + --------- + As example we create a `ModelObstype` for downward solar radiation at the + surface, to be used with the ERA5-land GEE dataset. + + >>> import metobs_toolkit + >>> dataset = metobs_toolkit.Dataset() #empty Dataset + >>> dataset.gee_datasets['ERA5-land'].modelobstypes + {'temp': ModelObstype instance of temp (linked to band: temperature_2m), 'pressure': ModelObstype instance of pressure (linked to band: surface_pressure), 'wind': ModelObstype_Vectorfield instance of wind (linked to bands: u_component_of_wind_10m and v_component_of_wind_10m)} + + There is no default solar radiation modeldata `ModelObstype` present for the + ERA5 `GeeDynamicModelData`. Thus we must create one. + + >>> dataset.obstypes + {'temp': Obstype instance of temp, 'humidity': Obstype instance of humidity, 'radiation_temp': Obstype instance of radiation_temp, 'pressure': Obstype instance of pressure, 'pressure_at_sea_level': Obstype instance of pressure_at_sea_level, 'precip': Obstype instance of precip, 'precip_sum': Obstype instance of precip_sum, 'wind_speed': Obstype instance of wind_speed, 'wind_gust': Obstype instance of wind_gust, 'wind_direction': Obstype instance of wind_direction} + + We see that there is no default `Obstype` that we can use for the + solar radiation. Therefore we must create a new `Obstype`. + + >>> sol_rad_down_surf = metobs_toolkit.Obstype( + ... obsname="solar_rad_down_at_surface", + ... std_unit='J/m²') + + By looking up the details of ERA5-land (https://developers.google.com/earth-engine/datasets/catalog/ECMWF_ERA5_LAND_HOURLY), + we find that the corresponding band is *surface_solar_radiation_downwards* and + the units are in "J/m²" (no coinsidence). + + >>> sol_rad_down_for_ERA5 = metobs_toolkit.ModelObstype( + ... obstype=sol_rad_down_surf, + ... model_unit="J/m²", + ... model_band="surface_solar_radiation_downwards") + >>> sol_rad_down_for_ERA5 + ModelObstype instance of solar_rad_down_at_surface (linked to band: surface_solar_radiation_downwards) + + In pracktice we add it to the knonw ModelObstypes of the ERA5 `GeeDynamicModelData`. + + >>> era5_mod = dataset.gee_datasets['ERA5-land'] + >>> era5_mod.add_modelobstype(sol_rad_down_for_ERA5) + >>> era5_mod.modelobstypes + {'temp': ModelObstype instance of temp (linked to band: temperature_2m), 'pressure': ModelObstype instance of pressure (linked to band: surface_pressure), 'wind': ModelObstype_Vectorfield instance of wind (linked to bands: u_component_of_wind_10m and v_component_of_wind_10m), 'solar_rad_down_at_surface': ModelObstype instance of solar_rad_down_at_surface (linked to band: surface_solar_radiation_downwards)} + + """ super().__init__( obsname=obstype.name, std_unit=obstype.std_unit, @@ -63,7 +133,7 @@ def _check_validity(self): self.model_unit = self._get_std_unit_name(self.model_unit) def __str__(self): - return f"{type(self).__name__} isntance of {self.name} (linked to band: {self.model_band})" + return f"{type(self).__name__} instance of {self.name} (linked to band: {self.model_band})" def __repr__(self): return str(self) @@ -122,7 +192,7 @@ def _check_validity(self): ) def __str__(self): - return f"{type(self).__name__} isntance of {self.name} (linked to bands: {self.get_modelband_u()} and {self.get_modelband_v()})" + return f"{type(self).__name__} instance of {self.name} (linked to bands: {self.get_modelband_u()} and {self.get_modelband_v()})" def __repr__(self): return str(self)