Skip to content

Commit

Permalink
improve support for using custom csv2bufr template in wis2box (#854)
Browse files Browse the repository at this point in the history
* improve support for using custom csv2bufr template in wis2box

* add forgotten l

* rephrase intro
  • Loading branch information
maaikelimper authored Jan 28, 2025
1 parent ba4aef5 commit a42ab2a
Show file tree
Hide file tree
Showing 3 changed files with 42 additions and 8 deletions.
2 changes: 1 addition & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ services:
interval: 5s
retries: 100
volumes:
- ${WIS2BOX_HOST_DATADIR}/mappings:/data/wis2box/mappings:ro
- ${WIS2BOX_HOST_DATADIR}/mappings:/data/wis2box/mappings:rw

minio:
container_name: wis2box-minio
Expand Down
Binary file added docs/source/_static/csv2bufr_custom_template.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
48 changes: 41 additions & 7 deletions docs/source/reference/running/data-pipeline-plugins.rst
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,11 @@ list below describes each plugin and provides an example data mappings configura

This plugin converts CSV observation data into BUFR using `csv2bufr`_.

A `template` is required to convert the CSV columns to BUFR encoded values. See `csv2bufr-examples`_ on how to create a template or use one of the built-in templates.
A `template` is required to convert the CSV columns to BUFR encoded values, you can use the built-in templates or create your own custom template.

A ``file-pattern`` is used to filter on incoming data based on a regular expression.

A typical csv2bufr plugin workflow definition would by defined as follows:
A typical csv2bufr plugin workflow using one of the built-in templates would be defined as follows:

.. code-block:: yaml
Expand All @@ -39,11 +39,41 @@ A typical csv2bufr plugin workflow definition would by defined as follows:
notify: true # trigger GeoJSON publishing for API and UI
file-pattern: '^.*\.csv$'
The default templates are defined by the `csv2bufr-templates`_ repository.
In this the example `aws-template` refers to the `aws_example.json` template defined by the `csv2bufr-templates`_ repository.

To use a custom template, the template should be located in the ``$WIS2BOX_HOST_DATADIR/mappings`` directory and the `wis2box.env` file should include `CSV2BUFR_TEMPLATES=${WIS2BOX_DATADIR}/mappings`.
Environment variables can be set in `wis2box.env` to customize the behavior of the csv2bufr-plugin within the wis2box, see `csv2bufr-environment-variables`_ for the full list of environment variables.

Using the csv2bufr plugin with a custom template
++++++++++++++++++++++++++++++++++++++++++++++++

To create a custom template for the csv2bufr plugin in the wis2box, you can follow these steps:

Login to the wis2box-api container:

.. code-block:: bash
python3 wis2box-ctl.py login wis2box-api
Within the wis2box-api container you can using the command `csv2bufr mappings create` and provide the BUFR-descriptors you want to encode as arguments.
Make sure to redirect the output to a file in the `/data/wis2box/mappings` directory (which is mapped to the host system at `$WIS2BOX_HOST_DATADIR/mappings`)

For example, to create a custom template for the csv2bufr plugin with the BUFR-descriptors 301150, 301011, 301012, 301021, 007031, 302001, run the following command:

.. code-block:: bash
csv2bufr mappings create 301150 301011 301012 301021 007031 302001 --output /data/wis2box/mappings/my_own_template.json
See the `Manual on Codes`_ to find the BUFR-descriptors you need.

Exit the container after creating the custom template:

.. code-block:: bash
exit
You can edit the template file on the host system at `$WIS2BOX_HOST_DATADIR/mappings/my_own_template.json` to customize the template further.

The plugin configuration would then be defined as follows:
After creating the custom template, you can use it in the csv2bufr plugin configuration in MCF as follows:

.. code-block:: yaml
Expand All @@ -53,7 +83,11 @@ The plugin configuration would then be defined as follows:
notify: true # trigger GeoJSON publishing for API and UI
file-pattern: '^.*\.csv$'
Environment variables can be set in `wis2box.env` to customize the behavior of the csv2bufr-plugin within the wis2box, see `csv2bufr-environment-variables`_ for the full list of environment variables.
And/or you can select the new template in the Plugin Configuration of the Dataset Mappings Editor in the wis2box-webapp:

.. image:: ../_static/csv2bufr_custom_template.png
:alt: csv2bufr custom template
:align: center

``wis2box.data.bufr4.ObservationDataBUFR2GeoJSON``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down Expand Up @@ -170,9 +204,9 @@ For example, to publish GRIB2 data matching the file-pattern ``^.*_(\d{8})\d{2}.
See :ref:`data-mappings` for a full example data mapping configuration.

.. _`csv2bufr`: https://csv2bufr.readthedocs.io/en/v0.8.5/
.. _`csv2bufr-examples`: https://csv2bufr.readthedocs.io/en/v0.8.5/example.html
.. _`csv2bufr-environment-variables`: https://csv2bufr.readthedocs.io/en/v0.8.5/installation.html#environment-variables
.. _`csv2bufr-templates`: https://github.com/wmo-im/csv2bufr-templates
.. _`bufr2geojson`: https://github.com/wmo-im/bufr2geojson
.. _`synop2bufr`: https://synop2bufr.readthedocs.io

.. _`Manual on Codes`: https://library.wmo.int/records/item/35625-manual-on-codes-volume-i-2-international-codes

0 comments on commit a42ab2a

Please sign in to comment.