Csv2bufr templates and mosquitto config updates (#821)

* add documentation and volume for custom cvs2bufr-mapping-templates * remove accidental commit of local image used * add mosquitto changes to this branch for later * add documentation and volume for custom cvs2bufr-mapping-templates * remove accidental commit of local image used * websockets for mqtt * set owner of mosquitto config files * add mappings directory in create-config script * Update data-ingest.rst * Update public-services-setup.rst --------- Co-authored-by: Tom Kralidis <[email protected]>
wmo-im · Dec 19, 2024 · c079059 · c079059
1 parent 78969f9
commit c079059
Show file tree

Hide file tree

Showing 12 changed files with 110 additions and 55 deletions.
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -110,6 +110,8 @@ services:
       context: ./wis2box-broker
     env_file:
       - wis2box.env
+    volumes:
+      - mosquitto-config:/mosquitto/config
 
   wis2box-management:
     container_name: wis2box-management
@@ -160,3 +162,4 @@ volumes:
   minio-data:
   auth-data:
   htpasswd:
+  mosquitto-config:
diff --git a/docs/source/reference/running/data-pipeline-plugins.rst b/docs/source/reference/running/data-pipeline-plugins.rst
@@ -32,7 +32,21 @@ A typical csv2bufr plugin workflow definition would by defined as follows:
 
    csv:
        - plugin: wis2box.data.csv2bufr.ObservationDataCSV2BUFR
-         template: /data/wis2box/synop_bufr.json  # locally created csv2bufr mapping (located in $WIS2BOX_HOST_DATADIR)
+         template: aws-template # using one of the built-in templates
+         notify: true  # trigger GeoJSON publishing for API and UI
+         file-pattern: '^.*\.csv$'
+
+The default templates are defined by the `csv2bufr-templates`_ repository.
+
+In the case the user wants to use a custom template, the template should be located in the ``$WIS2BOX_HOST_DATADIR/mappings`` directory.
+
+The plugin configuration would then be defined as follows:
+
+.. code-block:: yaml
+
+   csv:
+       - plugin: wis2box.data.csv2bufr.ObservationDataCSV2BUFR
+         template: /data/wis2box/mappings/my_own_template.json  # locally created csv2bufr mapping (located in $WIS2BOX_HOST_DATADIR/mappings)
          notify: true  # trigger GeoJSON publishing for API and UI
          file-pattern: '^.*\.csv$'
 
@@ -146,5 +160,7 @@ For example, to publish GRIB2 data matching the file-pattern ``^.*_(\d{8})\d{2}.
 See :ref:`data-mappings` for a full example data mapping configuration.
 
 .. _`csv2bufr`: https://csv2bufr.readthedocs.io
+.. _`csv2bufr-templates`: https://github.com/wmo-im/csv2bufr-templates
 .. _`bufr2geojson`: https://github.com/wmo-im/bufr2geojson
 .. _`synop2bufr`: https://synop2bufr.readthedocs.io
+
diff --git a/docs/source/user/data-ingest.rst b/docs/source/user/data-ingest.rst
@@ -40,7 +40,7 @@ The wis2box provides 3 types of built-in plugins to publish data in BUFR format:
 
 * `bufr2bufr` : the input is received in BUFR format and split by subset, where each subset is published as a separate bufr message
 * `synop2bufr` : the input is received in `FM-12 SYNOP format <https://library.wmo.int/idviewer/35713/33>`_ and converted to BUFR format. The year and month are extracted from the file pattern
-* `csv2bufr` : the input is received in csv format and converted to BUFR format
+* `csv2bufr` : the input is received in CSV format and converted to BUFR format, a mapping template is used to convert the CSV columns to BUFR encoded values. Custom mapping templates need to be placed in the ``$WIS2BOX_HOST_DATADIR/mappings`` directory. See :ref:`csv2bufr-templates` for examples of mapping templates
 
 To publish data for other data formats you can use the 'Universal' plugin, which will pass through the data without any conversion.
 Please note that you will need to ensure that the date timestamp can be extracted from the file pattern when using this plugin.

diff --git a/docs/source/user/public-services-setup.rst b/docs/source/user/public-services-setup.rst
@@ -24,7 +24,7 @@ Please ensure that you follow these best practices to ensure your wis2box-instan
 
 The wis2box development team is not responsible for the security of your wis2box-instance and it is your responsibility to ensure that your wis2box instance is secure.
 
-GitHub issues and discussions provide a resourece and forum to discuss general wis2box features, bugs and updates.  For specific security related questions, please write to ``wis2-support at wmo.int``.
+GitHub issues and discussions provide a resource and forum to discuss general wis2box features, bugs and updates.  For specific security related questions, please write to ``wis2-support at wmo.int``.
 
 web-proxy (nginx)
 ^^^^^^^^^^^^^^^^^
@@ -39,6 +39,7 @@ wis2box runs a local nginx container allowing access to the following HTTP based
    UI (wis2box-ui),`WIS2BOX_URL/`
    Storage (incoming data) (minio:wis2box-incoming),`WIS2BOX_URL/wis2box-incoming`
    Storage (public data) (minio:wis2box-public),`WIS2BOX_URL/data`
+   Websockets (WIS2 notifications),`WIS2BOX_URL/mqtt`
 
 You can edit ``nginx/nginx.conf`` to control which services are exposed through the nginx-container include in your stack.
 
@@ -142,6 +143,19 @@ By exposing port 1883 on your host, the Global Broker will be able to subscribe
 
    The ``everyone`` user is defined by default for public readonly access (``origin/#``) as per WIS2 Node requirements.
 
+When you add SSL to your wis2box instance, the internal MQTT broker will be accessible on port 8883 on the host running wis2box using the MQTT over SSL protocol (MQTTS).
+
+The mosquitto service within wis2box also has websockets enabled and is proxied on '/mqtt' by the nginx container. 
+
+The broker address for the Global Broker to subscribe to WIS2 notifications using the mosquitto service within wis2box is as follows:
+
+- `mqtt://everyone:everyone@WIS2BOX_HOST:1883` - for MQTT without SSL
+- `mqtts://everyone:everyone@WIS2BOX_HOST:8883` - for MQTT with SSL
+- `ws://everyone:everyone@WIS2BOX_HOST/mqtt:80` - for MQTT over websockets without SSL
+- `wss://everyone:everyone@WIS2BOX_HOST/mqtt:443` - for MQTT over websockets with SSL
+
+Where ``WIS2BOX_HOST`` is the hostname or IP address of the host running wis2box.	
+
 External broker
 ---------------
 

diff --git a/nginx/nginx-ssl.conf b/nginx/nginx-ssl.conf
@@ -86,6 +86,13 @@
             proxy_set_header        Authorization $http_authorization;
             proxy_pass_header       Authorization;
         }
+        location /mqtt {
+            proxy_pass http://mosquitto:8884;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_set_header Host $host;
+        }
         location / {
             proxy_pass http://wis2box-ui:80;
         }

diff --git a/nginx/nginx.conf b/nginx/nginx.conf
@@ -83,6 +83,13 @@
             proxy_set_header        Authorization $http_authorization;
             proxy_pass_header       Authorization;
         }
+        location /mqtt {
+            proxy_pass http://mosquitto:8884;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_set_header Host $host;
+        }
         location / {
             proxy_pass http://wis2box-ui:80;
         }

diff --git a/wis2box-broker/Dockerfile b/wis2box-broker/Dockerfile
@@ -24,10 +24,8 @@ FROM eclipse-mosquitto:2.0.20
 RUN mkdir -p /data/wis2box/mosquitto
 RUN ln -s /mosquitto /data/wis2box/mosquitto
 
-COPY mosquitto-ssl.conf /mosquitto/config/mosquitto-ssl.conf
 COPY mosquitto.conf /mosquitto/config/mosquitto.conf
 
-COPY acl.conf /mosquitto/config/acl.conf
 COPY entrypoint.sh /docker-entrypoint.sh
 
 RUN chmod +x /docker-entrypoint.sh
diff --git a/wis2box-broker/acl.conf b/wis2box-broker/acl.conf
diff --git a/wis2box-broker/entrypoint.sh b/wis2box-broker/entrypoint.sh
@@ -1,28 +1,49 @@
 #!/bin/sh
 
-if [ -f /tmp/wis2box.crt ]; then
-    echo "SSL enabled"
-    echo "setup /mosquitto/certs"
-    mkdir -p /mosquitto/certs
-    cp /tmp/wis2box.crt /mosquitto/certs
-    cp /tmp/wis2box.key /mosquitto/certs
-    chown -R mosquitto:mosquitto /mosquitto/certs
-    cp -f /mosquitto/config/mosquitto-ssl.conf /mosquitto/config/mosquitto.conf
-else
-    echo "SSL disabled"
-fi
-
 echo "Setting mosquitto authentication"
 if [ ! -e "/mosquitto/config/password.txt" ]; then
     echo "Adding wis2box users to mosquitto password file"
     mosquitto_passwd -b -c /mosquitto/config/password.txt $WIS2BOX_BROKER_USERNAME $WIS2BOX_BROKER_PASSWORD
     mosquitto_passwd -b /mosquitto/config/password.txt everyone everyone
 else
-    echo "Mosquitto password file already exists. Skipping wis2box user addition."
+    echo "Mosquitto password file already exists. Update it if needed"
+    mosquitto_passwd -b /mosquitto/config/password.txt everyone everyone
+    mosquitto_passwd -b /mosquitto/config/password.txt $WIS2BOX_BROKER_USERNAME $WIS2BOX_BROKER_PASSWORD
 fi
 
-sed -i "s#_WIS2BOX_BROKER_QUEUE_MAX#$WIS2BOX_BROKER_QUEUE_MAX#" /mosquitto/config/mosquitto.conf
-sed -i "s#_WIS2BOX_BROKER_USERNAME#$WIS2BOX_BROKER_USERNAME#" /mosquitto/config/acl.conf
+# add max_queued_messages to mosquitto.conf if not already there
+if ! grep -q "max_queued_messages" /mosquitto/config/mosquitto.conf; then
+    echo "max_queued_messages $WIS2BOX_BROKER_QUEUE_MAX" >> /mosquitto/config/mosquitto.conf
+fi
+
+# prepare the acl.conf file
+if [ ! -e "/mosquitto/config/acl.conf" ]; then
+    echo "Creating mosquitto acl file"	
+    echo "user everyone" >> /mosquitto/config/acl.conf
+    echo "topic read origin/#" >> /mosquitto/config/acl.conf
+    echo " " >> /mosquitto/config/acl.conf
+    echo "user $WIS2BOX_BROKER_USERNAME" >> /mosquitto/config/acl.conf
+    echo "topic readwrite origin/#" >> /mosquitto/config/acl.conf
+    echo "topic readwrite wis2box/#" >> /mosquitto/config/acl.conf
+    echo "topic readwrite data-incoming/#" >> /mosquitto/config/acl.conf
+    echo "topic read \$SYS/#" >> /mosquitto/config/acl.conf
+else
+    echo "Mosquitto acl file already exists. Update it if needed"
+    # add user everyone to acl.conf if not already there
+    if ! grep -q "user everyone" /mosquitto/config/acl.conf; then
+        echo "user everyone" >> /mosquitto/config/acl.conf
+        echo "topic read origin/#" >> /mosquitto/config/acl.conf
+        echo " " >> /mosquitto/config/acl.conf
+    fi
+    # add user $WIS2BOX_BROKER_USERNAME to acl.conf if not already there
+    if ! grep -q "user $WIS2BOX_BROKER_USERNAME" /mosquitto/config/acl.conf; then
+        echo "user $WIS2BOX_BROKER_USERNAME" >> /mosquitto/config/acl.conf
+        echo "topic readwrite origin/#" >> /mosquitto/config/acl.conf
+        echo "topic readwrite wis2box/#" >> /mosquitto/config/acl.conf
+        echo "topic readwrite data-incoming/#" >> /mosquitto/config/acl.conf
+        echo "topic read \$SYS/#" >> /mosquitto/config/acl.conf
+    fi
+fi
 
 for i in `env | grep -Ee "\<WIS2BOX_BROKER_USERNAME_[[:alnum:]]+"`; do
     NAME_TAIL=`echo $i | awk -FWIS2BOX_BROKER_USERNAME_ '{print $2}' | awk -F= '{print $1}'`
@@ -35,10 +56,27 @@ for i in `env | grep -Ee "\<WIS2BOX_BROKER_USERNAME_[[:alnum:]]+"`; do
     echo "topic readwrite ${!topic}" >>  /mosquitto/config/acl.conf
 done
 
-# set ownership of mosquitto files
-chown -R mosquitto:mosquitto /mosquitto
+if [ -f /tmp/wis2box.crt ]; then
+    echo "SSL enabled"
+    echo "setup /mosquitto/certs"
+    mkdir -p /mosquitto/certs
+    cp /tmp/wis2box.crt /mosquitto/certs
+    cp /tmp/wis2box.key /mosquitto/certs
+    chown -R mosquitto:mosquitto /mosquitto/certs
+    # add listener 8883 block to mosquitto.conf, if not already there
+    if ! grep -q "listener 8883" /mosquitto/config/mosquitto.conf; then
+        echo "listener 8883" >> /mosquitto/config/mosquitto.conf
+        echo "certfile /mosquitto/certs/wis2box.crt" >> /mosquitto/config/mosquitto.conf
+        echo "keyfile /mosquitto/certs/wis2box.key" >> /mosquitto/config/mosquitto.conf
+    fi
+else
+    echo "SSL disabled"
+fi
 
 # set permission of acl.conf to 0700
 chmod 0700 /mosquitto/config/acl.conf
 
+# set owner of mosquitto config folder to mosquitto
+chown -R mosquitto:mosquitto /mosquitto/config
+
 /usr/sbin/mosquitto -c /mosquitto/config/mosquitto.conf
diff --git a/wis2box-broker/mosquitto-ssl.conf b/wis2box-broker/mosquitto-ssl.conf
diff --git a/wis2box-broker/mosquitto.conf b/wis2box-broker/mosquitto.conf
@@ -4,7 +4,6 @@ log_dest file /mosquitto/log/mosquitto.log
 log_dest stdout
 log_timestamp_format %Y-%m-%dT%H:%M:%S
 password_file /mosquitto/config/password.txt
-max_queued_messages _WIS2BOX_BROKER_QUEUE_MAX
 
 # ACLs
 acl_file /mosquitto/config/acl.conf

diff --git a/wis2box-create-config.py b/wis2box-create-config.py
@@ -437,6 +437,11 @@ def create_host_datadir() -> str:
                 exit()
         print(f"The directory {host_datadir} has been created.")
 
+        # add mappings directory
+        mappings_dir = host_datadir / 'mappings'
+        mappings_dir.mkdir(parents=True)
+
+        # add downloads directory
         download_dir = host_datadir / 'downloads'
         download_dir.mkdir(mode=0o775)
         if not WINDOWS: