EIDA NG Mediator/Federator webservices

Note

Disclaimer: This is an archived repository. Here, no further development takes place. Instead, development continues at https://github.com/EIDA/eidaws and https://github.com/EIDA/eidaws-federator-deployment.

(py35, py36)

This repository is intended to contain the source code for three of the web services of EIDA NG: (i) the federator, (ii) stationlite and (iii) the mediator.

Federator: Federate fdsnws-station, fdsnws-dataselect, and eidaws-wfcatalog requests across all EIDA nodes. This means, a user can issue a request against a federator endpoint without having to know where the data is hosted. In order to discover the information location the eidaws-stationlite web service is used.

StationLite: A lightweight fdsnws-station web service providing routing information. The stream epoch information is returned fully resolved (Stream epochs do not contain wildcard characters anymore.). The information location is harvested making use of eidaws-routing configuration files and fdsnws-station.

Mediator: This service allows queries across different web service domains, e.g., fdsnws-station and fdsnws-dataselect. Example: Retrieve waveform data for all stations within a lat-lon box.

Currently, we provide an alpha version of the federator service for fdsnws-station, fdsnws-dataselect and eidaws-wfcatalog.

Besides an alpha version of the stationlite service is implemented.

Content

• Installation
• Run the Test WSGI servers
• StationLite harvesting
• Logging (application level)
• Missing features and limitations

Installation

• Development
• Docker

Run the Test WSGI servers

The examples bellow use the built-in Flask server, which is not recommended to use in production. In production environments the usage of a WSGI server should be preferred. An exemplary setup with mod_wsgi and Apache2 is described in the section Deploying to a webserver. Alternatively use Gunicorn or uWSGI.

Federator server

To launch a local test WSGI server (NOT for production environments) enter:

(venv) $ eida-federator-test --tmpdir='/path/to/tmp'

For further configuration options invoke

(venv) $ eida-federator-test -h

The service currently writes temporary files to the tmpdir. Make sure that plenty of space is available.

StationLite server

To launch a local test WSGI server (NOT for production environments) enter:

(venv) $ eida-stationlite-test URL

URL is a database url as described at the SQLAlchemy documentation. For further configuration options invoke

(venv) $ eida-stationlite-test -h

Mediator server (deprecated)

Note

The EIDA Mediator webservice currently still does not provide an installation routine. However, a test server can be started as described bellow. Note, that you have to install all dependencies required manually.

Add the repository directory to your PYTHONPATH. Then, the server can be started as

$ python -m eidangservices.mediator.server --port=5001 --tmpdir='/path/to/tmp'

The server writes temporary files to the tmpdir, so this directory will fill up. It is recommended to purge this directory regularly, e.g., using a tool like tmpreaper.

StationLite harvesting

The stationlite webservice data is stored in a database which periodically must be harvested. This is done with eida-stationlite-harvest. By means of the eidaws-routing configuration files and the fdsnws-station webservice eida-stationlite-harvest collects and updates the database. Information on how to use eida-stationlite-harvest is available with

(venv) $ eida-stationlite-harvest -h

In addition the software suite contains an empty exemplary preconfigured SQLite database (db/stationlite.db.empty) which must be filled initially after installing the stationlite webservice. I.e.

(venv) $ cd $PATH_INSTALLATION_DIRECTORY/mediatorws/
(venv) $ cp -v db/stationlite.db.empty db/stationlite.db
(venv) $ eida-stationlite-harvest sqlite:///$(pwd)/db/stationlite.db

Note, that harvesting may take some time until completed.

Logging (application level)

Note

EIDA Federator and StationLite webservices only.

For debugging purposes EIDA NG webservices also provide logging facilities. Simply configure your webservice with a logging configuration file. Use the INI logging configuration file format. In case initialization failed a fallback SysLogHandler is set up:

fallback_handler = logging.handlers.SysLogHandler('/dev/log',
                                                  'local0')
fallback_handler.setLevel(logging.WARN)
fallback_formatter = logging.Formatter(
    fmt=("<XXX> %(asctime)s %(levelname)s %(name)s %(process)d "
         "%(filename)s:%(lineno)d - %(message)s"),
    datefmt="%Y-%m-%dT%H:%M:%S%z")
fallback_handler.setFormatter(fallback_formatter)

An exemplary logging configuration using a SysLogHandler is located at $PATH_INSTALLATION_DIRECTORY/mediatorws/config/syslog.conf. At $PATH_INSTALLATION_DIRECTORY/mediatorws/config/logging.config a StreamHandler is configured.

Note

1. In order to keep the WSGI application portable you should avoid setting up a logger writing to sys.stdout. See also: http://modwsgi.readthedocs.io/en/develop/user-guides/debugging-techniques.html

2. When using an EIDA NG multithreaded webservice together with a mod_wsgi configuration processes logging to a single file is not supported. Instead initialize your logger with a handler which guarantees log messages to be serialized (e.g. SysLogHandler, SocketHandler).

Missing features and limitations

• AAI of the fdsnws-dataselect service is not implemented yet
• stationlite currently implements the eidaws-routing interface only partly (e.g. format={post,get})
• For issues also visit https://github.com/EIDA/mediatorws/issues.