Technical Overview

Frontend

The frontend is composed of a minimal static site (index.html) which includes all the JavaScript files which take care of all the dynamic functionality via API calls to the backend.

Currently, it's served via nginx on production. During development, it can be served with frontend/server.py

Backend

The backend consists of:

• The Flask API server (backend/api.py)
• Scripts for background tasks:
  • dqm_extractor.py which calls the following scripts:
    • oms_extractor.py
    • rr_extractory.py

The scripts are managed by services. More info can be found here.

Scripts

dqm_extractor.py

• Read Configs, get Monitoring elements, setup config metrics
• Add new configs or update old config. We are not removing old configs (but they are not updated with new run information). Even if we update the metric or reference in config used for calculation, we do not change the existed Trends in DB.
• Finds all the GUI ROOT files on EOS. Filtering ROOT files based on version number. Ignore file if it is already in DB.
• Extracts MEs from files and updates Trends
• If run is not known: updates Run, OMS and RR data.

oms_extractor.py

Fetches OMS run attributes. Requires the following env variables:

• CLIENT_ID
• CLIENT_SECRET

to access OMS.

Those can be acquired by registering the application to the Application Portal, but the client has to be authorized to access OMS.

You can find the existing registration in the Application Portal under the identifier hdqm-api-client (you should have been granted access through the cms-dqm-coreteam DQM e-group).

If you need to re-create a registration, follow this procedure to whitelist the client to the OMS API.

rr_extractor.py

Fetches run information from Run Registry (class and whether the run is significant or not). Requires the following env variables:

• CLIENT_ID
• CLIENT_SECRET

to access Run Registry. Reuses the client id and secret used for accessing OMS.

Services

Service configuration files are located here: /etc/systemd/system/

hdqm.service

Takes care of starting the API server. It's running the run.sh script with the api argument.

hdqm-extract.service

Runs the dqm_extractor.py script. It is triggered by hdqm-extract.timer.

Follow the service logs:

sudo journalctl -u hdqm-extract -f

hdqm-extract.timer

Takes care of triggering the hdqm-extract service every night, in order to get updated data.

In order to stop the timer run:

sudo systemctl stop hdqm-extract.timer

To get info about a timer run one of these two commands:

sudo systemctl list-timers
sudo systemctl status hdqm-extract.timer