Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lot of changes #39

Merged
merged 239 commits into from
Mar 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
239 commits
Select commit Hold shift + click to select a range
95f69ae
Merge pull request #96 from legend-exp/main
sofia-calgaro Apr 18, 2023
89c59cf
added std for 'per channel' style
Apr 27, 2023
6c9dd08
fixing channels
sofia-calgaro Apr 27, 2023
a20a858
Merge pull request #35 from sofia-calgaro/main
sofia-calgaro Apr 27, 2023
252a579
style: pre-commit fixes
pre-commit-ci[bot] Apr 27, 2023
185bd95
fixed fwhm
Apr 27, 2023
8912554
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Apr 27, 2023
035141b
Merge pull request #36 from sofia-calgaro/main
sofia-calgaro Apr 27, 2023
4971cd8
style: pre-commit fixes
pre-commit-ci[bot] Apr 27, 2023
5c9b409
Update plotting.py
sofia-calgaro Apr 27, 2023
6f50802
Update plotting.py
sofia-calgaro Apr 27, 2023
fbe6673
Merge pull request #97 from sofia-calgaro/dev
sofia-calgaro Apr 28, 2023
c5678fd
Merge pull request #37 from sofia-calgaro/dev
sofia-calgaro Apr 28, 2023
d564d38
added FC_bsln and puls_aux
Apr 28, 2023
1418e8b
Merge branch 'main' of github.com:sofia-calgaro/legend-data-monitor i…
Apr 28, 2023
74ca07c
Merge pull request #38 from sofia-calgaro/main
sofia-calgaro Apr 28, 2023
85ea598
style: pre-commit fixes
pre-commit-ci[bot] Apr 28, 2023
312c39d
Merge pull request #98 from sofia-calgaro/dev
sofia-calgaro Apr 28, 2023
51ca6cc
added event rate and cut for keys for given chs
May 2, 2023
18aa86f
new config folder
May 2, 2023
d3c7e76
fixed exposure plotting modules
May 2, 2023
5297c4b
minor fixes
May 2, 2023
f4c7180
added single/multiple QCs + handling of empty chs/dfs
May 2, 2023
24abe59
update keys to remove
May 3, 2023
2f38b0e
fixed anti-QCs and multiple QCs
May 3, 2023
6492e9e
style: pre-commit fixes
pre-commit-ci[bot] May 3, 2023
718f2e5
remvoe redefinitions
May 3, 2023
f70f2d4
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 3, 2023
729c9e8
style: pre-commit fixes
pre-commit-ci[bot] May 3, 2023
3b5b4d0
Merge pull request #101 from sofia-calgaro/dev
sofia-calgaro May 3, 2023
dc3e92f
fixed exposure, polished stuff up
sagitta42 May 3, 2023
f3c0429
style: pre-commit fixes
pre-commit-ci[bot] May 3, 2023
ba00f78
cuts in AnData
sagitta42 May 4, 2023
25297c4
pre-commit fixes
sagitta42 May 4, 2023
3616dea
cuts in init
sagitta42 May 4, 2023
a846d96
multi-param plot
sagitta42 May 8, 2023
a428353
style: pre-commit fixes
pre-commit-ci[bot] May 8, 2023
7f29e7f
Merge pull request #102 from sagitta42/sofiadev
sofia-calgaro May 15, 2023
b699e5e
fixed cc4 and FWHM labels
May 15, 2023
5cf628e
style: pre-commit fixes
pre-commit-ci[bot] May 15, 2023
ee4d29d
fixed channel mean for append and 1 param
May 15, 2023
07a0b7a
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 15, 2023
39fc6ed
style: pre-commit fixes
pre-commit-ci[bot] May 15, 2023
d7efddf
Merge pull request #103 from sofia-calgaro/dev
sofia-calgaro May 15, 2023
4b8388a
added MUON01 channel
May 15, 2023
913c73a
style: pre-commit fixes
pre-commit-ci[bot] May 15, 2023
5a1ad7d
Merge pull request #104 from sofia-calgaro/dev
sofia-calgaro May 15, 2023
61293fa
added selection of FC bsln/muon events
May 15, 2023
d9ffcc1
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 15, 2023
033b2c6
style: pre-commit fixes
pre-commit-ci[bot] May 15, 2023
4d0c74e
small fixes
May 15, 2023
86fed52
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 15, 2023
ad3729a
style: pre-commit fixes
pre-commit-ci[bot] May 15, 2023
aa2feb8
Merge pull request #105 from sofia-calgaro/dev
sofia-calgaro May 15, 2023
e0d9f92
new module for retrieving SC params
May 15, 2023
bae4a2f
added time interval from config[dataset] for SC querying
May 16, 2023
48d3bea
updated the list of available SC params
May 16, 2023
03abe09
added docu for SC parameters
May 16, 2023
5d7ab42
small fixes
May 16, 2023
d95570c
style: pre-commit fixes
pre-commit-ci[bot] May 16, 2023
bb49c65
return df with units/limits
May 16, 2023
9bfd21c
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 16, 2023
aaf8ddb
style: pre-commit fixes
pre-commit-ci[bot] May 16, 2023
ce6bc8e
fixed missing module
May 16, 2023
d034d74
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 16, 2023
ef72976
style: pre-commit fixes
pre-commit-ci[bot] May 16, 2023
555723b
Merge pull request #106 from sofia-calgaro/dev
sofia-calgaro May 16, 2023
7ad6680
added diode vmon/imon
May 16, 2023
c8be849
fixed units
May 16, 2023
f3d8560
style: pre-commit fixes
pre-commit-ci[bot] May 16, 2023
9c68ad1
small fix
May 16, 2023
23de774
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 16, 2023
6b8dc04
style: pre-commit fixes
pre-commit-ci[bot] May 16, 2023
49703c4
Merge pull request #107 from sofia-calgaro/dev
sofia-calgaro May 16, 2023
60d01d9
update
sofia-calgaro May 16, 2023
5c14511
fixed saving='overwrite' for par1 vs par2
May 16, 2023
826715c
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 16, 2023
67c2a7b
style: pre-commit fixes
pre-commit-ci[bot] May 16, 2023
fc6aa87
fixed flake8
May 16, 2023
836b845
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 16, 2023
b9ac751
minor fixes in utils.py structure
May 17, 2023
8ed8eb3
Merge pull request #108 from sofia-calgaro/dev
sofia-calgaro May 17, 2023
33d70e1
fixed limits for multi-param case
May 17, 2023
1a76504
style: pre-commit fixes
pre-commit-ci[bot] May 17, 2023
085d426
enabled status plot for multi-params case
May 17, 2023
ae9dade
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 17, 2023
1664b7b
style: pre-commit fixes
pre-commit-ci[bot] May 17, 2023
2cf32ea
small fixes
May 17, 2023
1b5b581
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 17, 2023
2f82b4c
style: pre-commit fixes
pre-commit-ci[bot] May 17, 2023
1070c2a
Merge pull request #109 from sofia-calgaro/dev
sofia-calgaro May 17, 2023
296433c
added 'appenging' saving option for par1 vs par2 case
May 17, 2023
f27e878
style: pre-commit fixes
pre-commit-ci[bot] May 17, 2023
4f264ad
fixed appending of multiple params
May 17, 2023
39a5ad2
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 17, 2023
59f818f
style: pre-commit fixes
pre-commit-ci[bot] May 17, 2023
4aa9066
Merge pull request #110 from sofia-calgaro/dev
sofia-calgaro May 17, 2023
0a3b6ad
fixed saving=append with var==true and multi-params case
May 17, 2023
52c8004
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 17, 2023
44b4a44
style: pre-commit fixes
pre-commit-ci[bot] May 17, 2023
abc5614
big fixes for appending new data for 1 or >1 params
May 18, 2023
9c622ba
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 18, 2023
07b4027
style: pre-commit fixes
pre-commit-ci[bot] May 18, 2023
da98ae1
minor style fixes
May 18, 2023
a742d39
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 18, 2023
c51855f
fixed output plot info
May 19, 2023
d475a4a
style: pre-commit fixes
pre-commit-ci[bot] May 19, 2023
67cd2d0
Merge pull request #111 from sofia-calgaro/dev
sofia-calgaro May 19, 2023
451f7c4
fixed append for special parameters
May 19, 2023
eab1f01
style: pre-commit fixes
pre-commit-ci[bot] May 19, 2023
0cdf465
fixed duplication of channel means in appended obj
May 19, 2023
3e9aefa
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 19, 2023
b3ec925
style: pre-commit fixes
pre-commit-ci[bot] May 19, 2023
eeb7e7f
fixed flake 8 errors and missing K_events entry in plot dict
May 30, 2023
2983d07
style: pre-commit fixes
pre-commit-ci[bot] May 30, 2023
ed6838a
fixed mean values when merging old+new df
May 30, 2023
9cd21a5
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 30, 2023
383fb38
style: pre-commit fixes
pre-commit-ci[bot] May 30, 2023
81d39ca
new module for saving data
May 30, 2023
d478675
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 30, 2023
5353d8b
style: pre-commit fixes
pre-commit-ci[bot] May 30, 2023
d2afd69
Merge pull request #112 from sofia-calgaro/dev
sofia-calgaro May 30, 2023
86503ad
including plot of AUX for other subvsys
May 30, 2023
8679d7f
style: pre-commit fixes
pre-commit-ci[bot] May 30, 2023
6dba73e
added if checks
Jun 1, 2023
a4c6864
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 1, 2023
c049dd6
style: pre-commit fixes
pre-commit-ci[bot] Jun 1, 2023
4872512
added multi-params case
Jun 1, 2023
2984238
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 1, 2023
9fbef9b
style: pre-commit fixes
pre-commit-ci[bot] Jun 1, 2023
6e3cea9
inclusion + saving of aux channel
Jun 8, 2023
d953d11
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 8, 2023
b2ed215
style: pre-commit fixes
pre-commit-ci[bot] Jun 8, 2023
fcad1b3
fixed flake errors
Jun 8, 2023
598426c
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 8, 2023
d1eaa92
style: pre-commit fixes
pre-commit-ci[bot] Jun 8, 2023
2e0f0d6
fixed FC bsln selection (no pulser events coincidence)
Jun 8, 2023
2b23704
style: pre-commit fixes
pre-commit-ci[bot] Jun 8, 2023
f97f2bd
fixed bsln selection
Jun 14, 2023
a565fe2
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 14, 2023
3e1fb28
style: pre-commit fixes
pre-commit-ci[bot] Jun 14, 2023
dc84d16
Merge pull request #113 from sofia-calgaro/dev
sofia-calgaro Jun 15, 2023
87dfbb5
fixed 'append' for new hdf structure
Jun 15, 2023
b99bb53
style: pre-commit fixes
pre-commit-ci[bot] Jun 15, 2023
ba89ee6
Merge pull request #115 from sofia-calgaro/dev
sofia-calgaro Jun 15, 2023
eb3fc05
fixed bsln for FC_bsln subsystem + puls channel from map
Jun 16, 2023
b824393
fixed all selection
Jun 16, 2023
48de27a
Merge pull request #116 from sofia-calgaro/dev
sofia-calgaro Jun 20, 2023
38fa221
fixed append option and added user_bunch to bunch long time windows o…
Jun 20, 2023
b10339b
style: pre-commit fixes
pre-commit-ci[bot] Jun 20, 2023
17a9300
added if over aux_analysis being empty or not
Jun 20, 2023
672aab2
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 20, 2023
8217084
Merge pull request #117 from sofia-calgaro/dev
sofia-calgaro Jun 20, 2023
d26bb25
fixes to append + notebook for hdf files
Jun 23, 2023
a581576
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 23, 2023
1b47f55
fixed case where there is 'mean' within param's name
Jun 23, 2023
6f33c32
polished notebook
Jun 23, 2023
e9865fb
style: pre-commit fixes
pre-commit-ci[bot] Jun 23, 2023
75e118b
removed 'whis' from codespell check
Jun 23, 2023
0f586db
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 23, 2023
7a5f636
fixed auto-removed aux hdf files
Jun 23, 2023
dfbcb35
style: pre-commit fixes
pre-commit-ci[bot] Jun 23, 2023
5775e30
beautified the notebook
Jun 26, 2023
682121e
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 26, 2023
38974d6
style: pre-commit fixes
pre-commit-ci[bot] Jun 26, 2023
6c4d6d6
Merge pull request #118 from sofia-calgaro/dev
sofia-calgaro Jun 26, 2023
a61e898
added warnings for QCs
Jun 26, 2023
f2ec790
style: pre-commit fixes
pre-commit-ci[bot] Jun 26, 2023
1362e31
Merge pull request #119 from sofia-calgaro/dev
sofia-calgaro Jun 26, 2023
03c9d71
fixed QC in hit config json and data columns
Jun 26, 2023
f65c5ae
style: pre-commit fixes
pre-commit-ci[bot] Jun 26, 2023
e405da4
better warning
Jun 26, 2023
c70fbd4
style: pre-commit fixes
pre-commit-ci[bot] Jun 26, 2023
350c568
Merge pull request #120 from sofia-calgaro/dev
sofia-calgaro Jun 26, 2023
105c92c
added summary plots option
morellam Jun 27, 2023
8791c70
style: pre-commit fixes
pre-commit-ci[bot] Jun 27, 2023
100b1ba
fixed widgets for summary plots
morellam Jun 27, 2023
19bf43a
solved conflict
morellam Jun 27, 2023
c90d3e1
style: pre-commit fixes
pre-commit-ci[bot] Jun 27, 2023
1d44db0
added fix for automatic checks errors
morellam Jun 27, 2023
fb23ca0
Merge branch 'dev' of https://github.com/morellam/legend-data-monitor…
morellam Jun 27, 2023
20f127a
style: pre-commit fixes
pre-commit-ci[bot] Jun 27, 2023
b99666d
changed import of lh5 from pygama
morellam Jun 27, 2023
328be8c
Merge branch 'dev' of https://github.com/morellam/legend-data-monitor…
morellam Jun 27, 2023
c6dec90
Merge pull request #121 from morellam/dev
morellam Jun 28, 2023
f77f334
new SlowControl class + fixed dataset checks
Jun 28, 2023
201e388
added working slow control class
Jun 28, 2023
07ba425
style: pre-commit fixes
pre-commit-ci[bot] Jun 28, 2023
2c2234c
changed core scdb function name
Jun 28, 2023
72ba87f
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 28, 2023
3afba8e
no more global scdb object
Jun 28, 2023
abb7970
lowered cov thr - fix when going to main
Jun 28, 2023
22defb1
Merge pull request #122 from sofia-calgaro/dev
sofia-calgaro Jun 28, 2023
7a813db
fixed 2H shift on x-axis label
morellam Jul 3, 2023
d8cd52b
style: pre-commit fixes
pre-commit-ci[bot] Jul 3, 2023
72829c4
Merge pull request #123 from morellam/dev
morellam Jul 3, 2023
77e9f6b
changed name for SC module
Jul 4, 2023
72daabd
established automatic ssh tunnel to SC db
Jul 4, 2023
177f7a1
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jul 4, 2023
0ce60a7
fixed with pre-commit
Jul 4, 2023
8442a5c
Merge pull request #125 from sofia-calgaro/dev
sofia-calgaro Jul 4, 2023
7f6f7f1
fixed append bug for hdf when file is not there
Jul 4, 2023
eed335b
Merge pull request #126 from sofia-calgaro/dev
sofia-calgaro Jul 4, 2023
c623c48
squeezed auto_control function
Jul 4, 2023
dd00666
fixed add_config_entries function
Jul 4, 2023
7d6d7f6
fixed None limits when not available to False
Jul 4, 2023
57cdafe
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jul 4, 2023
274dc07
Merge pull request #127 from sofia-calgaro/dev
sofia-calgaro Jul 4, 2023
73986b7
SC pswd as input
Jul 6, 2023
10920f0
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jul 6, 2023
38f70ee
Merge pull request #128 from sofia-calgaro/dev
sofia-calgaro Jul 6, 2023
01663b7
folder for example scripts + auto prod script
Jul 6, 2023
52dc1fb
imported utils logger
Jul 6, 2023
029d5a0
Update README.md
sofia-calgaro Jul 6, 2023
3480934
style: pre-commit fixes
pre-commit-ci[bot] Jul 6, 2023
8b6a7d5
removed executable
Jul 6, 2023
4a0f1b2
Merge pull request #129 from sofia-calgaro/dev
sofia-calgaro Jul 6, 2023
c676261
added possibility of selectin multiple runs/periods
morellam Jul 6, 2023
e7cb5b7
Merge branch 'dev' of https://github.com/morellam/legend-data-monitor…
morellam Jul 6, 2023
3c9ac28
style: pre-commit fixes
pre-commit-ci[bot] Jul 6, 2023
e1b8613
Merge pull request #130 from morellam/dev
morellam Jul 6, 2023
0130a27
polished notebooks
Jul 6, 2023
42829cb
fixed map selection
Jul 14, 2023
9ddb2a5
style: pre-commit fixes
pre-commit-ci[bot] Jul 14, 2023
94c7b4f
Merge pull request #131 from sofia-calgaro/dev
sofia-calgaro Jul 14, 2023
50167bb
added port input to SC db retrievement
Aug 28, 2023
3c2866b
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Aug 28, 2023
4fb7e33
added version to maps
Aug 28, 2023
d941ffa
updated following SC db changes
Aug 28, 2023
77385be
style: pre-commit fixes
pre-commit-ci[bot] Aug 28, 2023
128fead
fix codespell
Aug 28, 2023
8b309c3
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Aug 28, 2023
a7f0e27
Merge pull request #137 from sofia-calgaro/dev
sofia-calgaro Aug 28, 2023
44ccbf4
autogeneration of gain mtg plots
Nov 20, 2023
e6cca58
new pulser tagging based on trapTmax
Nov 20, 2023
bcaa0f6
fixed docu
Mar 8, 2024
c71542b
changes for new prodenv
Mar 8, 2024
5e12f00
Merge branch 'legend-exp:dev' into dev
sofia-calgaro Mar 8, 2024
d673037
Merge branch 'main' into dev
Mar 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions attic/auto_prod/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
This basic example file can be used to automatically generate monitoring plots, based on new .lh5 dsp/hit files appearing in the production folders. Slow Control data are automatically retrieved from the database (you need to provide the port you are using to connect to the database together with the password you can find on Confluence).

You need to specify the period and run you want to analyze in the script. You can then run the code through

```console
$ python main_sync_code.py
```

The output text is saved in an output file called "output.log".

You can run this command as a cronejob. Run

```console
$ crontab -e
```

and add a new line of the type

```console
0 */6 * * * rm output.log && python main_syc_code.py >> output.log 2>&1
```

This will automatically look for new processed .lh5 files every 6 hours.
You need to specify all input and output folders within the script itself.
334 changes: 334 additions & 0 deletions attic/auto_prod/main_sync_code.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,334 @@
import os
import re
import json
import subprocess
from pathlib import Path
import monitoring
from legendmeta import LegendMetadata
lmeta = LegendMetadata()
from legend_data_monitor import utils

# paths
auto_dir_path = "/data2/public/prodenv/prod-blind/tmp/auto"
rsync_path = "/data1/users/calgaro/rsync-env/output/"

search_directory = f'{auto_dir_path}/generated/tier/dsp/phy'
def search_latest_folder(my_dir):
directories = [d for d in os.listdir(my_dir) if os.path.isdir(os.path.join(my_dir, d))]
directories.sort(key=lambda x: Path(my_dir, x).stat().st_ctime)
return directories[-1]

# Period to monitor
period = "p07" # search_latest_folder(search_directory)
# Run to monitor
search_directory = os.path.join(search_directory, period)
run = search_latest_folder(search_directory)

source_dir = f"{auto_dir_path}/generated/tier/hit/phy/{period}/{run}/"

# commands to run the container
cmd = "apptainer run"
arg = "/data2/public/prodenv/containers/legendexp_legend-base_latest.sif"
output_folder = "/data1/users/calgaro/prod-ref-v2" #"auto_prova"

# ===========================================================================================
# BEGINNING OF THE ANALYSIS
# ===========================================================================================
# Configs definition

# define slow control dict
scdb = {
"output": output_folder,
"dataset": {
"experiment": "L200",
"period": period,
"version": "",
"path": auto_dir_path,
"type": "phy",
"runs": int(run.split('r')[-1])
},
"saving": "overwrite",
"slow_control": {
"parameters": [
"DaqLeft-Temp1",
"DaqLeft-Temp2",
"DaqRight-Temp1",
"DaqRight-Temp2",
"RREiT",
"RRNTe",
"RRSTe",
"ZUL_T_RR"
]
}
}
with open(f"{rsync_path}auto_slow_control.json", "w") as f:
json.dump(scdb, f)

# define geds dict
my_config = {
"output": output_folder,
"dataset": {
"experiment": "L200",
"period": period,
"version": "",
"path": auto_dir_path,
"type": "phy",
"runs": int(run.split('r')[-1])
},
"saving": "append",
"subsystems": {
"geds": {
"Event rate in pulser events": {
"parameters": "event_rate",
"event_type": "pulser",
"plot_structure": "per string",
"resampled": "only",
"plot_style": "vs time",
"time_window": "20S"
},
"Event rate in FCbsln events": {
"parameters": "event_rate",
"event_type": "FCbsln",
"plot_structure": "per string",
"resampled": "only",
"plot_style": "vs time",
"time_window": "20S"
},
"Baselines (dsp/baseline) in pulser events": {
"parameters": "baseline",
"event_type": "pulser",
"plot_structure": "per string",
"resampled": "only",
"plot_style": "vs time",
"AUX_ratio": True,
"variation": True,
"time_window": "10T"
},
"Baselines (dsp/baseline) in FCbsln events": {
"parameters": "baseline",
"event_type": "FCbsln",
"plot_structure": "per string",
"resampled": "only",
"plot_style": "vs time",
"variation": True,
"time_window": "10T"
},
"Mean baselines (dsp/bl_mean) in pulser events": {
"parameters": "bl_mean",
"event_type": "pulser",
"plot_structure": "per string",
"resampled": "only",
"plot_style": "vs time",
"AUX_ratio": True,
"variation": True,
"time_window": "10T"
},
"Mean baselines (dsp/bl_mean) in FCbsln events": {
"parameters": "bl_mean",
"event_type": "FCbsln",
"plot_structure": "per string",
"resampled": "only",
"plot_style": "vs time",
"variation": True,
"time_window": "10T"
},
"Uncalibrated gain (dsp/cuspEmax) in pulser events": {
"parameters": "cuspEmax",
"event_type": "pulser",
"plot_structure": "per string",
"resampled": "only",
"plot_style": "vs time",
"AUX_ratio": True,
"variation": True,
"time_window": "10T"
},
"Uncalibrated gain (dsp/cuspEmax) in FCbsln events": {
"parameters": "cuspEmax",
"event_type": "FCbsln",
"plot_structure": "per string",
"resampled": "only",
"plot_style": "vs time",
"AUX_ratio": True,
"variation": True,
"time_window": "10T"
},
"Calibrated gain (hit/cuspEmax_ctc_cal) in pulser events": {
"parameters": "cuspEmax_ctc_cal",
"event_type": "pulser",
"plot_structure": "per string",
"resampled": "only",
"plot_style": "vs time",
"variation": True,
"time_window": "10T"
},
"Calibrated gain (hit/cuspEmax_ctc_cal) in FCbsln events": {
"parameters": "cuspEmax_ctc_cal",
"event_type": "FCbsln",
"plot_structure": "per string",
"resampled": "only",
"plot_style": "vs time",
"variation": True,
"time_window": "10T"
},
"Noise (dsp/bl_std) in pulser events": {
"parameters": "bl_std",
"event_type": "pulser",
"plot_structure": "per string",
"resampled": "only",
"plot_style": "vs time",
"AUX_ratio": True,
"variation": True,
"time_window": "10T"
},
"Noise (dsp/bl_std) in FCbsln events": {
"parameters": "bl_std",
"event_type": "FCbsln",
"plot_structure": "per string",
"resampled": "only",
"plot_style": "vs time",
"AUX_ratio": True,
"variation": True,
"time_window": "10T"
},
"A/E (from dsp) in pulser events": {
"parameters": "AoE_Custom",
"event_type": "pulser",
"plot_structure": "per string",
"resampled": "only",
"plot_style": "vs time",
"variation": True,
"time_window": "10T"
},
"A/E (from dsp) in FCbsln events": {
"parameters": "AoE_Custom",
"event_type": "FCbsln",
"plot_structure": "per string",
"resampled": "only",
"plot_style": "vs time",
"variation": True,
"time_window": "10T"
}
}
}
}
with open(f"{rsync_path}auto_config.json", "w") as f:
json.dump(my_config, f)

# ===========================================================================================
# Get not-analyzed files
# ===========================================================================================

# File to store the timestamp of the last check
timestamp_file = f'{rsync_path}last_checked_{period}_{run}.txt'

# Read the last checked timestamp
last_checked = None
if os.path.exists(timestamp_file):
with open(timestamp_file, 'r') as file:
last_checked = file.read().strip()

# Get the current timestamp
current_files = os.listdir(source_dir)
new_files = []

# Compare the timestamps of files and find new files
for file in current_files:
file_path = os.path.join(source_dir, file)
if last_checked is None or os.path.getmtime(file_path) > float(last_checked):
new_files.append(file)

# If new files are found, check if they are ok or not
if new_files:
pattern = r"\d+"
correct_files = []

for new_file in new_files:
matches = re.findall(pattern, new_file)
# get only files with correct ending (and discard the ones that are still under processing)
if len(matches) == 6:
correct_files.append(new_file)

new_files = correct_files

# ===========================================================================================
# Analyze not-analyzed files
# ===========================================================================================

# If new files are found, run the shell command
if new_files:
# Replace this command with your desired shell command
command = 'echo New files found: \033[91m{}\033[0m'.format(' '.join(new_files))
subprocess.run(command, shell=True)

# create the file containing the keys with correct format to be later used by legend-data-monitor (it must be created every time with the new keys; NOT APPEND)
utils.logger.debug("\nCreating the file containing the keys to inspect...")
with open(f'{rsync_path}new_keys.filekeylist', 'w') as f:
for new_file in new_files:
new_file = new_file.split('-tier')[0]
f.write(new_file + '\n')
utils.logger.debug("...done!")

# ...run the plot production
utils.logger.debug("\nRunning the generation of plots...")
config_file = f"{rsync_path}auto_config.json"
keys_file = f"{rsync_path}new_keys.filekeylist"

bash_command = f"{cmd} --cleanenv {arg} ~/.local/bin/legend-data-monitor user_rsync_prod --config {config_file} --keys {keys_file}"
utils.logger.debug(f"...running command \033[95m{bash_command}\033[0m")
subprocess.run(bash_command, shell=True)
utils.logger.debug("...done!")

# ===========================================================================================
# Analyze Slow Control data (for the full run - overwrite of previous info)
# ===========================================================================================
# run slow control data retrieving
utils.logger.debug("\nRetrieving Slow Control data...")
scdb_config_file = f"{rsync_path}auto_slow_control.json"

bash_command = f"{cmd} --cleanenv {arg} ~/.local/bin/legend-data-monitor user_scdb --config {scdb_config_file} --port 8282 --pswd THE_PASSWORD"
utils.logger.debug(f"...running command \033[92m{bash_command}\033[0m")
subprocess.run(bash_command, shell=True)
utils.logger.debug("...SC done!")

# Update the last checked timestamp
with open(timestamp_file, 'w') as file:
file.write(str(os.path.getmtime(max([os.path.join(source_dir, file) for file in current_files], key=os.path.getmtime))))

# ===========================================================================================
# Generate Static Plots (eg gain monitoring)
# ===========================================================================================

# create monitoring-plots folder
mtg_folder = os.path.join(output_folder, 'generated/mtg')
if not os.path.exists(mtg_folder):
os.makedirs(mtg_folder)
utils.logger.debug(f"Folder '{mtg_folder}' created.")
mtg_folder = os.path.join(mtg_folder, 'phy')
if not os.path.exists(mtg_folder):
os.makedirs(mtg_folder)
utils.logger.debug(f"Folder '{mtg_folder}' created.")

# define dataset depending on the (latest) monitored period/run
avail_runs = sorted(os.listdir(os.path.join(mtg_folder.replace('mtg', 'plt'), period)))
dataset = {
period: avail_runs
}
utils.logger.debug(f'This is the dataset: {dataset}')

# get first timestamp of first run of the given period
start_key = (sorted(os.listdir(os.path.join(search_directory, avail_runs[0])))[0]).split('-')[4]
meta = LegendMetadata("/data2/public/prodenv/prod-blind/tmp/auto/inputs/")
# get channel map
chmap = meta.channelmap(start_key)
# get string info
str_chns = {}
for string in range(13):
if string in [0, 6]: continue
channels = [f"ch{chmap[ged].daq.rawid}" for ged, dic in chmap.items() if dic["system"]=='geds' and dic["analysis"]["processable"]==True and dic["location"]["string"]==string]
if len(channels)>0:
str_chns[string] = channels

# get pulser monitoring plot for a full period
phy_mtg_data = mtg_folder.replace('mtg', 'plt')
if dataset[period] != []:
monitoring.stability(phy_mtg_data, mtg_folder, dataset, chmap, str_chns, 1, False)
Loading