Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lot of changes #39

Merged
merged 239 commits into from
Mar 8, 2024
Merged
Changes from 1 commit
Commits
Show all changes
239 commits
Select commit Hold shift + click to select a range
95f69ae
Merge pull request #96 from legend-exp/main
sofia-calgaro Apr 18, 2023
89c59cf
added std for 'per channel' style
Apr 27, 2023
6c9dd08
fixing channels
sofia-calgaro Apr 27, 2023
a20a858
Merge pull request #35 from sofia-calgaro/main
sofia-calgaro Apr 27, 2023
252a579
style: pre-commit fixes
pre-commit-ci[bot] Apr 27, 2023
185bd95
fixed fwhm
Apr 27, 2023
8912554
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Apr 27, 2023
035141b
Merge pull request #36 from sofia-calgaro/main
sofia-calgaro Apr 27, 2023
4971cd8
style: pre-commit fixes
pre-commit-ci[bot] Apr 27, 2023
5c9b409
Update plotting.py
sofia-calgaro Apr 27, 2023
6f50802
Update plotting.py
sofia-calgaro Apr 27, 2023
fbe6673
Merge pull request #97 from sofia-calgaro/dev
sofia-calgaro Apr 28, 2023
c5678fd
Merge pull request #37 from sofia-calgaro/dev
sofia-calgaro Apr 28, 2023
d564d38
added FC_bsln and puls_aux
Apr 28, 2023
1418e8b
Merge branch 'main' of github.com:sofia-calgaro/legend-data-monitor i…
Apr 28, 2023
74ca07c
Merge pull request #38 from sofia-calgaro/main
sofia-calgaro Apr 28, 2023
85ea598
style: pre-commit fixes
pre-commit-ci[bot] Apr 28, 2023
312c39d
Merge pull request #98 from sofia-calgaro/dev
sofia-calgaro Apr 28, 2023
51ca6cc
added event rate and cut for keys for given chs
May 2, 2023
18aa86f
new config folder
May 2, 2023
d3c7e76
fixed exposure plotting modules
May 2, 2023
5297c4b
minor fixes
May 2, 2023
f4c7180
added single/multiple QCs + handling of empty chs/dfs
May 2, 2023
24abe59
update keys to remove
May 3, 2023
2f38b0e
fixed anti-QCs and multiple QCs
May 3, 2023
6492e9e
style: pre-commit fixes
pre-commit-ci[bot] May 3, 2023
718f2e5
remvoe redefinitions
May 3, 2023
f70f2d4
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 3, 2023
729c9e8
style: pre-commit fixes
pre-commit-ci[bot] May 3, 2023
3b5b4d0
Merge pull request #101 from sofia-calgaro/dev
sofia-calgaro May 3, 2023
dc3e92f
fixed exposure, polished stuff up
sagitta42 May 3, 2023
f3c0429
style: pre-commit fixes
pre-commit-ci[bot] May 3, 2023
ba00f78
cuts in AnData
sagitta42 May 4, 2023
25297c4
pre-commit fixes
sagitta42 May 4, 2023
3616dea
cuts in init
sagitta42 May 4, 2023
a846d96
multi-param plot
sagitta42 May 8, 2023
a428353
style: pre-commit fixes
pre-commit-ci[bot] May 8, 2023
7f29e7f
Merge pull request #102 from sagitta42/sofiadev
sofia-calgaro May 15, 2023
b699e5e
fixed cc4 and FWHM labels
May 15, 2023
5cf628e
style: pre-commit fixes
pre-commit-ci[bot] May 15, 2023
ee4d29d
fixed channel mean for append and 1 param
May 15, 2023
07a0b7a
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 15, 2023
39fc6ed
style: pre-commit fixes
pre-commit-ci[bot] May 15, 2023
d7efddf
Merge pull request #103 from sofia-calgaro/dev
sofia-calgaro May 15, 2023
4b8388a
added MUON01 channel
May 15, 2023
913c73a
style: pre-commit fixes
pre-commit-ci[bot] May 15, 2023
5a1ad7d
Merge pull request #104 from sofia-calgaro/dev
sofia-calgaro May 15, 2023
61293fa
added selection of FC bsln/muon events
May 15, 2023
d9ffcc1
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 15, 2023
033b2c6
style: pre-commit fixes
pre-commit-ci[bot] May 15, 2023
4d0c74e
small fixes
May 15, 2023
86fed52
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 15, 2023
ad3729a
style: pre-commit fixes
pre-commit-ci[bot] May 15, 2023
aa2feb8
Merge pull request #105 from sofia-calgaro/dev
sofia-calgaro May 15, 2023
e0d9f92
new module for retrieving SC params
May 15, 2023
bae4a2f
added time interval from config[dataset] for SC querying
May 16, 2023
48d3bea
updated the list of available SC params
May 16, 2023
03abe09
added docu for SC parameters
May 16, 2023
5d7ab42
small fixes
May 16, 2023
d95570c
style: pre-commit fixes
pre-commit-ci[bot] May 16, 2023
bb49c65
return df with units/limits
May 16, 2023
9bfd21c
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 16, 2023
aaf8ddb
style: pre-commit fixes
pre-commit-ci[bot] May 16, 2023
ce6bc8e
fixed missing module
May 16, 2023
d034d74
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 16, 2023
ef72976
style: pre-commit fixes
pre-commit-ci[bot] May 16, 2023
555723b
Merge pull request #106 from sofia-calgaro/dev
sofia-calgaro May 16, 2023
7ad6680
added diode vmon/imon
May 16, 2023
c8be849
fixed units
May 16, 2023
f3d8560
style: pre-commit fixes
pre-commit-ci[bot] May 16, 2023
9c68ad1
small fix
May 16, 2023
23de774
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 16, 2023
6b8dc04
style: pre-commit fixes
pre-commit-ci[bot] May 16, 2023
49703c4
Merge pull request #107 from sofia-calgaro/dev
sofia-calgaro May 16, 2023
60d01d9
update
sofia-calgaro May 16, 2023
5c14511
fixed saving='overwrite' for par1 vs par2
May 16, 2023
826715c
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 16, 2023
67c2a7b
style: pre-commit fixes
pre-commit-ci[bot] May 16, 2023
fc6aa87
fixed flake8
May 16, 2023
836b845
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 16, 2023
b9ac751
minor fixes in utils.py structure
May 17, 2023
8ed8eb3
Merge pull request #108 from sofia-calgaro/dev
sofia-calgaro May 17, 2023
33d70e1
fixed limits for multi-param case
May 17, 2023
1a76504
style: pre-commit fixes
pre-commit-ci[bot] May 17, 2023
085d426
enabled status plot for multi-params case
May 17, 2023
ae9dade
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 17, 2023
1664b7b
style: pre-commit fixes
pre-commit-ci[bot] May 17, 2023
2cf32ea
small fixes
May 17, 2023
1b5b581
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 17, 2023
2f82b4c
style: pre-commit fixes
pre-commit-ci[bot] May 17, 2023
1070c2a
Merge pull request #109 from sofia-calgaro/dev
sofia-calgaro May 17, 2023
296433c
added 'appenging' saving option for par1 vs par2 case
May 17, 2023
f27e878
style: pre-commit fixes
pre-commit-ci[bot] May 17, 2023
4f264ad
fixed appending of multiple params
May 17, 2023
39a5ad2
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 17, 2023
59f818f
style: pre-commit fixes
pre-commit-ci[bot] May 17, 2023
4aa9066
Merge pull request #110 from sofia-calgaro/dev
sofia-calgaro May 17, 2023
0a3b6ad
fixed saving=append with var==true and multi-params case
May 17, 2023
52c8004
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 17, 2023
44b4a44
style: pre-commit fixes
pre-commit-ci[bot] May 17, 2023
abc5614
big fixes for appending new data for 1 or >1 params
May 18, 2023
9c622ba
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 18, 2023
07b4027
style: pre-commit fixes
pre-commit-ci[bot] May 18, 2023
da98ae1
minor style fixes
May 18, 2023
a742d39
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 18, 2023
c51855f
fixed output plot info
May 19, 2023
d475a4a
style: pre-commit fixes
pre-commit-ci[bot] May 19, 2023
67cd2d0
Merge pull request #111 from sofia-calgaro/dev
sofia-calgaro May 19, 2023
451f7c4
fixed append for special parameters
May 19, 2023
eab1f01
style: pre-commit fixes
pre-commit-ci[bot] May 19, 2023
0cdf465
fixed duplication of channel means in appended obj
May 19, 2023
3e9aefa
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 19, 2023
b3ec925
style: pre-commit fixes
pre-commit-ci[bot] May 19, 2023
eeb7e7f
fixed flake 8 errors and missing K_events entry in plot dict
May 30, 2023
2983d07
style: pre-commit fixes
pre-commit-ci[bot] May 30, 2023
ed6838a
fixed mean values when merging old+new df
May 30, 2023
9cd21a5
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 30, 2023
383fb38
style: pre-commit fixes
pre-commit-ci[bot] May 30, 2023
81d39ca
new module for saving data
May 30, 2023
d478675
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
May 30, 2023
5353d8b
style: pre-commit fixes
pre-commit-ci[bot] May 30, 2023
d2afd69
Merge pull request #112 from sofia-calgaro/dev
sofia-calgaro May 30, 2023
86503ad
including plot of AUX for other subvsys
May 30, 2023
8679d7f
style: pre-commit fixes
pre-commit-ci[bot] May 30, 2023
6dba73e
added if checks
Jun 1, 2023
a4c6864
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 1, 2023
c049dd6
style: pre-commit fixes
pre-commit-ci[bot] Jun 1, 2023
4872512
added multi-params case
Jun 1, 2023
2984238
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 1, 2023
9fbef9b
style: pre-commit fixes
pre-commit-ci[bot] Jun 1, 2023
6e3cea9
inclusion + saving of aux channel
Jun 8, 2023
d953d11
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 8, 2023
b2ed215
style: pre-commit fixes
pre-commit-ci[bot] Jun 8, 2023
fcad1b3
fixed flake errors
Jun 8, 2023
598426c
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 8, 2023
d1eaa92
style: pre-commit fixes
pre-commit-ci[bot] Jun 8, 2023
2e0f0d6
fixed FC bsln selection (no pulser events coincidence)
Jun 8, 2023
2b23704
style: pre-commit fixes
pre-commit-ci[bot] Jun 8, 2023
f97f2bd
fixed bsln selection
Jun 14, 2023
a565fe2
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 14, 2023
3e1fb28
style: pre-commit fixes
pre-commit-ci[bot] Jun 14, 2023
dc84d16
Merge pull request #113 from sofia-calgaro/dev
sofia-calgaro Jun 15, 2023
87dfbb5
fixed 'append' for new hdf structure
Jun 15, 2023
b99bb53
style: pre-commit fixes
pre-commit-ci[bot] Jun 15, 2023
ba89ee6
Merge pull request #115 from sofia-calgaro/dev
sofia-calgaro Jun 15, 2023
eb3fc05
fixed bsln for FC_bsln subsystem + puls channel from map
Jun 16, 2023
b824393
fixed all selection
Jun 16, 2023
48de27a
Merge pull request #116 from sofia-calgaro/dev
sofia-calgaro Jun 20, 2023
38fa221
fixed append option and added user_bunch to bunch long time windows o…
Jun 20, 2023
b10339b
style: pre-commit fixes
pre-commit-ci[bot] Jun 20, 2023
17a9300
added if over aux_analysis being empty or not
Jun 20, 2023
672aab2
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 20, 2023
8217084
Merge pull request #117 from sofia-calgaro/dev
sofia-calgaro Jun 20, 2023
d26bb25
fixes to append + notebook for hdf files
Jun 23, 2023
a581576
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 23, 2023
1b47f55
fixed case where there is 'mean' within param's name
Jun 23, 2023
6f33c32
polished notebook
Jun 23, 2023
e9865fb
style: pre-commit fixes
pre-commit-ci[bot] Jun 23, 2023
75e118b
removed 'whis' from codespell check
Jun 23, 2023
0f586db
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 23, 2023
7a5f636
fixed auto-removed aux hdf files
Jun 23, 2023
dfbcb35
style: pre-commit fixes
pre-commit-ci[bot] Jun 23, 2023
5775e30
beautified the notebook
Jun 26, 2023
682121e
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 26, 2023
38974d6
style: pre-commit fixes
pre-commit-ci[bot] Jun 26, 2023
6c4d6d6
Merge pull request #118 from sofia-calgaro/dev
sofia-calgaro Jun 26, 2023
a61e898
added warnings for QCs
Jun 26, 2023
f2ec790
style: pre-commit fixes
pre-commit-ci[bot] Jun 26, 2023
1362e31
Merge pull request #119 from sofia-calgaro/dev
sofia-calgaro Jun 26, 2023
03c9d71
fixed QC in hit config json and data columns
Jun 26, 2023
f65c5ae
style: pre-commit fixes
pre-commit-ci[bot] Jun 26, 2023
e405da4
better warning
Jun 26, 2023
c70fbd4
style: pre-commit fixes
pre-commit-ci[bot] Jun 26, 2023
350c568
Merge pull request #120 from sofia-calgaro/dev
sofia-calgaro Jun 26, 2023
105c92c
added summary plots option
morellam Jun 27, 2023
8791c70
style: pre-commit fixes
pre-commit-ci[bot] Jun 27, 2023
100b1ba
fixed widgets for summary plots
morellam Jun 27, 2023
19bf43a
solved conflict
morellam Jun 27, 2023
c90d3e1
style: pre-commit fixes
pre-commit-ci[bot] Jun 27, 2023
1d44db0
added fix for automatic checks errors
morellam Jun 27, 2023
fb23ca0
Merge branch 'dev' of https://github.com/morellam/legend-data-monitor…
morellam Jun 27, 2023
20f127a
style: pre-commit fixes
pre-commit-ci[bot] Jun 27, 2023
b99666d
changed import of lh5 from pygama
morellam Jun 27, 2023
328be8c
Merge branch 'dev' of https://github.com/morellam/legend-data-monitor…
morellam Jun 27, 2023
c6dec90
Merge pull request #121 from morellam/dev
morellam Jun 28, 2023
f77f334
new SlowControl class + fixed dataset checks
Jun 28, 2023
201e388
added working slow control class
Jun 28, 2023
07ba425
style: pre-commit fixes
pre-commit-ci[bot] Jun 28, 2023
2c2234c
changed core scdb function name
Jun 28, 2023
72ba87f
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jun 28, 2023
3afba8e
no more global scdb object
Jun 28, 2023
abb7970
lowered cov thr - fix when going to main
Jun 28, 2023
22defb1
Merge pull request #122 from sofia-calgaro/dev
sofia-calgaro Jun 28, 2023
7a813db
fixed 2H shift on x-axis label
morellam Jul 3, 2023
d8cd52b
style: pre-commit fixes
pre-commit-ci[bot] Jul 3, 2023
72829c4
Merge pull request #123 from morellam/dev
morellam Jul 3, 2023
77e9f6b
changed name for SC module
Jul 4, 2023
72daabd
established automatic ssh tunnel to SC db
Jul 4, 2023
177f7a1
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jul 4, 2023
0ce60a7
fixed with pre-commit
Jul 4, 2023
8442a5c
Merge pull request #125 from sofia-calgaro/dev
sofia-calgaro Jul 4, 2023
7f6f7f1
fixed append bug for hdf when file is not there
Jul 4, 2023
eed335b
Merge pull request #126 from sofia-calgaro/dev
sofia-calgaro Jul 4, 2023
c623c48
squeezed auto_control function
Jul 4, 2023
dd00666
fixed add_config_entries function
Jul 4, 2023
7d6d7f6
fixed None limits when not available to False
Jul 4, 2023
57cdafe
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jul 4, 2023
274dc07
Merge pull request #127 from sofia-calgaro/dev
sofia-calgaro Jul 4, 2023
73986b7
SC pswd as input
Jul 6, 2023
10920f0
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Jul 6, 2023
38f70ee
Merge pull request #128 from sofia-calgaro/dev
sofia-calgaro Jul 6, 2023
01663b7
folder for example scripts + auto prod script
Jul 6, 2023
52dc1fb
imported utils logger
Jul 6, 2023
029d5a0
Update README.md
sofia-calgaro Jul 6, 2023
3480934
style: pre-commit fixes
pre-commit-ci[bot] Jul 6, 2023
8b6a7d5
removed executable
Jul 6, 2023
4a0f1b2
Merge pull request #129 from sofia-calgaro/dev
sofia-calgaro Jul 6, 2023
c676261
added possibility of selectin multiple runs/periods
morellam Jul 6, 2023
e7cb5b7
Merge branch 'dev' of https://github.com/morellam/legend-data-monitor…
morellam Jul 6, 2023
3c9ac28
style: pre-commit fixes
pre-commit-ci[bot] Jul 6, 2023
e1b8613
Merge pull request #130 from morellam/dev
morellam Jul 6, 2023
0130a27
polished notebooks
Jul 6, 2023
42829cb
fixed map selection
Jul 14, 2023
9ddb2a5
style: pre-commit fixes
pre-commit-ci[bot] Jul 14, 2023
94c7b4f
Merge pull request #131 from sofia-calgaro/dev
sofia-calgaro Jul 14, 2023
50167bb
added port input to SC db retrievement
Aug 28, 2023
3c2866b
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Aug 28, 2023
4fb7e33
added version to maps
Aug 28, 2023
d941ffa
updated following SC db changes
Aug 28, 2023
77385be
style: pre-commit fixes
pre-commit-ci[bot] Aug 28, 2023
128fead
fix codespell
Aug 28, 2023
8b309c3
Merge branch 'dev' of github.com:sofia-calgaro/legend-data-monitor in…
Aug 28, 2023
a7f0e27
Merge pull request #137 from sofia-calgaro/dev
sofia-calgaro Aug 28, 2023
44ccbf4
autogeneration of gain mtg plots
Nov 20, 2023
e6cca58
new pulser tagging based on trapTmax
Nov 20, 2023
bcaa0f6
fixed docu
Mar 8, 2024
c71542b
changes for new prodenv
Mar 8, 2024
5e12f00
Merge branch 'legend-exp:dev' into dev
sofia-calgaro Mar 8, 2024
d673037
Merge branch 'main' into dev
Mar 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
changes for new prodenv
Sofia Calgaro committed Mar 8, 2024
commit c71542be807707d658e883a335ba328b5e93ab0d
12 changes: 10 additions & 2 deletions src/legend_data_monitor/analysis_data.py
Original file line number Diff line number Diff line change
@@ -213,8 +213,8 @@ def select_events(self):
utils.logger.info("... keeping only muon events")
self.data = self.data[self.data["flag_muon"]]
elif self.evt_type == "phy":
utils.logger.info("... keeping only physical (non-pulser) events")
self.data = self.data[~self.data["flag_pulser"]]
utils.logger.info("... keeping only physical (non-pulser & non-FCbsln & non-muon) events")
self.data = self.data[(~self.data["flag_pulser"]) | (~self.data["flag_fc_bsln"]) | (~self.data["flag_muon"])]
elif self.evt_type == "K_events":
utils.logger.info("... selecting K lines in physical (non-pulser) events")
self.data = self.data[~self.data["flag_pulser"]]
@@ -467,6 +467,11 @@ def channel_mean(self):
# we need to repeat this operation for each param, otherwise only the mean of the last one survives
self.data = concat_channel_mean(self, channel_mean)

if self.data.empty:
utils.logger.error(
f"\033[91mFor '{self.evt_type}' there are no flagged data (empty dataframe) -> no entries in the output file! Stop here the study.\033[0m"
)

def calculate_variation(self):
"""
Add a new column containing the percentage variation of a given parameter.
@@ -485,6 +490,9 @@ def calculate_variation(self):

def is_spms(self) -> bool:
"""Return True if 'location' (=fiber) and 'position' (=top, bottom) are strings."""
if self.data.empty:
return False

if isinstance(self.data.iloc[0]["location"], str) and isinstance(
self.data.iloc[0]["position"], str
):
5 changes: 5 additions & 0 deletions src/legend_data_monitor/slow_control.py
Original file line number Diff line number Diff line change
@@ -150,6 +150,11 @@ def get_sc_param(self):
get_table_df["lower_lim"] = lower_lim
get_table_df["upper_lim"] = upper_lim

# fix time column
get_table_df['tstamp'] = pd.to_datetime(get_table_df['tstamp'], utc=True)
# fix value column
get_table_df['value'] = pd.to_numeric(get_table_df['value'], errors='coerce') # handle errors as NaN

# remove unnecessary columns
remove_cols = ["rack", "group", "sensor", "name", "almask"]
for col in remove_cols:
135 changes: 111 additions & 24 deletions src/legend_data_monitor/subsystem.py
Original file line number Diff line number Diff line change
@@ -85,6 +85,12 @@ def __init__(self, sub_type: str, **kwargs):
self.path = data_info["path"]
self.version = data_info["version"]

# data stored under these folders have been partitioned!
if "tmp-auto" not in self.path:
self.partition = True
else:
self.partition = False

(
self.timerange,
self.first_timestamp,
@@ -180,14 +186,19 @@ def get_data(self, parameters: typing.Union[str, list_of_str, tuple_of_str] = ()
now = datetime.now()
self.data = dl.load()
utils.logger.info(f"Total time to load data: {(datetime.now() - now)}")

# -------------------------------------------------------------------------
# polish things up
# -------------------------------------------------------------------------

tier = "hit" if "hit" in dbconfig["columns"] else "dsp"
tier = "dsp"
if "hit" in dbconfig["columns"]:
tier = "hit"
if self.partition and "pht" in dbconfig["columns"]:
tier = "pht"
# remove columns we don't need
self.data = self.data.drop([f"{tier}_idx", "file"], axis=1)
if "{tier}_idx" in list(self.data.columns):
self.data = self.data.drop([f"{tier}_idx", "file"], axis=1)
# rename channel to channel
self.data = self.data.rename(columns={f"{tier}_table": "channel"})

@@ -357,15 +368,6 @@ def flag_pulser_events(self, pulser=None):

else:
# --- if no object was provided, it's understood that this itself is a pulser
# find timestamps over threshold
# if self.below_period_3_excluded():
# high_thr = 12500
# if self.above_period_3_included():
# high_thr = 2500
#high_thr = 12500
#self.data = self.data.set_index("datetime")
#wf_max_rel = self.data["wf_max"] - self.data["baseline"]
#pulser_timestamps = self.data[wf_max_rel > high_thr].index
trapTmax = self.data["trapTmax"]
pulser_timestamps = self.data[trapTmax > 200].index
# flag them
@@ -490,10 +492,10 @@ def get_channel_map(self):
# -------------------------------------------------------------------------

# for L60-p01 and L200-p02, keep using 'fcid' as channel
if int(self.period[-1]) < 3:
if int(self.period.split('p')[-1]) < 3:
ch_flag = "fcid"
# from L200-p03 included, uses 'rawid' as channel
if int(self.period[-1]) >= 3:
if int(self.period.split('p')[-1]) >= 3:
ch_flag = "rawid"

# dct_key is the subdict corresponding to one chmap entry
@@ -675,7 +677,7 @@ def get_channel_status(self):
if channel_name in self.channel_map.index:
self.channel_map.at[channel_name, "status"] = full_status_map[
channel_name
]["usability"] #if full_status_map[channel_name]["processable"] is not False else "off"
]["usability"]

# -------------------------------------------------------------------------
# quick-fix to remove detectors while status maps are not updated
@@ -723,12 +725,12 @@ def get_parameters_for_dataloader(self, parameters: typing.Union[str, list_of_st
# some parameters might be repeated twice - remove
return list(np.unique(params))


def construct_dataloader_configs(self, params: list_of_str):
"""
Construct DL and DB configs for DataLoader based on parameters and which tiers they belong to.

params: list of parameters to load
data_info: dict of containing type:, path:, version:
"""
# -------------------------------------------------------------------------
# which parameters belong to which tiers
@@ -740,6 +742,9 @@ def construct_dataloader_configs(self, params: list_of_str):
# ...
param_tiers = pd.DataFrame.from_dict(utils.PARAMETER_TIERS.items())
param_tiers.columns = ["param", "tier"]
# change from 'hit' to 'pht' when loading data for partitioned files
if self.partition:
param_tiers["tier"] = param_tiers["tier"].replace("hit", "pht")

# which of these are requested by user
param_tiers = param_tiers[param_tiers["param"].isin(params)]
@@ -764,18 +769,26 @@ def construct_dataloader_configs(self, params: list_of_str):
# set up tiers depending on what parameters we need
# -------------------------------------------------------------------------

# only load channels that are on (off channels will crash DataLoader)
chlist = list(self.channel_map[self.channel_map["status"] == "on"]["channel"])
# only load channels that are on or ac
chlist = list(self.channel_map[(self.channel_map["status"] == "on") | (self.channel_map["status"] == "ac")]["channel"])
# remove off channels
removed_chs = list(
self.channel_map[self.channel_map["status"] == "off"]["name"]
)
utils.logger.info(f"...... not loading channels with status off: {removed_chs}")
"""
# remove on channels that are not processable (ie have no hit entries)
removed_unprocessable_chs = list(
self.channel_map[self.channel_map["status"] == "on_not_process"]["name"]
)
utils.logger.info(f"...... not loading on channels that are not processable: {removed_unprocessable_chs}")
"""

# for L60-p01 and L200-p02, keep using 3 digits
if int(self.period[-1]) < 3:
if int(self.period.split('p')[-1]) < 3:
ch_format = "ch:03d"
# from L200-p03 included, uses 7 digits
if int(self.period[-1]) >= 3:
if int(self.period.split('p')[-1]) >= 3:
ch_format = "ch:07d"

# --- settings for each tier
@@ -791,7 +804,8 @@ def construct_dataloader_configs(self, params: list_of_str):
+ tier
+ ".lh5"
)
dict_dbconfig["table_format"][tier] = "ch{" + ch_format + "}/" + tier
dict_dbconfig["table_format"][tier] = "ch{" + ch_format + "}/"
dict_dbconfig["table_format"][tier] += "hit" if tier == "pht" else tier

dict_dbconfig["tables"][tier] = chlist

@@ -800,7 +814,7 @@ def construct_dataloader_configs(self, params: list_of_str):
# dict_dlconfig['levels'][tier] = {'tiers': [tier]}

# --- settings based on tier hierarchy
order = {"hit": 3, "dsp": 2, "raw": 1}
order = {"pht": 3, "dsp": 2, "raw": 1} if self.partition else {"hit": 3, "dsp": 2, "raw": 1}
param_tiers["order"] = param_tiers["tier"].apply(lambda x: order[x])
# find highest tier
max_tier = param_tiers[param_tiers["order"] == param_tiers["order"].max()][
@@ -813,6 +827,79 @@ def construct_dataloader_configs(self, params: list_of_str):

return dict_dlconfig, dict_dbconfig


def construct_dataloader_configs_unprocess(self, params: list_of_str):
"""
Construct DL and DB configs for DataLoader based on parameters and which tiers they belong to.

params: list of parameters to load
"""

param_tiers = pd.DataFrame.from_dict(utils.PARAMETER_TIERS.items())
param_tiers.columns = ["param", "tier"]

param_tiers = param_tiers[param_tiers["param"].isin(params)]
utils.logger.info("...... loading parameters from the following tiers:")
utils.logger.debug(param_tiers)

# -------------------------------------------------------------------------
# set up config templates
# -------------------------------------------------------------------------

dict_dbconfig = {
"data_dir": os.path.join(self.path, self.version, "generated", "tier"),
"tier_dirs": {},
"file_format": {},
"table_format": {},
"tables": {},
"columns": {},
}
dict_dlconfig = {"channel_map": {}, "levels": {}}

# -------------------------------------------------------------------------
# set up tiers depending on what parameters we need
# -------------------------------------------------------------------------

chlist = list(self.channel_map[(self.channel_map["status"] == "on_not_process") | (self.channel_map["status"] == "ac")]["channel"])
utils.logger.info(f"...... loading on channels that are not processable: {chlist}")

# for L60-p01 and L200-p02, keep using 3 digits
if int(self.period.split('p')[-1]) < 3:
ch_format = "ch:03d"
# from L200-p03 included, uses 7 digits
if int(self.period.split('p')[-1]) >= 3:
ch_format = "ch:07d"

for tier, tier_params in param_tiers.groupby("tier"):
dict_dbconfig["tier_dirs"][tier] = f"/{tier}"
dict_dbconfig["file_format"][tier] = (
"/{type}/"
+ self.period # {period}
+ "/{run}/{exp}-"
+ self.period # {period}
+ "-{run}-{type}-{timestamp}-tier_"
+ tier
+ ".lh5"
)
dict_dbconfig["table_format"][tier] = "ch{" + ch_format + "}/" + tier

dict_dbconfig["tables"][tier] = chlist

dict_dbconfig["columns"][tier] = list(tier_params["param"])

# --- settings based on tier hierarchy
order = {"hit": 3, "dsp": 2, "raw": 1}
param_tiers["order"] = param_tiers["tier"].apply(lambda x: order[x])
max_tier = param_tiers[param_tiers["order"] == param_tiers["order"].max()][
"tier"
].iloc[0]
dict_dlconfig["levels"][max_tier] = {
"tiers": list(param_tiers["tier"].unique())
}

return dict_dlconfig, dict_dbconfig


def remove_timestamps(self, remove_keys: dict):
"""Remove timestamps from the dataframes for a given channel.

@@ -846,13 +933,13 @@ def remove_timestamps(self, remove_keys: dict):
self.data = self.data.reset_index()

def below_period_3_excluded(self) -> bool:
if int(self.period[-1]) < 3:
if int(self.period.split('p')[-1]) < 3:
return True
else:
return False

def above_period_3_included(self) -> bool:
if int(self.period[-1]) >= 3:
if int(self.period.split('p')[-1]) >= 3:
return True
else:
return False