-
Notifications
You must be signed in to change notification settings - Fork 0
MTD DB INSTRUCTIONS
- Preliminary Infos
- Remote login on workstation HPZ4
- How to register LYSO Parts
- How to register LYSO Optical measurements
- How to register LYSO Mechanical measurements
- Read from OMS DB
- X2Go Client to read OMS DB
- Setting Proxy
- Forget about Kerberos
- Obsolete Instructions
Python 3.7 required
Code repository: git clone https://github.com/CMSROMA/MTDDB.git
User must be authorised (please contact [email protected] for that must declare you as belonging to the CERN e-group cms-mtd-db in https://e-groups.cern.ch/e-groups/)
- hpz4 dedicated machine (Mac PC in lab) with miniconda installed and the environment “mtddb” created/maintained)
+++++++++++++++++++++++++
hpz4 workstation INFO
+++++++++++++++++++++++++
* Conda environment installed
* two directories:
- MTDDB: here we have a copy of Giovanni's libraries Git repository
- mtdbbin: here we have the executable files: at the moment contains the script to register the parts (single crystals and arrays)
MTDDB code repository: git clone https://github.com/CMSROMA/MTDDB.git
Parts and optical measurements registration has been developed both for single crystals and arrays
If you are outside the CERN network you need to setup a proxy as described, e.g., here: https://security.web.cern.ch/recommendations/en/ssh_browsing.shtml
DB content can be browsed (tunnel is mandatory):
-
using DCA https://cmsdcadev.cern.ch/ (it's slow, be patient)
-
using OMS https://vocms0183.cern.ch/mtd/ (faster, best option to read from DB)
DCA Twiki Page: https://twiki.cern.ch/twiki/bin/view/Sandbox/DcaForDetectors
- Connect with INFN OpenVPN service
- Login to workstation
ssh -p3333 [email protected]
Whenever a new batch of LYSO arrives:
- Dowload .xlsx file from Rome-Tianle CERN box and
- Add a column named Extra to each sheet of .xlsx file
- copy it on workstation folder:0
**if working remotely**
scp -p 3333 <path-to-.xlsx-file> [email protected]:/home/cmsdaq/mtddbbin/Tianle-STP
- Run the script on workstation to produce .csv files
cd /home/cmsdaq/mtddbbin
conda activate mtddb
python3 Tianle-STP/produce-STP-csv.py -f Tianle-STP/<file-name>.xlsx -a # for LYSO arrays
python3 Tianle-STP/produce-STP-csv.py -f Tianle-STP/<file-name>.xlsx -b # for LYSO bars
- the script will produce .csv files with list of parts to be loaded in dir
/home/cmsdaq/mtddbbin/Tianle-STP/STP_LYSO_Parts/
- to upload LYSO parts on OMS DB launch the script:
for Arrays:
python3 registerMTDPart.py -f /home/cmsdaq/mtddbbin/Tianle-STP/STP_LYSO_Parts/<batch>_STP_LYSO_Arrays.csv -o <batch>_STP_LYSO_Arrays.xml -w -T mtdloadb
for Bars:
python3 registerMTDPart.py -f /home/cmsdaq/mtddbbin/Tianle-STP/STP_LYSO_Parts/<batch>_STP_LYSO_Bars.csv -o <batch>_STP_LYSO_Bars.xml -w -T mtdloadb
🔴 EXTRA INFO: .csv files containing producer measurements of array/bar dimensions and optical properties are created in the corresponding directory for the upload. **Further instruction in **
🔹 How to register LYSO Optical measurements
🔹 How to register LYSO Mechanical measurements
- Log into hpz4 (alias workstation, alias 10.0.0.43)
conda activate barlist_env
cd /home/cmsdaq/mtddbbin
python3 select_bars_for_irradiation.py -f Tianle-STP/Original-files/Quality-document-of-the-X-batch-of-Production-units-of-LYSO-for-MTD_20240129.xlsx
- output lists are in /home/cmsdaq/mtddbbin/sortedBarLists/
- N.B.: change X in the filename according to batch number on Tianle file name (1st, 2nd, 3rd, etc.)
After registration of all new parts on OMS DB, you must update the list of parts on AirTable BEFORE any measurement of the part properties. From the workstation (HPZ4):
cd /home/cmsdaq/AirTableTools
./registerNewPartsFromCDB.sh
The synchronization is automatically done every night at 23:59, so you may skip this step if you don't want to measure the new LYSO parts the same day.
If during the visual inspection some LYSO Parts seem damaged, have some scratches or something else you will have to take note of this:
1. cd /home/cmsdaq/MTDDB
2. python3 visualInspection.py -x <barcode> -c ''comment''
Optical measurements of LYSO parts properties are automatically uploaded on OMS DB every night.
The summary of the measurement of the day is stored in a .csv file named after the day the measurements were taken (format: YYYY_MM_DD.csv).
The uploaded files are stored on the workstation in:
/home/cmsdaq/MTDDB/uploader/files_uploaded/udoodaq01 (Absolute LY and DT of LYSO bars - PMT bench)
/home/cmsdaq/MTDDB/uploader/files_uploaded/pccmsdaq01 (Relative LY,XT,sigmaT of LYSO arrays grease-coupled with SiPMs - Single Array bench)
/home/cmsdaq/MTDDB/uploader/files_uploaded/pccmsdaq03 (Relative LY,XT,sigmaT of LYSO arrays dry-coupled with SiPMs - Array bench)
If the automatic upload did not work, un-sended files should be saved in the files_to_upload directory of the workstation:
/home/cmsdaq/MTDDB/uploader/files_to_upload/udoodaq01 (Absolute LY and DT of LYSO bars - PMT bench)
/home/cmsdaq/MTDDB/uploader/files_to_upload/pccmsdaq01 (Relative LY,XT,sigmaT of LYSO arrays grease-coupled with SiPMs - Single Array bench)
/home/cmsdaq/MTDDB/uploader/files_to_upload/pccmsdaq03 (Relative LY,XT,sigmaT of LYSO arrays dry-coupled with SiPMs - Array bench)
If there are files in any of these directories, you can manually launch the upload running the corresponding script:
cd /home/cmsdaq/MTDDB/uploader/
python3 upload-udoodaq01.py (to upload PMT bench measurements)
python3 upload-pccmsdaq01.py (to upload Single Array bench measurements)
python3 upload-pccmsdaq03.py (to upload Array bench measurements)
Measurements of LYSO parts dimensions are automatically uploaded on OMS DB every night.
The summary of the measurement of the day is stored in a .csv file named after the day the measurements were taken (format: YYYY_MM_DD.csv).
The uploaded files are stored on the workstation in:
/home/cmsdaq/MTDDB/uploader/files_uploaded/galaxy-bars (Measurements of LYSO bars dimensions)
/home/cmsdaq/MTDDB/uploader/files_uploaded/galaxy-arrays (Measurements of LYSO arrays dimensions)
If the automatic upload did not work, un-sended files should be saved in the files_to_upload directory of the workstation:
/home/cmsdaq/MTDDB/uploader/files_to_upload/galaxy-bars (Measurements of LYSO bars dimensions)
/home/cmsdaq/MTDDB/uploader/files_to_upload/galaxy-arrays (Measurements of LYSO arrays dimensions)
If there are files in any of these directories, you can manually launch the upload running the corresponding script:
cd /home/cmsdaq/MTDDB/uploader/
python3 upload-galaxy-bars.py (to upload measurmements of LYSO bar dimensions)
python3 upload-galaxy-arrays.py (to upload measurmements of LYSO array dimensions)
To get arrays and single crystal bars information:
From Lab workstation:
1️⃣ Open X2Go and select the session: (already installed on workstation with proxy properly set)
- click on hpz4 small window on right side
- select session (mtd)
- click on resume (to close X2Go session click on X on the top right corner but NEVER click on terminate)
- you will see the display of hpz4 machine
2️⃣ from a terminal open a tunnel to CERN
ssh -D 8080 lxtunnel.cern.ch -N -l <your CERN User>
3️⃣ Open Firefox to navigate.
4️⃣ Go to OMS MTD section web page (https://vocms0183.cern.ch/mtd):
From your PC:
1️⃣ from a terminal open a tunnel to CERN
ssh -D 8080 lxtunnel.cern.ch -N -l <your CERN User>
2️⃣ Edit proxy settings of your system (MAC user) or browser (others) as shown in [Setting Proxy](#user-content-how-setup-proxy-to-read-db)
3️⃣ Go to OMS MTD section web page (https://vocms0183.cern.ch/mtd):
This part is to install X2Go on your personal pc with proper setting
1️⃣ Download X2Go Client X2Go
2️⃣ Setting
- Session name: HPZ4
- Host: pccmsdaq01.roma1.infn.it
- Login: cmsdaq
- SSH Port: 3333
- Session Type: XFCE
3️⃣ Open X2Go and select the session:
- The session to resume is the number 52 (mtd)
- click on resume (to close X2Go session click on suspend NEVER on terminate)
4️⃣ you will see the display of hpz4 machine and from here you can navigate using Firefox
This part is to setup Proxy in your browser
You need this to access DCA/OMS to read DB
1️⃣ ssh -D 8080 lxtunnel.cern.ch -N -l <your CERN User>
2️⃣ macOS setting
- Open the Network Preferences ("System Preferences" -> "Network")
- Select your Interface which you are using and click on "Advanced..."
- Go to Proxy
- Select "SOCKS Proxy"
- Enter localhost and 8080 in
- Click "OK"
3️⃣ Configuring Firefox
Set the proxy in the preferences:
1️⃣ Go to about: preferences
2️⃣ Search for "Proxy"
3️⃣ Click on "Settings..."
- Select "Manual proxy configuration"
- Insert localhost for "SOCKS Host" and 8080 for "Port"
- Select "SOCKS v5"
- Select "Proxy DNS when using SOCKS v5"
- Click "OK"
- or install an extension for proxy management and set a new socks5 proxy with localhost as host and 8080 as port.
Alternatively you might consider possible extensions like "FoxyProxy Standard"
Download the CERN.CH realm configuration for Kerberos from here or from any lxplus machine and copy it all over!
$ sudo cp -f /etc/krb5.conf /etc/krb5.conf.ori
$ sudo cp -f ./krb5.conf /etc/krb5.conf
You need to generate the keytab for your [email protected] in your home folder before proceeding your adventure.
$ ktutil
ktutil: addent -password -p [email protected] -k 1 -e aes256-cts
ktutil: addent -password -p [email protected] -k 1 -e arcfour-hmac-md5
ktutil: wkt .keytab
ktutil: q
Check your file:
$ ktutil
ktutil: rkt .keytab
ktutil: list
You should see something like this:
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
1 1 [email protected]
2 1 [email protected]
Test if your keytab works. If you get no error message it means that everything is OK.
$ kinit -kt .keytab username
Now, this is the final step. We want the Kerberos token to be renewed automatically and the authentication done completely under the hood.
You need to place this command in a script that will get executed at login or by a cronjob everyday under your user:
k5start -b -f ~/.keytab -K 10 -l 7d -r CERN.CH -u username
And with this folks you're good to go!
You can insert two types of LYSO parts: bars and arrays. If you have multiple parts of the same type, you can upload them one-by-one or all at once. Below are listed few examples explaining how to do:
(example: barcode 1 producer 1 type 3 batch TestBatch and comment '1Fake array just for test')
From Workstation (HPZ4):
cd /home/cmsdaq/mtddbbin
python3 registerMTDPart.py -b TestBatch -x 1 -p 8 -t 'LYSOMatrix #3' -l Roma -o 1Fake_Array.xml -c '1Fake array just for test' -d 'pdata info' -w -T mtdloadb
[switch -b -x, -p, -t, -l, -o, -u, -c, -d, -w, -i (this to write in int2r test DB)]
- At the end of the session an automatic check of all barcodes sent is done reading data from DB
- In case of SUCCESS you will get this output message:
File uploaded...waiting for completion...done ed...waiting for completion...
.....! SUCCESS
done
=== terminating session...
Exit request sent.
Operation summary:
PRE0000000017: Success
BEFORE taking measurements for new parts registered on OMS DB, you should first SYNC AIRTABLE part list (see SYNC PART LIST ON AIRTABLE)
Convention used to define crystal type
LYSOMatrix #1 for LYSO matrices of type 1 (the thickest ones: 3.75 mm nominal thickness)
LYSOMatrix #2 for LYSO matrices of type 2 (the intermediated ones: 3.00 mm nominal thickness)
LYSOMatrix #3 for LYSO matrices of type 3 (the thinnest ones: 2.40 mm nominal thickness)
singleCrystal #1 for single bars 3.75 mm thick
singleCrystal #2 for single bars 3.00 mm thick
singleCrystal #3 for single bars 2.40 mm thick
Help for all available switch
python 3 registerMTDPart.py -h
./registerMTDPart.py
h (help): shows this help
b (batch=<value>): specify the batch to which the part belongs
x (barcode=<value>): specify the barcode of the part
s (serial=<value>): specify the serial number of the part
p (producer=<value>): specify the producer
t (type=<value>): specify the part type
l (lab=<value>): specify the laboratory in which the registration is done
f (file=<value>): the filename of a CSV file formatted such as the barcodes are under the
"Barcode" column while the type is given in a column named "Type".
The producer is expected in column "Producer". An optional column
"Serial Number" is expected to contain the corresponding information.
It can be left blank.
o (output=<value>): the filename of the XML output file
n (n=<value>): the number of barcodes to generate
w (write): upload the XML file automatically at the end of the processing (requires --file)
u (user=<value>): the CERN username authorised to permanently write data to DB
(default to current username)
d (data=<value>): producer provided data to be associated to the part
c (comment=<value>): operator comments
D (debug): activate debugging mode
i (int2r): use test database int2r
(example: starting from barcode 250)
python3 registerMTDPart.py -b TestBatch -x 250 -p 8 -t 'LYSOMatrix #2' -l Roma -o 20Fake_Arrays.xml -n 20 -c '20Fake array just for test' -d 'pdata info' -w -T mtdloadb
[switch -b -x, -p, -t, -l, -o, -n, -u, -c, -d, -w, -i (this to write in int2r test DB)]
BEFORE taking measurements for new parts registered on OMS DB, you should first SYNC AIRTABLE part list (see SYNC PART LIST ON AIRTABLE)
switch -s for Single Crystals (Bars)
python3 registerMTDPart.py -b TestBatch -x 4 -p 8 -t 'singleCrystal #1' -l Roma -o 1SingleXtalBar1.xml -c '1 Fake Xtal Bar just for test purpose' -d 'pdata info' -w -T mtdloadb
[switch -b -x, -p, -t, -l, -o, -n, -u, -c, -d, -w, -i (this to write in int2r test DB)]
BEFORE taking measurements for new parts registered on OMS DB, you should first SYNC AIRTABLE part list (see SYNC PART LIST ON AIRTABLE)
(example: starting from barcode 250) switch -s for Single Crystals (Bars)
python3 registerMTDPart.py -b TestBatch -x 250 -p 8 -t 'singleCrystal #2' -l Roma -o 20SingleXtalBars.xml -n 20 -c '20 Fake Xtal Bars just for test purpose' -d 'pdata info' -w -T mtdloadb
[switch -b -x, -p, -t, -l, -o, -n, -u, -c, -d, -w, -i (this to write in int2r test DB)]
BEFORE taking measurements for new parts registered on OMS DB, you should first SYNC AIRTABLE part list (see SYNC PART LIST ON AIRTABLE)
Create csv file for LYSO Parts
+++++++++++++++++++++++++++++++++++
csv file for LYSO parts: Array&Bar
+++++++++++++++++++++++++++++++++++
python3 CSV_To_Register_LYSO_Parts.py
[Output: LYSO_Parts_2022_09_30_14_45_16.csv]
Here is how the script behaves: the csv file will be create according to the given input
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
This script creates csv file to register LYSO Parts in MTDDB
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
-------------------------------------------------------------------------
********* BEFORE TO START **********
-------------------------------------------------------------------------
LYSO Part Type 1 = 3,75
LYSO Part Type 2 = 3,00
LYSO Part Type 3 = 2,40
LYSOMatrix #1 for LYSO matrices of type 1 (3.75 mm nominal thickness)
LYSOMatrix #2 for LYSO matrices of type 2 (3.00 mm nominal thickness)
LYSOMatrix #3 for LYSO matrices of type 3 (2.40 mm nominal thickness)
singleCrystal #1 for single bars 3.75 mm thick
singleCrystal #2 for single bars 3.00 mm thick
singleCrystal #3 for single bars 2.40 mm thick
-------------------------------------------------------------------------
pdata: fill with any information (tag/label/number) provided by vendor
serial number: this will be used during the production phase with same
info as pdata (at the moment leave it empty
-------------------------------------------------------------------------
How many LYSO Parts to register: 2
------> LYSO Part #: 1
Producer: 10
Type: LYSOMatrix #1
Barcode: 100
pdata - any number in tag/label/paper provided by vendor:
Serial number (to use in production phase for now leave it empty):
Comments: tets
Location: Roma
------> LYSO Part #: 2
Producer: 10
Type: singleCrystal #2
Barcode: 101
pdata - any number in tag/label/paper provided by vendor: test
Serial number (to use in production phase for now leave it empty):
Comments: test
Location: Roma
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
---- csv file to register LYSO Parts in MTDDB has been created: Bye Bye !! ----
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
Insert LYSO Part in DB
python3 registerMTDPart.py -b TestBatch -f LYSO_Parts_2022_10_21_16_23_16.csv -o 2LYSO_Parts.xml -l Roma -u <your CERN User> -w -i
On hpz4 this is done automatically for user mtdloadb at midnight, so you don't need to EVER renew a token or input a password! In order to ship data to OMS DB, you need to setup a tunnel as follows:
-
Open a terminal and login on lxplus
ssh -Y <your-cern-username>@lxplus.cern.ch -
From the same lxplus terminal open a tunnel to dbloader
ssh -L 50022:dbloader-mtd.cern.ch:22 <your-cern-username>@lxplus.cern.ch
* Using the switch -w (=write) (this is not included in the examples so you can test them in a safe mode without writing on DB) the XML file is automatically generated&transfered to dbloader to be loaded on DB
* The script try to transfer the first file: if it fails is because the tunnel is not open so it opens it and try with another attempt
* This is the reason why it could ask the pwd three times (the pwd is the one you are using at CERN):
1. to open the tunnel
2. to start the transfer session
3. to terminate the transfer session
* The switch -u sets the user who is doing the operation: the user has to be your same CERN username and it's needed:
1. to register the operation you are doing
2. to open the tunnel
python3 registerMTDPart.py -b TestBatch -x 1 -p 8 -t 'LYSOMatrix #3' -l Roma -o 1Fake_Array.xml -u <your CERN User> -c '1Fake array just for test' -w -i
[switch -b -x, -p, -t, -l, -o, -u, -c, -d, -w, -i (this to write in int2r test DB)]
python3 registerMTDPart.py -b TestBatch -x 250 -p 8 -t 'LYSOMatrix #2' -l Roma -o 20Fake_Arrays.xml -n 20 -u <your CERN User> -c '20 Fake arrays for test purpose' -w -i
[switch -b -x, -p, -t, -l, -o, -n, -u, -c, -d, -w, -i (this to write in int2r test DB)]
python3 registerMTDPart.py -b TestBatch -x 4 -p 8 -t 'singleCrystal #1' -l Roma -o 1SingleXtalBar1.xml -u <your CERN User> -c '1 Fake Xtal Bar just to test purpose' -w -i
[switch -b -x, -p, -t, -l, -o, -n, -u, -c, -d, -w, -i (this to write in int2r test DB)]
``
## REGISTER 20 BARS WITH CONSECUTIVE BARCODES ON TEST DB
python3 registerMTDPart.py -b TestBatch -x 250 -p 8 -t 'singleCrystal #2' -l Roma -o 20SingleXtalBars.xml -n 20 -u -c '20 Fake Xtal Bars for test' -w -i [switch -b -x, -p, -t, -l, -o, -n, -u, -c, -d, -w, -i (this to write in int2r test DB)]
## REGISTER GALAXY/MITUTOYO MEASUREMENTS
**In order to ship data to CERN, you need to setup a tunnel as follows:**
ssh -Y @lxplus.cern.ch ssh -L 50022:dbloader-mtd.cern.ch:22 @lxplus.cern.ch
**Help for all available switch**
python 3 register3DMeasurements.py -h
./register3DMeasurements.py
h (help): shows this help
d (dryrun): activate debug mode (automatically set a dryrun)
f (file=<value>): sets the csv filename to read from
m (Mitutoyo): sets the source of data as Mitutoyo
g (galaxy_array): sets the source of data as Galaxy3D (default)
b (galaxy_bar): sets the source of data as Galaxy3D for Single Crystal Bars
o (output=<value>): the filename of the XML output file
w (write): the CERN username authorised to permanently write data to DB
(default to current username)
u (user=<value>): activate debug mode (automatically set a dryrun)
D (debug): activate debugging mode
i (int2r): use test database int2r
* switch -g if we want to register Galaxy data for Array
* switch -b if we want to register Galaxy data for Bars
* switch -m if we want to register Mitutoyo data
* switch -d dryrun/debug: we see the xml output on the screen
* switch -o output file
* switch -u user
* switch -w write in DB
## Galaxy
python3 register3DMeasurements.py -d -f Array_2022_09_20_16_00_21.csv -o Array_2022_09_20_16_00_21.xml -g -u -w -i (Array) python3 register3DMeasurements.py -d -f 1110.csv -o 1110.xml -b -u -w -i (Bar)
[switch -d for dryrun/debug]
## Mitutoyo
python3 register3DMeasurements.py -d -f MS3_Mitutoyo_Array.csv -o MS3_Mitutoyo_Array.xml -m -u -w -i (Array/Bars)
[switch -d for dryrun/debug]
## HOW TO UPDATE LOCATION AFTER SHIPPING
After any shipment you will need to update the location
- cd /home/cmsdaq/MTDDB
- edit file testship.py
-
modifify barcodes = range(842,852,1): add barcode list for which you have to change location
-
modify tracking_number = '' # PLEASE insert shipping tracking number!!!!!
-
modify shipping_company = '' # PLEASE add shipping company name: DHL for shipping to USA and CHINA, UPS for shipping to Italy
-
modify xml = mtdcdb.xml2ship(barcodes, from_institution = 'Roma', from_location = 'Roma',to_institution = 'Bicocca', to_location ='Bicocca'): write new location where LYSO parts will be delivered
- python3 testship.py
## HOW TO SKIP A LYSO PART FROM MEASUREMENTS
In case there is a LYSO Part that for some reasons we won't measure we need to skip it and it won't be visible anymore in the list of part to measure.
python3 ./registerMTDPart.py -x PRE0000000002 -t 'LYSOMatrix #3' -S -w -D -T mtdloadb
switch info: -x --> Barcode (mandatory) -t --> Part type (mandatory) -D --> Debug (in case of dryrun) -T --> Tunnel (you need to add your CERN username) (mandatory)