share data with pilot project team

[prjemian]

In today's conference call, we decided to package up some data from our recent beam time with others in the pilot project team. We need to pick the scans for packaging. We have a tool (rube) to gather the data for export.

[prjemian]

list the last 20 scans (listruns()):

short_uid	date/time	exit	scan_id	command
93b616b	2020-02-10 14:10:38.820923	success	6853	lambda_test(detector_name=lambdadet, ...
a118c45	2020-02-10 13:30:49.978397	success	6852	lambda_test(detector_name=lambdadet, ...
6ad6350	2020-02-10 13:29:32.988123	success	6851	lambda_test(detector_name=lambdadet, ...
f77286b	2020-02-10 13:06:36.788908	success	6850	lambda_test(detector_name=lambdadet, ...
d8f9d78	2020-02-07 12:35:30.146093	success	6849	lambda_test(detector_name=lambdadet, ...
defa9b6	2020-02-07 12:12:25.037005	success	6848	lambda_test(detector_name=lambdadet, ...
57da744	2020-02-07 12:03:26.677343	success	6847	lambda_test(detector_name=lambdadet, ...
538313e	2020-02-07 12:03:14.116846	success	6846	lambda_test(detector_name=lambdadet, ...
cacdc16	2020-02-07 12:02:51.911826	success	6845	lambda_test(detector_name=lambdadet, ...
855c957	2020-02-07 12:02:46.984878	success	6844	lambda_test(detector_name=lambdadet, ...
ae7f5d5	2020-02-07 12:02:41.881250	success	6843	lambda_test(detector_name=lambdadet, ...
d0d0eb6	2020-02-07 12:02:36.890868	success	6842	lambda_test(detector_name=lambdadet, ...
37dbbec	2020-02-07 12:02:31.792362	success	6841	lambda_test(detector_name=lambdadet, ...
9915659	2020-02-07 12:02:26.707210	success	6840	lambda_test(detector_name=lambdadet, ...
00f616d	2020-02-07 12:02:21.723238	success	6839	lambda_test(detector_name=lambdadet, ...
19b1b15	2020-02-07 12:02:16.783343	success	6838	lambda_test(detector_name=lambdadet, ...
15ff957	2020-02-07 12:02:11.481077	success	6837	lambda_test(detector_name=lambdadet, ...
8c9197e	2020-02-07 12:02:05.686189	success	6836	lambda_test(detector_name=lambdadet, ...
db6ea46	2020-02-07 12:02:00.577515	success	6835	lambda_test(detector_name=lambdadet, ...
6ad9125	2020-02-07 12:01:55.498382	success	6834	lambda_test(detector_name=lambdadet, ...

[prjemian]

We need more info from the start document:

In [24]: db["d8f9d78"].start
Out[24]:
{'time': 1581100530.146093,
 'uid': 'd8f9d78b-bfbb-43dc-99de-4153d07e99ea',
 'beamline_id': 'APS TRR 8-ID-I',
 'proposal_id': None,
 'pid': 24701,
 'login_id': '[email protected]',
 'versions': {'bluesky': '1.4.1',
  'ophyd': '1.4.0rc4',
  'databroker': '0.13.3',
  'apstools': '1.1.17rc6',
  'epics': '3.4.0',
  'numpy': '1.18.1',
  'matplotlib': '3.0.3',
  'spec2nexus': '2021.1.3',
  'pyRestTable': '2020.0.2'},
 'scan_id': 6849,
 'plan_type': 'generator',
 'plan_name': 'lambda_test',
 'file_name': 'PJ078_bluesky_test_with_sleep_Lq0_001',
 'file_path': '/home/8-id-i/2020-1/bluesky/PJ078_bluesky_test_with_sleep_Lq0_001/',
 'sample_name': 'bluesky_test_with_sleep',
 'ARun_number': 'PJ078_bluesky_test_with_sleep_Lq0_001',
 'plan_args': {'detector_name': 'lambdadet',
  'acquire_time': 0.001,
  'acquire_period': 0.001,
  'num_images': 1000,
  'file_name': 'PJ078_bluesky_test_with_sleep_Lq0_001',
  'submit_xpcs_job': 'False',
  'path': '/home/8-id-i/2020-1/bluesky/'}}

We want num_images. The listruns() command is too limited to filter that.

[prjemian]

All are 1000 images:

In [29]: for h in db(since="2020-02-07 12:00", until="2020-02-08"):
    ...:     st = h.start
    ...:     print(st["uid"], st["plan_args"]["num_images"])
    ...:
d8f9d78b-bfbb-43dc-99de-4153d07e99ea 1000
defa9b6e-244f-44f1-8490-63c5a2b37b28 1000
57da7445-a87e-4123-87a1-43095325036b 1000
538313ef-5cb4-4548-9316-0b073fd6a67c 1000
cacdc16d-d071-4013-93b1-8494aa1a8f0f 1000
855c957e-4030-4c0c-8f79-27be01a8377c 1000
ae7f5d52-bdeb-453b-aad3-f8730aad6d56 1000
d0d0eb68-d270-428c-81b6-03892893d0fa 1000
37dbbece-3609-46d6-bed9-43ac6005950e 1000
99156597-4c84-42b2-8000-f257e30987e2 1000
00f616db-3054-4079-b993-366e7e31d07f 1000
19b1b15b-1fee-48ed-95f1-966220071e1a 1000
15ff957e-af47-441d-aa2b-bc47e22275ed 1000
8c9197ec-595e-4546-9c80-8e088ec51910 1000
db6ea469-f4e2-4381-9972-3f3955355919 1000
6ad91259-af61-40cc-b816-b17f92b7db49 1000
516073ed-0495-4f68-8d4d-aa5395a95749 1000
815ca33a-6bb3-49ec-9c37-8e80a67f2f81 1000
86ff8b8e-8c54-4edc-9117-fe45b69b9e1c 1000

Anything else to help us select which scans?

[prjemian]

Let's start with the most recent 5 scans from our last day of pilot team data collection (2020-02-07) and see where that takes us:

short_uid	date/time	exit	scan_id	command
f77286b	2020-02-10 13:06:36.788908	success	6850	lambda_test(detector_name=lambdadet, ...
d8f9d78	2020-02-07 12:35:30.146093	success	6849	lambda_test(detector_name=lambdadet, ...
defa9b6	2020-02-07 12:12:25.037005	success	6848	lambda_test(detector_name=lambdadet, ...
57da744	2020-02-07 12:03:26.677343	success	6847	lambda_test(detector_name=lambdadet, ...
538313e	2020-02-07 12:03:14.116846	success	6846	lambda_test(detector_name=lambdadet, ...

[prjemian]

@qzhang234 : I have packed up these 5 scans into snow:/local/pilot

The *.msgpack files have the databroker content and the ./files/ subdirectory has the .imm image files.

[prjemian]

Investigate Feb 5-6 for data. Mid-day on Feb 6 might be ideal.

[prjemian]

Basic data mining command:

apstools.utils.listruns(
    keys=["file_name",], 
    until="2020-02-07", 
    since="2020-02-05 15:24", 
    num=2500, 
    plan_name="lambda_test", 
    exit_status="success")

[prjemian]

Yikes! That's almost 2,000 matching runs. Perhaps we can filter that? Full list of runs is in snow:/local/pilot/catalog.txt

@qzhang234 : let's review and select for @dylanmcreynolds

[dylanmcreynolds]

@qzhang234 as I said to @prjemian, I think the most interesting things were on Feb 6, which is the last day we were there. That's when you had a sample and we were testing with different numbers of frames, right?

[prjemian]

This is done, confirmed in weekly telco.