Histograms from timed dataframe

[rettigl]

This adds support for a per-time-unit dataframe, and histogram calculation based on this dataframe.

Histogram calculation in the example notebook takes on our machine ~55s. This can potentially still be improved, e.g. by taking our fast binning method (currently fails because of xarray generation and concatenation issues)

[rettigl]

This does not produce the right kind of normalization histogram yet, I think...

[rettigl]

This implements both variants now. Both routes produce almst identical results, but histograms from timed dataframes are substantially faster. Also a normalization routine for the compute function is added.

One issue remains: If histogram axes are jittered, the actual values of the jitter are different during computation of the data and of the normalization histogram, leading to small errors in the histogram on the order of a percent or so. A workaround would be to include the histogram calculation into bin_partition. This would also avoid reading the data twice, which I think is the major time bottleneck at the moment.

[rettigl]

Histogram ontop of data (continously scanned delay stage):

Performance comparison:

Two histogram methods compared (bottom: difference):

Histogram computed twice and compared (jittered):

Bottomline: All effects of randomness and difference between methods are < 1%.

[rettigl]

@zainsohail04 This one is pending on your implementation into the Flash loader (and some review)...

[rettigl]

Rebased, and added dummy implementation for flash loader. Tests are also still pending.

[coveralls]

Pull Request Test Coverage Report for Build 6760852807

• 291 of 314 (92.68%) changed or added relevant lines in 14 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.7%) to 90.759%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sed/loader/flash/loader.py	6	7	85.71%
sed/loader/generic/loader.py	3	4	75.0%
sed/loader/mpes/loader.py	34	39	87.18%
sed/core/processor.py	92	108	85.19%

Totals
Change from base Build 6733527384:	0.7%
Covered Lines:	4901
Relevant Lines:	5400

---

💛 - Coveralls

[rettigl]

@zain-sohail For me the flash timed_dataframes don't work. They consume endless amounts of memory and never finish computation.

[zain-sohail]

@zain-sohail For me the flash timed_dataframes don't work. They consume endless amounts of memory and never finish computation.

Just tried and it seems to work fine. Where are you facing a problem?

[rettigl]

After merging the fast loading code from main, it is at lead performing faster now, but still does not seem to do the right thing:
I am loadings runs=[44824, 44825, 44826, 44827]


It does not make sense that the timed dataframe has 8 times more rows...

[rettigl]

Or does it? Are there so many empty macrobunches?

[rettigl]

Hmm, okay the generated histogram does indeed not look to bad

[rettigl]

The time-stamp version does produce some strange artefacts, though:

[steinnymir]

Or does it? Are there so many empty macrobunches?

if you look at the tables you show, the pulse IDs in the electron dataframe are very sparse (7, 38, 69...) so it seems like count rate was really bad, and therefore I expect many empty pulses.

[rettigl]

rebased to current main

[rettigl]

Or does it? Are there so many empty macrobunches?

if you look at the tables you show, the pulse IDs in the electron dataframe are very sparse (7, 38, 69...) so it seems like count rate was really bad, and therefore I expect many empty pulses.

I realized that after posting my previous post, but don't undertand the data format completely yet. What exactly are the Train, pulse, and electron IDs? Are these Macrobunches, microbunches, and electrons per microbunch? And what is the time-base for the timed_dataframe? I was expecting macrobunches, but then I would expect one row per trainID, no? Now it seems one row per microbunch. This is certainly typically more than electrons. And why is it going up to 1000, if tpyically there are 400 or 500 microbunches in a macrobunch?

[steinnymir]

How you described it is all correct.
the reason there are 1000 is I think because the data structure accomodates up to 1000 pulses, as flash can theoretically provide 800, which it never does, max 500 in reality.
I'm not sure if using trainID (macrobunches) would be still ok, don't you think a coarser step size induces more noise?

[rettigl]

I'm not sure if using trainID (macrobunches) would be still ok, don't you think a coarser step size induces more noise?

All you want here is to capture changes of the scanned variable, which typically is anyways only read out once per macrobunch, no? So no, there should be no difference. It would only make a (marginal) difference for parameters that are varied within the macrobunch. Not sure, though, how this would look like if you consider, e.g. BAM correction etc.

[steinnymir]

All you want here is to capture changes of the scanned variable, which typically is anyways only read out once per macrobunch, no? So no, there should be no difference. It would only make a (marginal) difference for parameters that are varied within the macrobunch. Not sure, though, how this would look like if you consider, e.g. BAM correction etc.

Right! but still there might be some pulse resolved normalizations one would want, like the FEL intensity for example. I'm afraid we need to keep the per-pulse dataframe for those

[zain-sohail]

After merging the fast loading code from main, it is at lead performing faster now, but still does not seem to do the right thing: I am loadings runs=[44824, 44825, 44826, 44827]

Wouldn't make sense that there is a timing difference in the compute of dataframe, only makes a difference in creating buffer files quicker, I would imagine. Regarding your question about why the dataframe is bigger, I think Steinn managed to answer it.

[steinnymir]

Looks good! I tested it together with the hextof changes from #169 and seems to work correctly. Only exception the comments I made. Once those are resolved you can merge for what concerns me.

[steinnymir]

I accidentally uploaded the branch where i tested this. I'll leave it till this is merged. its hist_testing

[rettigl]

I fixed this, and modified the tests to test the timed dataframes.

[rettigl]

I accidentally uploaded the branch where i tested this. I'll leave it till this is merged. its hist_testing

I also updated your test branch

[steinnymir]

I tested this and seems to work fine. LGTM!

[steinnymir]

closes #101

[rettigl]

I am working on improving tests for this still, and update the new processor functions, then I will merge