Add update_environment function to manage.sh #170

jmbenlloch · 2017-03-13T15:37:16Z

Add new function update_environment X.Y to update all packages within one particular IC environment to their latest versions available in conda.

Basically one needs to run:

source manage.sh update_environment 2.7
source manage.sh update_environment 3.5
source manage.sh update_environment 3.6

This is the low-hanging fruit from #166. Opinions?

jacg · 2017-03-13T15:44:52Z

Updating conda-installed packages is not enough: some things we get through pip!

jacg · 2017-03-13T15:47:38Z

I'd also like an update_all function, so that we don't have to do the same thing N times for however many python versions we happend to be supporting at the time.

jmbenlloch · 2017-03-13T15:53:29Z

You are right, I forgot about those, I'll updated the code and also create update_all.

jmbenlloch · 2017-03-13T16:20:20Z

Ok, I have added update_all so we can run only one command to update all environments. Not sure whether awk is available in MacOS, if not it can be changed easily by something else.

Regarding pip-installed packages, it is not obvious how to update them. There is no command to update all packages in pip (pypa/pip#59). If we want to iterate manually over all packages installed by pip we need to know which ones are those, but pip list and pip freeze shows all packages installed (also those from conda).

jacg · 2017-03-13T16:27:04Z

This whole manage.sh business makes me nervous because

I don't see how we can sensibly test it automatically, so we have to test it by eye.
There's lots of repetition in there, and because it's never of highest urgency, we're putting up with it and slowly letting it grow into a monster.

The conda env | grep IC, while a great idea in principle, makes me nervous in practice. I just hope that I (and everyone else) remember never to have any non-IC related envs with IC in their name.

As for pip ... hmm, some thoughts

What if we just trash whole of pip, and re-install everything. To what extent is pip shared between different conda envs?
Iterate over the things mentioned in the pip section of the environment specification. This will probably fail to update dependencies.

Traffic of arguments in function events_loop for the various cities expressed more concisely with the user of DataVectors and PmapVectors containers.

A KrCity computes and writes KrEvents, e.g, point-like events. This means that the baricenter integrates all the time slices for a given SiPM. Instead, a HitCity computes and writes HitEvents, in which the baricenter (or any other position-finding algo) is computed slice-by-slice. A HitCity is derived from a KrCity since both of them use the same file_loop() function (which starts from pmaps).

Without the exit, the command-line test was passing with pretty greens when the city was not found (e.g, a captured ModuleNotFound, print that pytest captures and digests without telling anything and happy ending in spite of wrong situation). Adding traceback still gives printout when running in normal mode but provides diagnostics of the problem

The first few events are rejected by selection in the test sample in Penthesilea, running 10 events guarantees that the city runs fully.

this function allows takes an s2sid_peak and slice number and returns the sipm ids of sipms in that s2si, with their charges at slice number

This is needed to use the corresponding cython function rather than previous python version.

Include in the list only slices with positive charge.

One has to take into account the possibility that the slice results in an empty SiPM list. This is handled via SipmEmptyList exception.

In this case, the slice is looped away.

Cecilia is removed from cities. The simulation of the trigger will be done shortly in Diomira.

Although it may be argued that we reinvent the wheel, the use of a numerical NN is quite convenient when writing/reading pytables to/from disk, as a proxy for NaN.

Cluster takes the std (not the rms) as argument. A number of properties (Xrm and such) are modified accordingly.

The change of definition in Cluster (takes std, not rms) requires to fix a number of tests.

hit_city inherits now from kr_city. Config. files modified accordingly.

@jmbenlloch

This pr will allow irene to make use of the ipmt pmap finders and writers in order to save ipmt pmaps. I've run this irene in my notebook a few times and everything looks good so far. In order to tell irene to write ipmt pmaps (or not) I had to put a flag compute_ipmt_pmaps in pmap_city.conf. This means that previous irene config files will not work without first adding the flag. This may be kind of annoying for people running productions @jmbenlloch @jerenner @gonzaponte. If there is a better way of doing this, please let me know, and I will implement it. The only alternative I could think of would be to create another city, but that sounds potentially even more annoying for you guys. I did some time/memory profiling for on Kr data. From run 4446 with thr_sipm = 1pes and 'thr_sipm_s2 = 2pes' and found after running 1k events: speed 1.92 evt / sec with IPMT 2.06 evt / sec without file size 2.37 x larger than without IPMT So, time does not appear to be an issue, but the file size does grow substantially... @jjgomezcadenas I don't remember if we decided we wanted to put the ipmt pmaps in a different group, called IPMT in the hdf5 file... did we?

We were computing a new event number in Diomira because in the previous MC production, each file restarted the event counter to zero. This is not the case anymore, each file has a unique event number. Since this was a patch intended to use that old production, it could be removed now.

jacg · 2017-09-07T22:17:12Z

This is stale: any plan to move it forward, @jacg, @jmbenlloch?

We'll get around to it eventually.

Fixes next-exp#328.

Closes next-exp#328 Previously, the code was crashing when a hit collection contained only one item.

jacg · 2018-07-26T06:47:17Z

I suspect that this is going to be superseded by #496.

jacg · 2018-07-28T11:02:40Z

I think that this is being superseded by #496.

jjgomezcadenas and others added 25 commits July 19, 2017 18:02

Propagate change (params.py --> ic_containers).

86d881c

Add DataVectors and PmapVectors to ic_containers

66e7c80

Use of DataVectors and PmapVectors in cities.

49805dc

Traffic of arguments in function events_loop for the various cities expressed more concisely with the user of DataVectors and PmapVectors containers.

rebin_s2si uses new Peak class

9c3aba1

Cosmetics

752d882

Fixed wrong module names in cecilia and penthesilea

9b31ccf

Run 10 events by dafault in Penthesilea

3f9eba7

The first few events are rejected by selection in the test sample in Penthesilea, running 10 events guarantees that the city runs fully.

work in progress

349221c

write sipm_ids_and_charges_in_slice with test

e6a9bb9

this function allows takes an s2sid_peak and slice number and returns the sipm ids of sipms in that s2si, with their charges at slice number

cythonize sipm_ids_and_charges_in_slice

a0d3b51

fix tests for rebin_s2si

a8a2bb1

Add pmaps_functions_c to base.

2eb9dd7

This is needed to use the corresponding cython function rather than previous python version.

Return only non-empty slices in sipm_ids_and_charges...

1a9d5ca

Add writer to penthesilea.

dae8bc0

work in progress on cecilia.

d6bceab

Modify functions sipm_ids_and_charges_in_slice.

874cf49

Include in the list only slices with positive charge.

Fix function compute_xy_position

f5e2b05

One has to take into account the possibility that the slice results in an empty SiPM list. This is handled via SipmEmptyList exception.

Take into account that clusters for a given slice_no can be empty.

4e5a0fc

In this case, the slice is looped away.

Remove Cecilia from cities.

f113f36

Cecilia is removed from cities. The simulation of the trigger will be done shortly in Diomira.

Add NN (NoNumber) with value -999999

1fb9928

Although it may be argued that we reinvent the wheel, the use of a numerical NN is quite convenient when writing/reading pytables to/from disk, as a proxy for NaN.

Cluster takes std (not rms) as argument.

097cdaf

Cluster takes the std (not the rms) as argument. A number of properties (Xrm and such) are modified accordingly.

Various tests involving cluster fixed.

6900bc6

The change of definition in Cluster (takes std, not rms) requires to fix a number of tests.

Redefined conf. files for kr_city and hit_city.

d157523

hit_city inherits now from kr_city. Config. files modified accordingly.

abotas and others added 14 commits September 6, 2017 12:40

first functional ipmt irene

15816a8

create and use sum_waveforms in pmt_transformation

8bb93ef

cythonize get_ipmtd

84e9fdb

add tests for find_s12_ipmt functions

de96b52

prevent crash in search for s2si when s2 is None

ff58314

Test implemented to deal with 1 hit case

2d9235a

Test return an assertion

5b49eae

Fix broken test (voxelize_single_hit)

96543f9

Add missing whitespace in return type annotations

fa27747

Sanitize docstrings

5f33f69

Clean up implementation of shortest_paths

aacd947

Shorten voxelize_hits

7d09020

jacg force-pushed the manage.sh branch from 1b3c1cb to e0b4540 Compare September 7, 2017 22:15

Ensure that at least one bin is required

aa3ca94

Fixes next-exp#328.

jacg mentioned this pull request Sep 7, 2017

Failure in two diomira tests. #299

Closed

Jacek Generowicz and others added 3 commits September 8, 2017 11:18

Deal with 1 hit in voxelize_hits

7817404

Closes next-exp#328 Previously, the code was crashing when a hit collection contained only one item.

Add update_environment function to manage.sh

4a68042

Add update all

e0680b7

jacg force-pushed the manage.sh branch from e0b4540 to e0680b7 Compare September 8, 2017 09:21

jacg force-pushed the master branch 5 times, most recently from 71dda93 to eb903c7 Compare September 15, 2017 05:08

jacg closed this Jul 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add update_environment function to manage.sh #170

Add update_environment function to manage.sh #170

jmbenlloch commented Mar 13, 2017

jacg commented Mar 13, 2017

jacg commented Mar 13, 2017

jmbenlloch commented Mar 13, 2017

jmbenlloch commented Mar 13, 2017

jacg commented Mar 13, 2017

jacg commented Sep 7, 2017

jacg commented Jul 26, 2018

jacg commented Jul 28, 2018

Add update_environment function to manage.sh #170

Add update_environment function to manage.sh #170

Conversation

jmbenlloch commented Mar 13, 2017

jacg commented Mar 13, 2017

jacg commented Mar 13, 2017

jmbenlloch commented Mar 13, 2017

jmbenlloch commented Mar 13, 2017

jacg commented Mar 13, 2017

jacg commented Sep 7, 2017

jacg commented Jul 26, 2018

jacg commented Jul 28, 2018