MemoryError -- flydra_analysis_export_flydra_hdf5

[maxlklaxl]

tired it 2 times on the fishtrax machine, both times the same error
the .h5 file is 13G

(PY)strawlab@fishtrax:~/FLYDRA$ flydra_analysis_export_flydra_hdf5 20140719_152830.mainbrain.h5
STAGE 1: finding timestamps
opening file 20140719_152830.mainbrain.h5...
caching raw 2D data... done
(cached index of 145054462 frame values of dtype int64)
hostname time_gain time_offset

---

'localhost' 1.0 -0.00697165842925

caching Kalman obj_ids...
finding unique obj_ids...
(found 57384)
(will export 57384)
finding 2d data for each obj_id...
skipping frame 403 (obj 1): no data2d_distorted data
skipping frame 404 (obj 2): no data2d_distorted data
skipping frame 405 (obj 3): no data2d_distorted data
skipping frame 4439 (obj 47): no data2d_distorted data
skipping frame 4621 (obj 52): no data2d_distorted data
/home/strawlab/src/flydra/flydra/a2/data2smoothed.py:168: UserWarning: no host fishtrax in timestamp data. making up data.
'data.'%remote_hostname)
STAGE 2: running Kalman smoothing operation
detected file loaded with dynamic model "EKF fishbowl40"
for smoothing, will use dynamic model "fishbowl40"
/home/strawlab/src/flydra/flydra/a2/core_analysis.py:1467: UserWarning: passing data_file as string to core_analysis.CachingAnalyzer.load_data()
warnings.warn('passing data_file as string to '
/home/strawlab/src/flydra/flydra/a2/core_analysis.py:555: RuntimeWarning: invalid value encountered in true_divide
velocity_direction = velocity/speed[:,np.newaxis]
/home/strawlab/src/flydra/flydra/a2/data2smoothed.py:243: UserWarning: algebra error smoothing obj_id 17414, skipping.
warnings.warn('algebra error smoothing obj_id %d, skipping.'%(obj_id,))
/home/strawlab/src/flydra/flydra/a2/data2smoothed.py:243: UserWarning: algebra error smoothing obj_id 47242, skipping.
warnings.warn('algebra error smoothing obj_id %d, skipping.'%(obj_id,))
[obj_id: 57432] 2.07 obj_ids /s100%|##########################################################|Time: 7:41:40
Traceback (most recent call last):
File "/home/strawlab/PY/bin/flydra_analysis_export_flydra_hdf5", line 9, in
load_entry_point('flydra==0.5.9', 'console_scripts', 'flydra_analysis_export_flydra_hdf5')()
File "/home/strawlab/src/flydra/flydra/a2/data2smoothed.py", line 280, in export_flydra_hdf5
main(hdf5_only=True)
File "/home/strawlab/src/flydra/flydra/a2/data2smoothed.py", line 377, in main
**kwargs)
File "/home/strawlab/src/flydra/flydra/a2/data2smoothed.py", line 262, in convert
recarray = numpy.rec.array(allrows)
File "/usr/lib/python2.7/dist-packages/numpy/core/records.py", line 790, in array
new = new.copy()
MemoryError
Closing remaining open files: /home/strawlab/FLYDRA/20140719_152830.mainbrain.kh5-smoothcache... done 20140719_152830.mainbrain.h5... done

[nzjrs]

Heh, not enough memory. Maybe run it on strawcore (the pipeline)