Error when run chess_extract

[yanchunzhang]

Hi, I always got this error recently when run the chess_extarct command:
do you have any idea about this error?

Traceback (most recent call last):
File "/home/ch220811/.local/bin/chess", line 585, in
Chess()
File "/home/ch220811/.local/bin/chess", line 75, in init
getattr(self, args.command)([sys.argv[0]] + sys.argv[option_ix:])
File "/home/ch220811/.local/bin/chess", line 567, in extract
args.closing_square)
File "/home/ch220811/.local/lib/python3.6/site-packages/chess/get_structures.py", line 148, in extract_structures
filter1 = filters.threshold_otsu(filter_positive, nbins=size)
File "/home/ch220811/.local/lib/python3.6/site-packages/skimage/filters/thresholding.py", line 285, in threshold_otsu
hist, bin_centers = histogram(image.ravel(), nbins, source_range='image')
File "/home/ch220811/.local/lib/python3.6/site-packages/skimage/exposure/exposure.py", line 139, in histogram
hist, bin_edges = np.histogram(image, bins=nbins, range=hist_range)
File "<array_function internals>", line 6, in histogram
File "/home/ch220811/software/lib/python3.6/site-packages/numpy/lib/histograms.py", line 856, in histogram
decrement = tmp_a < bin_edges[indices]
IndexError: index -9223372036854775808 is out of bounds for axis 0 with size 102

Somtimes I also got this error:
/slurm/reports/var/spool/slurm/d/job466467/slurm_script: line 15: 45401 Killed /home/ch220811/.local/bin/chess extract ${TSV} ${MATRIX1} ${MATRIX2} ${OUT}
Is this because my task got killed by my IT manager or the system due to memory size limit?

Thank you!

[sgalan]

Dear Yanchun Zhang, It seems that this error comes from a bug from numpy. The threshold for the histogram is too high for numpy to work properly. Maybe you may consider some fixes that are suggested in here: numpy/numpy#8627. I will also think about alternatives, Sorry for the inconvenience, S El lun, 14 dic 2020 a las 18:24, Yanchun Zhang (<[email protected]>) escribió:

…

Hi, I always got this error recently when run the chess_extarct command: do you have any idea about this error? Traceback (most recent call last): File "/home/ch220811/.local/bin/chess", line 585, in Chess() File "/home/ch220811/.local/bin/chess", line 75, in *init* getattr(self, args.command)([sys.argv[0]] + sys.argv[option_ix:]) File "/home/ch220811/.local/bin/chess", line 567, in extract args.closing_square) File "/home/ch220811/.local/lib/python3.6/site-packages/chess/get_structures.py", line 148, in extract_structures filter1 = filters.threshold_otsu(filter_positive, nbins=size) File "/home/ch220811/.local/lib/python3.6/site-packages/skimage/filters/thresholding.py", line 285, in threshold_otsu hist, bin_centers = histogram(image.ravel(), nbins, source_range='image') File "/home/ch220811/.local/lib/python3.6/site-packages/skimage/exposure/exposure.py", line 139, in histogram hist, bin_edges = np.histogram(image, bins=nbins, range=hist_range) File "<*array_function* internals>", line 6, in histogram File "/home/ch220811/software/lib/python3.6/site-packages/numpy/lib/histograms.py", line 856, in histogram decrement = tmp_a < bin_edges[indices] IndexError: index -9223372036854775808 is out of bounds for axis 0 with size 102 Thank you! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#25>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADZPOZHIXHFFQGJB7ENEJD3SUZC6NANCNFSM4U3CGDRQ> .

-- *Silvia Galan Martínez - PhD student * *Centre Nacional d'Anàlisi Genòmica-Centre de Regulació Genòmica (CNAG-CRG)* *Structural Genomics Dpt.* Parc Científic de Barcelona – Torre I Baldiri Reixac, 4 08028 Barcelona Tel +34 9340 20580 Email [email protected]

[yanchunzhang]

Thank you for your reply, @sgalan !
I see the numpy_histogram bug you mentioned but I'm still not sure how to avoid this happening.
Why do you think there could be a number larger than 2**53 generated in the process of chess_extract? What the numbers used for histogam reresent? And do i need to modify the code of chess_extract or numpy_histogram by myself?

[sgalan]

Hi Yanchun Zhang, Which matrices are you using as an input for this command? The first thing that you can try to do is to change the lines 148 and 154 in which the histogram is computed in order to binarize the image. It uses all the values from the image in order to compute a threshold to binarize the image. Now it's using the whole image to do so, you can try to put a smaller value, like the half of the size of your image in the parameter nbins. Hope it helps, S El lun, 14 dic 2020 a las 20:59, Yanchun Zhang (<[email protected]>) escribió:

…

Thank you for your reply, @sgalan <https://github.com/sgalan> ! I see the numpy_histogram bug you mentioned but I'm still not sure how to avoid this happening. Why do you think there could be a number larger than 2**53 generated in the process of chess_extract? What the numbers used for histogam reresent? And do i need to modify the code of chess_extract or numpy_histogram by myself? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADZPOZCYTNYTRQGHNZMONVDSUZVCBANCNFSM4U3CGDRQ> .

-- *Silvia Galan Martínez - PhD student * *Centre Nacional d'Anàlisi Genòmica-Centre de Regulació Genòmica (CNAG-CRG)* *Structural Genomics Dpt.* Parc Científic de Barcelona – Torre I Baldiri Reixac, 4 08028 Barcelona Tel +34 9340 20580 Email [email protected]