killms does not find npz file

[amisk]

Hi,

the pipeline was running for a few days, and now it crashed with:

2019-05-12 18:41:23:  - 18:41:23 - ClassVisServer            |   Loading solution file DDS3_full_smoothed in AP mode
2019-05-12 18:41:23:  - 18:41:23 - killMS                    |   Traceback (most recent call last):
2019-05-12 18:41:23:   File "/opt/soft/killms/killMS/kMS.py", line 1157, in <module>
2019-05-12 18:41:23:     main(OP=OP,MSName=MSName)
2019-05-12 18:41:23:   File "/opt/soft/killms/killMS/kMS.py", line 608, in main
2019-05-12 18:41:23:     Load=VS.LoadNextVisChunk()
2019-05-12 18:41:23:   File "/opt/soft/killms/killMS/Data/ClassVisServer.py", line 1003, in LoadNextVisChunk
2019-05-12 18:41:23:     S=np.load(SolFileLoad)
2019-05-12 18:41:23:   File "/usr/local/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 415, in load
2019-05-12 18:41:23:     fid = open(os_fspath(file), "rb")
2019-05-12 18:41:23: IOError: [Errno 2] No such file or directory: '/aux/sblex/ngc3079/killms/mrz/L206646_SB000_uv.dppp_124239E2Bt_113MHz.pre-cal.ms/killMS.DDS3_full_smoothed.sols.npz'
2019-05-12 18:41:23: 
2019-05-12 18:41:27: Traceback (most recent call last):
2019-05-12 18:41:27:   File "/opt/soft/killms/killMS/kMS.py", line 1157, in <module>
2019-05-12 18:41:27:     main(OP=OP,MSName=MSName)
2019-05-12 18:41:27:   File "/opt/soft/killms/killMS/kMS.py", line 608, in main
2019-05-12 18:41:27:     Load=VS.LoadNextVisChunk()
2019-05-12 18:41:27:   File "/opt/soft/killms/killMS/Data/ClassVisServer.py", line 1003, in LoadNextVisChunk
2019-05-12 18:41:27:     S=np.load(SolFileLoad)
2019-05-12 18:41:27:   File "/usr/local/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 415, in load
2019-05-12 18:41:27:     fid = open(os_fspath(file), "rb")
2019-05-12 18:41:27: IOError: [Errno 2] No such file or directory: '/aux/sblex/ngc3079/killms/mrz/L206646_SB000_uv.dppp_124239E2Bt_113MHz.pre-cal.ms/killMS.DDS3_full_smoothed.sols.npz'

I also noticed in previous logs, that it often has errors that are ignored. For example:
KillMS-L206646_SB000_uv.dppp_124239E2Bt_132MHz.pre-cal.ms_DDS0.log:1086:2019-05-01 14:01:58: Exception AttributeError: "ClassFacetMachineTessel instance has no attribute '_delete_cf_in_destructor'" in <bound method ClassFacetMachineTessel.__del__ of <DDFacet.Imager.ClassFacetMachineTessel.ClassFacetMachineTessel instance at 0x7f073e85eb00>> ignored

Any ideas? Do you want all log files?

[mhardcastle]

Looks like a pipeline issue since that file should have been created at an earlier stage. If running the v2 pipeline, are you using the appropriate versions of all the software (i.e. in particular NOT the version of killMS hosted here)?

[mangla-sarvesh]

I am having the same issue.
I am using these version
DDFacet version is 0.7.2.0; kMS version is 3.1.0

These ran successfully

os.system(f"DDF.py --Output-Name={di_image} --Data-MS {msfile} --Deconv-PeakFactor 0.001000 --Data-ColName DATA --Predict-ColName MODEL_DATA --Parallel-NCPU=40 --Output-Mode=Clean --Deconv-CycleFactor=0 --Deconv-MaxMajorIter=3 --Deconv-MaxMinorIter=10000 --Deconv-Mode HMP --Weight-Robust -0.15 --Image-NPix=6400 --CF-wmax 30000 --CF-Nw 100 --Output-Also onNeds --Image-Cell 1.5 --Facets-NFacets=11 --Facets-PSFOversize=1.5 --Freq-NDegridBand 5 --Beam-Model=None --Beam-NBand 1 --Deconv-RMSFactor=3.000000 --Data-Sort 1 --Cache-Dir=. --Freq-NBand=2 --Mask-Auto=1 --Mask-SigTh=5.00 --Cache-Reset 0 --Comp-GridDecorr=0.01 --Comp-DegridDecorr=0.01 --Facets-DiamMax 1.5 --Facets-DiamMin 0.1 --Misc-ConserveMemory 1 --Log-Memory 1")

os.system(f"MakeModel.py --BaseImageName {di_image} --NCluster 10")

os.system(f"kMS.py --MSName {msfile} --SolverType KAFCA --PolMode Scalar --BaseImageName {di_image} --dt 1 --NCPU 40 --OutSolsName testKAFCA --NChanSols 1 --InCol CORRECTED_DATA --OutCol CORRECTED_DATA --Weighting Natural --NodesFile {node_file} --MaxFacetSize 1.5")

Error recieved in this

os.system(f"DDF.py --Output-Name={dd_image} --Data-MS {msfile} --Deconv-PeakFactor 0.001000 --Data-ColName CORRECTED_DATA --Predict-ColName MODEL_DATA --Parallel-NCPU=40 --Output-Mode=Clean --Deconv-CycleFactor=0 --Deconv-MaxMajorIter=3 --Deconv-Mode HMP --Weight-Robust -0.15 --Image-NPix=6400 --CF-wmax 100000 --CF-Nw 100 --Output-Also onNeds --Image-Cell 1.5 --Facets-NFacets=11 --SSDClean-NEnlargeData 0 --Freq-NDegridBand 5 --Beam-NBand 1 --Beam-CenterNorm 1 --Deconv-RMSFactor=3.000000 --Data-Sort 1 --Cache-Dir=. --Freq-NBand=2 --Mask-Auto=1 --Mask-SigTh=5.00 --Cache-Reset 0 --DDESolutions-DDSols testKAFCA --Predict-InitDicoModel {dico_model} --Facets-DiamMax 1.5 --Facets-DiamMin 0.1 ")