Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

util.py KeyError returned while indexing error codes #12

Closed
bourque opened this issue Nov 11, 2016 · 9 comments
Closed

util.py KeyError returned while indexing error codes #12

bourque opened this issue Nov 11, 2016 · 9 comments

Comments

@bourque
Copy link

bourque commented Nov 11, 2016

@sosey Myself and others (@cmartlin, @mrafelski, @ivastar) are running into an issue where wfc3tools.util.py crashes when attempting to return an error code, e.g.:

  File "/grp/hst/wfc3h/bourque/repository/pyql/pyql/logging/logging_functions.py", line 212, in wrapped
    func(*a, **kw)
  File "/grp/hst/wfc3h/bourque/repository/pyql/pyql/logging/logging_functions.py", line 175, in wrapped
    func(*a, **kw)
  File "cal_uvis_make_darks.py", line 276, in process_darks_main
    preprocess_data_main(paths, postflash)
  File "/grp/hst/wfc3h/bourque/repository/automated_scripts/cal_uvis_make_darks/preprocess_data.py", line 87, in preprocess_data_main
    make_blvs_main(directory)
  File "/grp/hst/wfc3h/bourque/repository/automated_scripts/cal_uvis_make_darks/make_blvs.py", line 175, in make_blvs_main
    run_calwf3()
  File "/grp/hst/wfc3h/bourque/repository/automated_scripts/cal_uvis_make_darks/make_blvs.py", line 126, in run_calwf3
    calwf3(input='dark_asn.fits', save_tmp=True)
  File "/grp/hst/wfc3p/bourque/envs/anaconda3/envs/astroconda3/lib/python3.5/site-packages/wfc3tools/calwf3.py", line 77, in calwf3
    ec = error_code(return_code)
  File "/grp/hst/wfc3p/bourque/envs/anaconda3/envs/astroconda3/lib/python3.5/site-packages/wfc3tools/util.py", line 64, in error_code
    return codes[code]
KeyError: -7

Digging into the valid error codes in wfc3tools.util.py, it does't look like -7 is a choice:

def error_code(code=None):
    """ return the error code text, or all if code is None."""

    codes = {2: "ERROR_RETURN",
             111: "OUT_OF_MEMORY",
             114: "OPEN_FAILED",
             115: "CAL_FILE_MISSING",
             116: "NOTHIN_TO_DO",
             117: "KEYWORD_MISSING",
             118: "ALLOCATION_PROBLEM",
             119: "HEADER_PROBLEM",
             120: "SIZE_MISMATCH",
             130: "CAL_STEP_NOT_DONE",
             141: "TABLE_ERROR",
             142: "COLUMN_NOT_FOUND",
             143: "ELEMENT_NOT_FOUND",
             144: "ROW_NOT_FOUND",
             151: "NO_GOOD_DATA",
             152: "NO_CHIP_FOUND",
             171: "REF_TOO_SMALL",
             999: "INTERNAL_ERROR",
             1001: "INVALID_EXPTIME",
             1011: "INVALID_FILENAME",
             1020: "WRITE_FAILED",
             1021: "INVALID_TEMP_FILE",
             1023: "FILE_NOT_READABLE",
             1025: "COPY_NOT_POSSIBLE",
             1111: "INVALID_VALUE",
             1030: "UNSUPPORTED_APERTURE",
             }

    if code is None:
        return codes
    else:
        return codes[code]

Any idea what might be going on here?

This prompted me to dig into why my call to calwf3 might be triggering an error in the first place. Upon digging, I noticed that though I explicitly set the input variable to point to an association table (which contains the list of files I want to process) e.g ...

from wfc3tools.calwf3 import calwf3
calwf3(input='dark_asn.fits', save_tmp=True)

... calwf3 appears to be gathering any FITS files in the CWD and setting them as the infiles, e.g.:

        infiles, dummy_out = parseinput.parseinput(input)
        logging.info(infiles)
        for image in infiles:
            if not os.path.exists(image):
                raise IOError('Input file not found: ' + image)

        call_list.append(','.join(infiles))
['id0k2wkrq_raw.fits', 'id0k2vi6q_raw.fits', 'id0k2wklq_raw.fits', 'id0k2yn7q_raw.fits', 'id0k2xmiq_raw.fits', 'id0k2vi0q_raw.fits', 'id0k2uclq_raw.fits', 'id0k2zsmq_raw.fits', 'id0k2yn3q_raw.fits', 'id0k2zskq_raw.fits', 'id0k3aupq_raw.fits', 'id0k2xmhq_raw.fits', 'id0k3auoq_raw.fits', 'id0k3fujq_raw.fits', 'id0k3fuiq_raw.fits', 'id0k3ifnq_raw.fits', 'id0k3hcxq_raw.fits', 'id0k3hcwq_raw.fits', 'id0k3ga5q_raw.fits', 'id0k3jfsq_raw.fits', 'id0k3ga1q_raw.fits', 'id0k3jfqq_raw.fits', 'id0k3lq6q_raw.fits', 'id0k3lq5q_raw.fits', 'id0k3qhyq_raw.fits', 'id0k3qhxq_raw.fits', 'id0k3pfhq_raw.fits', 'id0k3tbkq_raw.fits', 'id0k3tbgq_raw.fits', 'id0k3obsq_raw.fits', 'id0k3obrq_raw.fits', 'id0k4areq_raw.fits', 'id0k4ardq_raw.fits', 'id0k4dc1q_raw.fits', 'id0k4ec9q_raw.fits', 'id0k4ec5q_raw.fits', 'id0k4jknq_raw.fits', 'id0k4jkiq_raw.fits', 'id0k4ikbq_raw.fits', 'id0k4ijpq_raw.fits']

When I bypass this code and explicitly set call_list, I can avoid an error and calwf3 works fine:

call_list = ['calwf3.e', 'dark_asn.fits', '-s']
proc = subprocess.Popen(
    call_list,
    stderr=subprocess.STDOUT,
    stdout=subprocess.PIPE,
)

Could you advise?

@sosey
Copy link
Member

sosey commented Nov 11, 2016

yes, @ivastar emailed me earlier this week and I have a fix for this, but have been conferencing all week. I'll update this when I get a chance

@bourque
Copy link
Author

bourque commented Nov 11, 2016

@sosey Great, thanks!

@sosey
Copy link
Member

sosey commented Nov 16, 2016

@bourque so these are a couple different issues playing with each other. -7 is a unix return error, not a calwf3 return error. parseinput comes from stsci.tools. It's an attempt to take into account different kinds of input that the user might provide and return a parsed list of expected inputs but the code is then attaching the wrong string to the calwf3 call. I think this section of the code got messed up when I applied a recent pull request. In any case, the code was trying to be helpful and check to make sure the files you want to process actually exist on disk before kicking off, for ASN files, this means opening them and returning the list of members to check

infiles,out= parseinput.parseinput('*raw.fits')

returns this to infiles:

['iaao01k8q_raw.fits',
 'iaao01k9q_raw.fits',
 'iacr51ohq_raw.fits',
 'iacs01t4q_raw.fits',
 'ibh719grq_raw.fits']

while, using an asn file actually open the asn file:

 infiles,out= parseinput.parseinput('iaao01010_asn.fits')

In [34]: infiles
Out[34]: ['iaao01k8q_raw.fits', 'iaao01k9q_flt.fits']

In [35]: out
Out[35]: 'IAAO01011'

It's supposed to return the list of input files from the association table, I'm unsure why in this example, it's returning the raw file for the first member and the processed (flt) file for the second. The output looks good, which is the association output name. Because of this I'm switching to a different function to check the file existence and fixing the bug that uses the list instead of the user input...as well as the fact that the calwf3 executable should only accept one input file...multiple bugs! showing how often this has been used :/

multiple input files (when not in an ASN file) should be taken care of by the user by calling calwf3 from a list.

for image in mylist:
calwf3(image)

check latest pull request out for yourself and see if it's working now?

*** parseinput starting returning expected output for me, no idea what was messed up so I switched back to that.

@sosey
Copy link
Member

sosey commented Nov 16, 2016

@ivastar check it out and see if it works for you as well?

@bourque
Copy link
Author

bourque commented Nov 16, 2016

@sosey Thanks for looking into this. The version in your pull request is working!

@sosey
Copy link
Member

sosey commented Nov 16, 2016

@bourque great.

@bourque
Copy link
Author

bourque commented Nov 17, 2016

@sosey I forgot to ask ... when can I expect to see the changes in astroconda?

@sosey
Copy link
Member

sosey commented Nov 17, 2016

@bourque I was going to give @ivastar a couple days to try it out but I can update the version information for astroconda today or tomorrow. Since you've already installed the update locally you shouldn't see any changes if you later do a conda update for wfc3tools. Anyone who wants the update today can pip install from the git master right now, or just do the update after I change the version in astroconda.

@bourque
Copy link
Author

bourque commented Nov 17, 2016

@sosey great, thanks!

@sosey sosey closed this as completed May 11, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants