sage-test doesn't handle all of sage-doctest's return values

[wjp]

The sage-doctest script returns some status info in its exit code, like if it was aborted with a KeyboardInterrupt. The sage-ptest script interprets this information, but sage-test mostly ignores it.

One symptom is that Ctrl-C-ing a sage -t run of multiple files doesn't work.

Component: doctest coverage

Author: Willem Jan Palenstijn

Reviewer: Alex Ghitza

Merged: sage-4.3.2.alpha0

Issue created by migration from https://trac.sagemath.org/ticket/7995

[wjp]

comment:2

Patch attached.

In the future we may want to merge the sage-ptest and sage-test scripts entirely. They have a lot of duplicate code and minor unnecessary differences.

[wjp]

Attachment: scripts_7995_sage-test_error_handling.patch.gz

[nthiery]

comment:3

I don't know much the doctest framework. But the code looks good, and this removes two seriously annoying misfeatures of doctests. Thanks so much for scratching this itch for me!

Quick remarks:

• When there is an exception in the doctesting framework; will we get
  a traceback? That would be most handy!

• in "... Check the process exit codes ...": codes -> code?

• The following two comments seem contradictory. Or am I
  misunderstanding something?

# Return value in process exit code:
# 0: all tests passed
# 1: file not found
# 2: KeyboardInterrupt
# 3: doctest process was terminated by a signal
# 4: the doctesting framework raised an exception
# 100: failed doctests
####################################################################

def test_code(filename):
# Return value in the doctest process exit code:
# 0: everything passed
# 1-253: number of failed doctests
# 254: >= 254 doctests failed
# 255: exception raised by doctesting framework

[nthiery]

comment:4

Sorry, I screwed up my citation. Here it is again.

• The following two comments seem contradictory. Or am I
  misunderstanding something?

 # Return value in process exit code: 
 # 0: all tests passed 
 # 1: file not found 
 # 2: KeyboardInterrupt 
 # 3: doctest process was terminated by a signal 
 # 4: the doctesting framework raised an exception 
 # 100: failed doctests 
 #################################################################### 

 def test_code(filename): 
     # Return value in the doctest process exit code: 
     # 0: everything passed 
     # 1-253: number of failed doctests 
     # 254: >= 254 doctests failed 
     # 255: exception raised by doctesting framework 

[nthiery]

comment:5

Replying to @nthiery:

Sorry, I screwed up my citation. Here it is again.

Oops again. This remark is actually about #7993. I was looking at both, and got confused.

[wjp]

comment:6

Thanks for the feedback.

The sage-doctest script generates a new python file that runs the actual doctests, and runs this script in a seperate thread. The second block of process exit codes are for this 'inner' doctest process. I'll clarify the confusing comment (and fix the typo you mentioned).

As for the exception: you can get a traceback if you re-run with the -verbose. Do you think we should show them by default?

[wjp]

comment:7

I attached a new patch to #7993 that changes the exit code comments.

[aghitza]

Author: Willem Jan Palenstijn

[aghitza]

comment:9

I played with this a bit and it works well.

[aghitza]

Reviewer: Alex Ghitza

[sagetrac-mvngu]

comment:10

Merged in the script repository.

[sagetrac-mvngu]

Merged: sage-4.3.2.alpha0

[wjp]

comment:12

I tested this, and it worked for me, as well as matching what the python docs say it does on unix systems. Does it not work on your system?

The python docs for os.system:

On Unix, the return value is the exit status of the process encoded in
the format specified for wait().

And for os.wait:

Wait for completion of a child process, and return a tuple containing
its pid and exit status indication: a 16-bit number, whose low byte is
the signal number that killed the process, and whose high byte is the
exit status (if the signal number is zero); the high bit of the low byte
is set if a core file was produced. Availability: Unix.

[jhpalmieri]

comment:13

See #8641 for a followup.

[jhpalmieri]

comment:15

I put in a few print statements, like Dan did:

    print err,
    err = err // 256
    print err

On my iMac running OS X 10.6, when I hit ctrl-C, I see

515 2

which looks okay.

On sage.math, I see

2 0

which doesn't.