I/O operation on closed file when sage -t --optional=sage,external --logfile is_provided.log #25378

seblabbe · 2018-05-17T06:39:38Z

From 8.3.beta1 post on sage-release:

sage -t src/sage/misc/latex.py --optional=sage,external --logfile a.log
Running doctests with ID 2018-05-17-08-30-16-d8cc67d9.
Git branch: develop
Using --optional=external,sage
External software to be detected: cplex,gurobi,internet,latex,macaulay2,magma,maple,mathematica,matlab,octave,scilab
Doctesting 1 file.
sage -t --warn-long 80.1 src/sage/misc/latex.py
    [315 tests, 1.52 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 1.8 seconds
    cpu time: 0.5 seconds
    cumulative wall time: 1.5 seconds
External software detected for doctesting: latex
Traceback (most recent call last):
  File "/home/slabbe/GitBox/sage/src/bin/sage-runtests", line 127, in <module>
    err = DC.run()
  File "/home/slabbe/GitBox/sage/local/lib/python2.7/site-packages/sage/doctest/control.py", line 1176, in run
    + ','.join(available_software.seen()))
  File "/home/slabbe/GitBox/sage/local/lib/python2.7/site-packages/sage/doctest/control.py", line 583, in log
    self.logger.write(s + end)
  File "/home/slabbe/GitBox/sage/local/lib/python2.7/site-packages/sage/doctest/control.py", line 250, in write
    f.write(x)
ValueError: I/O operation on closed file

This confirms that the last line (External software detected for doctesting: latex) is not written to the log file because of I/O errors:

$ tail a.log
External software to be detected: cplex,gurobi,internet,latex,macaulay2,magma,maple,mathematica,matlab,octave,scilab
Doctesting 1 file.
sage -t --warn-long 80.1 src/sage/misc/latex.py
    [315 tests, 1.52 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 1.8 seconds
    cpu time: 0.5 seconds
    cumulative wall time: 1.5 seconds

The error is not there if external software are not tested:

sage -t src/sage/misc/latex.py --optional=sage --logfile a.log
Running doctests with ID 2018-05-17-08-30-28-7b637155.
Git branch: develop
Using --optional=sage
Doctesting 1 file.
sage -t --warn-long 80.1 src/sage/misc/latex.py
    [310 tests, 1.14 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 1.2 seconds
    cpu time: 0.5 seconds
    cumulative wall time: 1.1 seconds

The error is not there if no log file is provided:

sage -t src/sage/misc/latex.py --optional=sage,external
Running doctests with ID 2018-05-17-08-30-40-21f560b3.
Git branch: develop
Using --optional=external,sage
External software to be detected: cplex,gurobi,internet,latex,macaulay2,magma,maple,mathematica,matlab,octave,scilab
Doctesting 1 file.
sage -t --warn-long 80.1 src/sage/misc/latex.py
    [315 tests, 1.45 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 1.7 seconds
    cpu time: 0.5 seconds
    cumulative wall time: 1.4 seconds
External software detected for doctesting: latex

CC: @vinklein

Component: doctest framework

Keywords: thursdaysbdx

Author: Vincent Klein, Sébastien Labbé

Branch/Commit: 76f0177

Reviewer: Sébastien Labbé, Vincent Klein

Issue created by migration from https://trac.sagemath.org/ticket/25378

The text was updated successfully, but these errors were encountered:

vinklein · 2018-07-05T12:23:47Z

Branch: u/vklein/25378

vinklein · 2018-07-05T12:24:49Z

Commit: 6bccacc

vinklein · 2018-07-05T12:24:49Z

Author: Vincent Klein

sagetrac-git · 2018-07-05T14:28:56Z

Changed commit from 6bccacc to 221a7db

sagetrac-git · 2018-07-05T14:28:56Z

Branch pushed to git repo; I updated commit sha1. New commits:

`221a7db`	`Trac 25378: Don't close logfile before last message`

seblabbe · 2018-07-05T19:49:27Z

comment:8

I found a way to raise the issue from sage:

sage: from sage.doctest.control import DocTestDefaults, DocTestController
sage: DD = DocTestDefaults(optional=set(['sage', 'external']), logfile=tmp_filename())
sage: filename = os.path.join(SAGE_SRC, "sage", "misc", "latex.py")
sage: DC = DocTestController(DD, [filename])
sage: DC.run()

Thus, I added a commit with a doctest + a commit fixing the two pyflakes issues (see patchbot report).

New commits:

`4f2d544`	`25378: adding a doctest`
`4eb2799`	`25378: fixing issues seen by pyflakes`

seblabbe · 2018-07-05T19:49:27Z

Changed branch from u/vklein/25378 to public/25378

seblabbe · 2018-07-05T19:49:27Z

Changed commit from 221a7db to 4eb2799

seblabbe · 2018-07-05T19:49:27Z

Reviewer: Sébastien Labbé

seblabbe · 2018-07-05T19:52:33Z

comment:9

I give positive review to commit 221a7db. I let someone else review my two commits.

seblabbe · 2018-07-06T14:11:27Z

comment:10

Of course, "someone else" may include you Vincent Klein as this is often done, see #19494 :)

vinklein · 2018-07-18T09:35:04Z

Changed reviewer from Sébastien Labbé to Sébastien Labbé, Vincent Klein

vinklein · 2018-07-18T09:35:04Z

comment:11

I m ok for the last two commits.

vinklein · 2018-07-18T09:35:04Z

Changed author from Vincent Klein to Vincent Klein, Sébastien Labbé

vbraun · 2018-07-30T12:19:11Z

comment:12

File "src/sage/doctest/control.py", line 1145, in sage.doctest.control.DocTestController.run
Failed example:
    DC.run()
Expected:
    Running doctests with ID ...
    Using --optional=external,sage
    External software to be detected: ...
    Doctesting 1 file.
    sage -t .../sage/misc/latex.py
        [... tests, ... s]
    ----------------------------------------------------------------------
    All tests passed!
    ----------------------------------------------------------------------
    Total time for all tests: ... seconds
        cpu time: ... seconds
        cumulative wall time: ... seconds
    External software detected for doctesting: ...
    0
Got:
    Running doctests with ID 2018-07-30-14-14-47-59620918.
    Git branch: test
    Using --optional=external,sage
    External software to be detected: cplex,ffmpeg,graphviz,gurobi,imagemagick,internet,latex,macaulay2,magma,maple,mathematica,matlab,octave,pandoc,scilab
    Doctesting 1 file.
    sage -t /mnt/disk/home/release/Sage/src/sage/misc/latex.py
    **********************************************************************
    File "/mnt/disk/home/release/Sage/src/sage/misc/latex.py", line 1548, in sage.misc.latex.Latex.add_package_to_preamble_if_available
    Failed example:
        latex.extra_preamble()       # optional - latex
    Expected:
        '\\usepackage{xypic}\n'
    Got:
        ''
    **********************************************************************
    1 item had failures:
       1 of   5 in sage.misc.latex.Latex.add_package_to_preamble_if_available
        [316 tests, 1 failure, 1.29 s]
    ----------------------------------------------------------------------
    sage -t /mnt/disk/home/release/Sage/src/sage/misc/latex.py  # 1 doctest failed
    ----------------------------------------------------------------------
    Total time for all tests: 1.6 seconds
        cpu time: 0.4 seconds
        cumulative wall time: 1.3 seconds
    External software detected for doctesting: imagemagick,latex
    1
**********************************************************************

videlec · 2018-08-03T19:20:18Z

comment:13

update milestone 8.3 -> 8.4

sagetrac-git · 2018-08-25T17:06:49Z

Branch pushed to git repo; I updated commit sha1. New commits:

`1040475`	`25378: more robust doctest`

sagetrac-git · 2018-08-25T17:06:49Z

Changed commit from 7c7e278 to 1040475

vbraun · 2018-08-26T09:41:00Z

comment:24

On OSX I get lots of failures of the form

**********************************************************************
File "src/sage/interfaces/expect.py", line 763, in sage.interfaces.expect.Expect._eval_line_using_file
Failed example:
    singular._eval_line_using_file('def a=3;')
Exception raised:
    Traceback (most recent call last):
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 650, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1061, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.interfaces.expect.Expect._eval_line_using_file[3]>", line 1, in <module>
        singular._eval_line_using_file('def a=3;')
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 813, in _eval_line_using_file
        self._synchronize()
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1284, in _synchronize
        self._sendstr(cmd)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1226, in _sendstr
        self.quit()
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 597, in quit
        self._expect.close()
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pexpect/pty_spawn.py", line 327, in close
        self.ptyproc.close(force=force)
      File "sage/interfaces/sagespawn.pyx", line 202, in sage.interfaces.sagespawn.SagePtyProcess.close (build/cythonized/sage/interfaces/sagespawn.c:3059)
        self.fileobj.close()
    IOError: [Errno 5] Input/output error
**********************************************************************
File "src/sage/interfaces/expect.py", line 766, in sage.interfaces.expect.Expect._eval_line_using_file
Failed example:
    singular('a')
Exception raised:
    Traceback (most recent call last):
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 650, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1061, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.interfaces.expect.Expect._eval_line_using_file[4]>", line 1, in <module>
        singular('a')
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/singular.py", line 798, in __call__
        return SingularElement(self, type, x, False)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/singular.py", line 1279, in __init__
        self._name = parent._create(value, type)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/singular.py", line 759, in _create
        self.set(type, name, value)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/singular.py", line 702, in set
        self.eval(cmd)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/singular.py", line 655, in eval
        s = Expect.eval(self, x, **kwds)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1354, in eval
        for L in code.split('\n') if L != ''])
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 942, in _eval_line
        self._synchronize()
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1284, in _synchronize
        self._sendstr(cmd)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1226, in _sendstr
        self.quit()
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 597, in quit
        self._expect.close()
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pexpect/pty_spawn.py", line 327, in close
        self.ptyproc.close(force=force)
      File "sage/interfaces/sagespawn.pyx", line 199, in sage.interfaces.sagespawn.SagePtyProcess.close (build/cythonized/sage/interfaces/sagespawn.c:2940)
        self.write(str_to_bytes(self.quit_string))
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/ptyprocess/ptyprocess.py", line 562, in write
        return self._writeb(s, flush=flush)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/ptyprocess/ptyprocess.py", line 552, in _writeb
        n = self.fileobj.write(b)
    ValueError: write to closed file
**********************************************************************
File "src/sage/interfaces/expect.py", line 768, in sage.interfaces.expect.Expect._eval_line_using_file
Failed example:
    singular.eval('quit;')
Exception raised:
    Traceback (most recent call last):
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 650, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1061, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.interfaces.expect.Expect._eval_line_using_file[5]>", line 1, in <module>
        singular.eval('quit;')
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/singular.py", line 655, in eval
        s = Expect.eval(self, x, **kwds)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1354, in eval
        for L in code.split('\n') if L != ''])
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 942, in _eval_line
        self._synchronize()
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1284, in _synchronize
        self._sendstr(cmd)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1226, in _sendstr
        self.quit()
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 597, in quit
        self._expect.close()
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pexpect/pty_spawn.py", line 327, in close
        self.ptyproc.close(force=force)
      File "sage/interfaces/sagespawn.pyx", line 199, in sage.interfaces.sagespawn.SagePtyProcess.close (build/cythonized/sage/interfaces/sagespawn.c:2940)
        self.write(str_to_bytes(self.quit_string))
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/ptyprocess/ptyprocess.py", line 562, in write
        return self._writeb(s, flush=flush)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/ptyprocess/ptyprocess.py", line 552, in _writeb
        n = self.fileobj.write(b)
    ValueError: write to closed file
**********************************************************************

vbraun · 2018-08-26T18:21:29Z

comment:25

Might have been due to other ticket, trying again

vbraun · 2018-08-30T22:25:00Z

Changed branch from public/25378 to 1040475

vbraun · 2018-09-01T07:55:42Z

Changed commit from 1040475 to none

vbraun · 2018-09-01T07:55:42Z

comment:27

**********************************************************************
File "src/sage/doctest/control.py", line 1161, in sage.doctest.control.DocTestController.run
Failed example:
    DC.run()
Expected:
    Running doctests with ID ...
    Using --optional=external,sage
    External software to be detected: ...
    Doctesting 1 file.
    sage -t ....py
        [0 tests, 0.00 s]
    ----------------------------------------------------------------------
    All tests passed!
    ----------------------------------------------------------------------
    Total time for all tests: 0.0 seconds
        cpu time: 0.0 seconds
        cumulative wall time: 0.0 seconds
    External software detected for doctesting:...
    0
Got:
    Running doctests with ID 2018-09-01-03-28-58-9687e9f6.
    Git branch: develop
    Using --optional=external,sage
    External software to be detected: cplex,ffmpeg,graphviz,gurobi,imagemagick,internet,latex,macaulay2,magma,maple,mathematica,matlab,octave,pandoc,scilab
    Doctesting 1 file.
    sage -t /home/buildbot/slave/sage_git/dot_sage/temp/sagebd07_64s02/8118/tmp_5ksam4.py
        [0 tests, 0.00 s]
    ----------------------------------------------------------------------
    All tests passed!
    ----------------------------------------------------------------------
    Total time for all tests: 0.1 seconds
        cpu time: 0.0 seconds
        cumulative wall time: 0.0 seconds
    External software detected for doctesting: 
    0
**********************************************************************
1 item had failures:
   1 of  13 in sage.doctest.control.DocTestController.run
    [206 tests, 1 failure, 4.49 s]
----------------------------------------------------------------------
sage -t --long src/sage/doctest/control.py  # 1 doctest failed
----------------------------------------------------------------------

seblabbe · 2018-09-01T09:46:17Z

Commit: 76f0177

seblabbe · 2018-09-01T09:46:17Z

Changed branch from 1040475 to public/25378

seblabbe · 2018-09-01T09:46:17Z

New commits:

`221a7db`	`Trac 25378: Don't close logfile before last message`
`4f2d544`	`25378: adding a doctest`
`4eb2799`	`25378: fixing issues seen by pyflakes`
`4f8a7ac`	`Merge branch 'public/25378' into 8.4.beta1`
`7c7e278`	`25378: better doctest`
`1040475`	`25378: more robust doctest`
`76f0177`	`25378: doctesting an empty file can take time`

vinklein · 2018-09-04T12:00:18Z

comment:30

Can't the line Git branch: develop cause a doctest error too ?

vinklein · 2018-09-20T08:18:35Z

comment:31

No it can't as end of line ellipsis marker allow to skip lines as regular ellipsis marker.

seblabbe · 2018-09-20T08:43:41Z

comment:32

Exactly, this is what I thought. Thanks for the review. Hopefully everything is okay now. Last 6 patchbots reports are green (or contain unrelated failures with pbori.pyx and magma.py).

Sébastien

vbraun · 2018-09-21T22:20:36Z

Changed branch from public/25378 to 76f0177

seblabbe added this to the sage-8.3 milestone May 17, 2018

seblabbe added c: doctest framework labels May 17, 2018

This comment has been minimized.

Sign in to view

vinklein mannequin added s: needs review and removed s: needs review labels Jul 5, 2018

vinklein mannequin added s: needs review and removed s: needs work labels Jul 5, 2018

vinklein mannequin added s: positive review and removed s: needs review labels Jul 18, 2018

vbraun added s: needs work and removed s: positive review labels Jul 30, 2018

seblabbe added the s: needs review label Aug 25, 2018

vbraun added s: positive review and removed s: needs review labels Aug 25, 2018

vbraun added s: needs work and removed s: positive review labels Aug 26, 2018

vbraun added s: positive review and removed s: needs work labels Aug 26, 2018

vbraun removed the s: positive review label Aug 30, 2018

vbraun closed this as completed Aug 30, 2018

vbraun reopened this Sep 1, 2018

seblabbe added the s: needs review label Sep 1, 2018

vinklein mannequin added s: positive review and removed s: needs review labels Sep 20, 2018

vbraun removed the s: positive review label Sep 21, 2018

vbraun closed this as completed in c7877f5 Sep 21, 2018

seblabbe mentioned this issue Oct 21, 2022

meta-ticket: failing internet/optional doctests #25536

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

I/O operation on closed file when sage -t --optional=sage,external --logfile is_provided.log #25378

I/O operation on closed file when sage -t --optional=sage,external --logfile is_provided.log #25378

seblabbe commented May 17, 2018

This comment has been minimized.

vinklein mannequin commented Jul 5, 2018

vinklein mannequin commented Jul 5, 2018

vinklein mannequin commented Jul 5, 2018

sagetrac-git mannequin commented Jul 5, 2018

sagetrac-git mannequin commented Jul 5, 2018

seblabbe commented Jul 5, 2018

seblabbe commented Jul 5, 2018

seblabbe commented Jul 5, 2018

seblabbe commented Jul 5, 2018

seblabbe commented Jul 5, 2018

seblabbe commented Jul 6, 2018

vinklein mannequin commented Jul 18, 2018

vinklein mannequin commented Jul 18, 2018

vinklein mannequin commented Jul 18, 2018

vbraun commented Jul 30, 2018

videlec commented Aug 3, 2018

sagetrac-git mannequin commented Aug 25, 2018

sagetrac-git mannequin commented Aug 25, 2018

vbraun commented Aug 26, 2018

vbraun commented Aug 26, 2018

vbraun commented Aug 30, 2018

vbraun commented Sep 1, 2018

vbraun commented Sep 1, 2018

seblabbe commented Sep 1, 2018

seblabbe commented Sep 1, 2018

seblabbe commented Sep 1, 2018

vinklein mannequin commented Sep 4, 2018

vinklein mannequin commented Sep 20, 2018

seblabbe commented Sep 20, 2018

vbraun commented Sep 21, 2018

I/O operation on closed file when sage -t --optional=sage,external --logfile is_provided.log #25378

I/O operation on closed file when sage -t --optional=sage,external --logfile is_provided.log #25378

Comments

seblabbe commented May 17, 2018

This comment has been minimized.

vinklein mannequin commented Jul 5, 2018

vinklein mannequin commented Jul 5, 2018

vinklein mannequin commented Jul 5, 2018

sagetrac-git mannequin commented Jul 5, 2018

sagetrac-git mannequin commented Jul 5, 2018

seblabbe commented Jul 5, 2018

seblabbe commented Jul 5, 2018

seblabbe commented Jul 5, 2018

seblabbe commented Jul 5, 2018

seblabbe commented Jul 5, 2018

seblabbe commented Jul 6, 2018

vinklein mannequin commented Jul 18, 2018

vinklein mannequin commented Jul 18, 2018

vinklein mannequin commented Jul 18, 2018

vbraun commented Jul 30, 2018

videlec commented Aug 3, 2018

sagetrac-git mannequin commented Aug 25, 2018

sagetrac-git mannequin commented Aug 25, 2018

vbraun commented Aug 26, 2018

vbraun commented Aug 26, 2018

vbraun commented Aug 30, 2018

vbraun commented Sep 1, 2018

vbraun commented Sep 1, 2018

seblabbe commented Sep 1, 2018

seblabbe commented Sep 1, 2018

seblabbe commented Sep 1, 2018

vinklein mannequin commented Sep 4, 2018

vinklein mannequin commented Sep 20, 2018

seblabbe commented Sep 20, 2018

vbraun commented Sep 21, 2018