Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Configure Python with pydebug when SAGE_DEBUG is set #13864

Closed
jpflori opened this issue Dec 26, 2012 · 156 comments
Closed

Configure Python with pydebug when SAGE_DEBUG is set #13864

jpflori opened this issue Dec 26, 2012 · 156 comments
Assignees
@jasongrout
Labels
p: major / 3
Milestone
sage-5.7

Comments

@jpflori
Copy link
Contributor

jpflori commented Dec 26, 2012

Instructions for getting a debug version of Python:

This gives you a Sage version that passes all tests, even with MALLOC_CHECK_=3.

Depends on #10352
Depends on #13832
Depends on #13865
Depends on #13867
Depends on #13868
Depends on #13876
Depends on #13878
Depends on #13882
Depends on #13883
Depends on #13889
Depends on #13896
Depends on #13921
Depends on #14059
Depends on #14067

CC: @vbraun @simon-king-jena

Component: misc

Keywords: python spkg

Reviewer: Volker Braun, Simon King, Jean-Pierre Flori

Issue created by migration from https://trac.sagemath.org/ticket/13864
@jpflori jpflori added this to the sage-5.6 milestone Dec 26, 2012
@jpflori
Copy link
Contributor Author

jpflori commented Dec 26, 2012

Work Issues: tag problem

@jpflori
Copy link
Contributor Author

jpflori commented Dec 26, 2012

Changed work issues from tag problem to none

@vbraun
Copy link
Member

vbraun commented Dec 26, 2012

Reviewer: Volker Braun

@vbraun
Copy link
Member

vbraun commented Dec 26, 2012

Dependencies: #13865

@vbraun
Copy link
Member

vbraun commented Dec 26, 2012

comment:5

Looks good to me! I'm adding my proposed change to the documentation as a prerequisite to not deviate from whats documented...

@simon-king-jena
Copy link
Member

comment:6

Sage claims that the spkg built successfully. However, I see things like

/home/simon/SAGE/debug/sage-5.5.rc0/spkg/build/python-2.7.3.p4/src/Modules/_curses_panel.c:17:19: fatal error: panel.h: Datei oder Verzeichnis nicht gefunden
compilation terminated.

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             bsddb185           dbm             
dl                 gdbm               imageop         
sunaudiodev                                           
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

in the install log. Is that normal?

@jpflori
Copy link
Contributor Author

jpflori commented Dec 26, 2012

comment:7

Yes it is (or at least I always had this behavior).
I think these "optional" modules are only built if the needed prerequisites are present, but are not really required to get a working Python.

@jdemeyer
Copy link
Contributor

comment:8

Replying to @jpflori:

I think these "optional" modules are only built if the needed prerequisites are present, but are not really required to get a working Python.

Exactly. Some of these optional modules (like "crypt" for example) are required for Sage, that's why we try to import them in Python's spkg-install.

@simon-king-jena
Copy link
Member

comment:9

One observation, potentially related with what we discussed on #715:

Starting python and importing sage works:

> ./sage -python
Python 2.7.3 (default, Dec 26 2012, 19:09:45) 
[GCC 4.6.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sage
[39290 refs]

However, importing sage.categories.map.Map does not work:

>>> from sage.categories.map import Map
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name Map
[102454 refs]

I don't know whether this should work, but anyway: It doesn't.

@jpflori
Copy link
Contributor Author

jpflori commented Dec 26, 2012

comment:10

I'd say (although I don't know why) it is because it is a Cython module.
Did you try with a plain Python build?

@vbraun
Copy link
Member

vbraun commented Dec 26, 2012

comment:11

I made a gdb spkg at #13866 so you can get a working debugger with the Python debug build.

@vbraun
Copy link
Member

vbraun commented Dec 26, 2012

comment:12

Some debugging hint: Use python -v to print imports as they happen:

[vbraun@volker-desktop sage-5.6.beta0]$ ./sage -python -v
...
>>> import sage.all_cmdline
...
import sage.rings.padics.padic_ext_element # dynamically loaded from /home/vbraun/opt/sage-5.6.beta0/local/lib/python2.7/site-packages/sage/rings/padics/padic_ext_element.so
Fatal Python error: UNREF invalid object
/home/vbraun/opt/sage-5.6.beta0/local/lib/libcsage.so(print_backtrace+0x31)[0x7f0c88f3120d]
/home/vbraun/opt/sage-5.6.beta0/local/lib/libcsage.so(sigdie+0x14)[0x7f0c88f3123f]
/home/vbraun/opt/sage-5.6.beta0/local/lib/libcsage.so(sage_signal_handler+0x1da)[0x7f0c88f30deb]

@jpflori
Copy link
Contributor Author

jpflori commented Dec 26, 2012

comment:13

Replying to @vbraun:

Some debugging hint: Use python -v to print imports as they happen:

[vbraun@volker-desktop sage-5.6.beta0]$ ./sage -python -v
...
>>> import sage.all_cmdline
...
import sage.rings.padics.padic_ext_element # dynamically loaded from /home/vbraun/opt/sage-5.6.beta0/local/lib/python2.7/site-packages/sage/rings/padics/padic_ext_element.so
Fatal Python error: UNREF invalid object
/home/vbraun/opt/sage-5.6.beta0/local/lib/libcsage.so(print_backtrace+0x31)[0x7f0c88f3120d]
/home/vbraun/opt/sage-5.6.beta0/local/lib/libcsage.so(sigdie+0x14)[0x7f0c88f3123f]
/home/vbraun/opt/sage-5.6.beta0/local/lib/libcsage.so(sage_signal_handler+0x1da)[0x7f0c88f30deb]

Could you add this to the dev doc in #13865, or here?

@simon-king-jena
Copy link
Member

comment:14

Thank you! I was just about to ask how one can see what stuff is being imported. So, -v it is...

@vbraun
Copy link
Member

vbraun commented Dec 26, 2012

comment:15

Well ideally we'd ship Sage in a state where it at least starts up with a debug build. If I find something thats particularly useful in debugging then I'll document it. For now:

(sage-sh) vbraun@volker-desktop:sage-5.6.beta0$ gdb python
...
(gdb) run -v
...
>>> import sage.all_cmdline
...
Fatal Python error: UNREF invalid object

Program received signal SIGABRT, Aborted.
0x0000003da9835ba5 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x0000003da9835ba5 in raise () from /lib64/libc.so.6
#1  0x0000003da9837358 in abort () from /lib64/libc.so.6
#2  0x00007ffff7d1a63b in Py_FatalError (msg=0x7ffff7d63eb1 "UNREF invalid object")
    at Python/pythonrun.c:1677
#3  0x00007ffff7c49f18 in _Py_ForgetReference (op=0x21194c0) at Objects/object.c:2223
#4  0x00007ffff7c49f6b in _Py_Dealloc (op=0x21194c0) at Objects/object.c:2242
#5  0x00007fffd9de6e7c in initpadic_ZZ_pX_element () at sage/rings/padics/padic_ZZ_pX_element.cpp:9634
#6  0x00007ffff7d0a496 in _PyImport_LoadDynamicModule (
    name=0x7ffffffe4360 "sage.rings.padics.padic_ZZ_pX_element", 
    pathname=0x7ffffffe3260 "/home/vbraun/opt/sage-5.6.beta0/local/lib/python2.7/site-packages/sage/rings/padics/padic_ZZ_pX_element.so", fp=0x20d23b0) at ./Python/importdl.c:53
...
(gdb) frame 5
#5  0x00007fffd9de6e7c in initpadic_ZZ_pX_element () at sage/rings/padics/padic_ZZ_pX_element.cpp:9634
9634	  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
(gdb) l
9629	  __Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0;
9630	  __pyx_t_2 = __Pyx_GetName(__pyx_m, __pyx_n_s__one); if (unlikely(!__pyx_t_2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 34; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
9631	  __Pyx_GOTREF(__pyx_t_2);
9632	  __pyx_t_3 = PyNumber_Add(__pyx_t_1, __pyx_t_2); if (unlikely(!__pyx_t_3)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 34; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
9633	  __Pyx_GOTREF(__pyx_t_3);
9634	  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
9635	  __Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0;
9636	  if (PyObject_SetAttr(__pyx_m, __pyx_n_s__big, __pyx_t_3) < 0) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 34; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
9637	  __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;

@jpflori
Copy link
Contributor Author

jpflori commented Dec 26, 2012

comment:16

See my long lonely rant at sage-devel https://groups.google.com/d/topic/sage-devel/Wt7uxbDkh_A/discussion, especially the links:

@simon-king-jena
Copy link
Member

comment:17

Interestingly, the error on my laptop looks different from what Volker reported in comment:15:

(sage-sh) simon@linux-sqwp:sage-5.5.rc0$ gdb python
...
(gdb) run -v
...
>>> import sage.all_cmdline
...
import sage.libs.pari.gen # dynamically loaded from /home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/libs/pari/gen.so
dlopen("/home/simon/SAGE/debug/sage-5.5.rc0/local/lib/python2.7/site-packages/sage/rings/integer.so", 2);
python: Modules/gcmodule.c:326: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6d95d95 in raise () from /lib64/libc.so.6

Hence: No Fatal Python error: UNREF invalid object.
Further:

(gdb) bt
#0  0x00007ffff6d95d95 in raise () from /lib64/libc.so.6
#1  0x00007ffff6d972ab in abort () from /lib64/libc.so.6
#2  0x00007ffff6d8e8fe in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff6d8e9a2 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff7b12a19 in visit_decref (op=0x1462f40, data=0x0) at Modules/gcmodule.c:326
#5  0x00007fffeecdcfe1 in __pyx_tp_traverse_4sage_9structure_15category_object_CategoryObject (o=0x14b7d30, v=0x7ffff7b1295e <visit_decref>, a=0x0) at sage/structure/category_object.c:8971
#6  0x00007fffeef4724f in __pyx_tp_traverse_4sage_9structure_6parent_Parent (o=0x14b7d30, v=0x7ffff7b1295e <visit_decref>, a=0x0) at sage/structure/parent.c:21293
#7  0x00007fffef18a97c in __pyx_tp_traverse_4sage_9structure_10parent_old_Parent (o=0x14b7d30, v=0x7ffff7b1295e <visit_decref>, a=0x0) at sage/structure/parent_old.c:7253
#8  0x00007fffef39ab57 in __pyx_tp_traverse_4sage_9structure_11parent_base_ParentWithBase (o=0x14b7d30, v=0x7ffff7b1295e <visit_decref>, a=0x0) at sage/structure/parent_base.c:1892
#9  0x00007ffff7a4c572 in subtype_traverse (self=0x14b7d30, visit=0x7ffff7b1295e <visit_decref>, arg=0x0) at Objects/typeobject.c:838
...
#32 0x00007fffecc2bcbe in __Pyx_ImportModule (name=0x7fffecc2fc6c "sage.rings.integer") at sage/rings/complex_double.c:18370
#33 0x00007fffecc2b951 in __Pyx_ImportType (module_name=0x7fffecc2fc6c "sage.rings.integer", class_name=0x7fffecc2fc64 "Integer", size=64, strict=1) at sage/rings/complex_double.c:18319
#34 0x00007fffecc25b46 in initcomplex_double () at sage/rings/complex_double.c:16572
#35 0x00007ffff7ae9282 in _PyImport_LoadDynamicModule (name=0x7ffffffed660 "sage.rings.complex_double", 
...

So, "Volker's" error is in padic_ZZ_pX_element, but mine is in sage.rings.integer. Moreover, my gdb backtrace is a lot shorter, just 109 lines.

By the way:

>>> from sage.rings.integer_ring import IntegerRing

fails with an import error -- but there is no crash! So, there seems to be something else wrong, not just the refcount.

@jpflori
Copy link
Contributor Author

jpflori commented Dec 26, 2012

comment:18

What you get is similar to what I get (without gdb).
Although the failing place is different as I call ./sage -c "1+1" which must import something else (I'd say sage/all.py) as all_cmdline.py.

Not sure why Volker's output differs.

Anyway, I think we should now focus on trying all of this with #13832.
(Don't forget to rebuild the Sage library after upgrading Cython.)

@simon-king-jena
Copy link
Member

comment:19

Replying to @jpflori:

See my long lonely rant at sage-devel https://groups.google.com/d/topic/sage-devel/Wt7uxbDkh_A/discussion, especially the links:

I don't find these links in the discussion at sage-devel, but indeed the announcement on cython-users looks promising! I am now trying to install the cython spkg from #13832. I guess after installing it I have to do sage -ba, right?

@simon-king-jena
Copy link
Member

comment:117

However, even with #13878, the following problem with pari/gp persists:

sage -t -force_lib "devel/sage/sage/schemes/plane_conics/con_rational_field.py"
**********************************************************************
File "/scratch/sking/sage-5.6.beta2/devel/sage/sage/schemes/plane_conics/con_rational_field.py", line 130:
    sage: C.has_rational_point(point = True)
Exception raised:
    Traceback (most recent call last):
      File "/scratch/sking/sage-5.6.beta2/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/scratch/sking/sage-5.6.beta2/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/scratch/sking/sage-5.6.beta2/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_3[10]>", line 1, in <module>
        C.has_rational_point(point = True)###line 130:
    sage: C.has_rational_point(point = True)
      File "/scratch/sking/sage-5.6.beta2/local/lib/python/site-packages/sage/schemes/plane_conics/con_rational_field.py", line 174, in has_rational_point
        pt = qfsolve(M)
      File "/scratch/sking/sage-5.6.beta2/local/lib/python/site-packages/sage/quadratic_forms/qfsolve.py", line 100, in qfsolve
        ret = pari(gp('Qfsolve(%s)' % G._pari_()))
      File "/scratch/sking/sage-5.6.beta2/local/lib/python/site-packages/sage/interfaces/interface.py", line 195, in __call__
        return cls(self, x, name=name)
      File "/scratch/sking/sage-5.6.beta2/local/lib/python/site-packages/sage/interfaces/expect.py", line 1311, in __init__
        raise TypeError, x
    TypeError: Error executing code in GP:
    CODE:
        sage[4]=Qfsolve([1, 0, 0; 0, 113922743, 0; 0, 0, -310146482690273725409]);
    PARI/GP ERROR:
      ***   at top-level: sage[4]=Qfsolve([1,0,0;0,113
      ***                         ^--------------------
      ***   in function Qfsolve: ... du determinant"));factD=factor(-abs(2*d));if
      ***                                                    ^--------------------
      *** factor: bug in PARI/GP (Segmentation Fault), please report
**********************************************************************
1 items had failures:
   1 of  18 in __main__.example_3
***Test Failed*** 1 failures.
For whitespace errors, see the file /Users/SimonKing/.sage//tmp/con_rational_field_24352.py
         [7.4 s]
 
----------------------------------------------------------------------
The following tests failed:


        sage -t -force_lib "devel/sage/sage/schemes/plane_conics/con_rational_field.py"

Is that a known problem? I sense that we will soon have another dependency...

@simon-king-jena
Copy link
Member

comment:118

The total result of make ptest on bsd.math is:

        sage -t  -force_lib devel/sage/doc/en/bordeaux_2008/l_series.rst # 5 doctests failed
        sage -t  -force_lib devel/sage/doc/en/bordeaux_2008/elliptic_curves.rst # 7 doctests failed
        sage -t  -force_lib devel/sage/doc/en/reference/coercion.rst # 2 doctests failed
        sage -t  -force_lib devel/sage/sage/interfaces/genus2reduction.py # 14 doctests failed
        sage -t  -force_lib devel/sage/sage/interfaces/interface.py # 2 doctests failed
        sage -t  -force_lib devel/sage/sage/interfaces/gp.py # 8 doctests failed
        sage -t  -force_lib devel/sage/sage/lfunctions/lcalc.py # 2 doctests failed
        sage -t  -force_lib devel/sage/sage/lfunctions/dokchitser.py # 45 doctests failed
        sage -t  -force_lib devel/sage/sage/misc/misc.py # 3 doctests failed
        sage -t  -force_lib devel/sage/sage/misc/sage_eval.py # 1 doctests failed
        sage -t  -force_lib devel/sage/sage/modular/buzzard.py # 2 doctests failed
        sage -t  -force_lib devel/sage/sage/modular/modform/eis_series.py # 6 doctests failed
        sage -t  -force_lib devel/sage/sage/modular/modform/element.py # 14 doctests failed
        sage -t  -force_lib devel/sage/sage/plot/line.py # 1 doctests failed
        sage -t  -force_lib devel/sage/sage/quadratic_forms/qfsolve.py # 1 doctests failed
        sage -t  -force_lib devel/sage/sage/rings/number_field/number_field.py # 5 doctests failed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/BSD.py # 4 doctests failed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/ell_number_field.py # 13 doctests failed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/gp_simon.py # 3 doctests failed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/lseries_ell.py # 8 doctests failed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py # 17 doctests failed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/period_lattice.py # 3 doctests failed
        sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/sha_tate.py # 11 doctests failed
        sage -t  -force_lib devel/sage/sage/schemes/plane_conics/con_number_field.py # 1 doctests failed
        sage -t  -force_lib devel/sage/sage/schemes/plane_conics/con_rational_field.py # 1 doctests failed
        sage -t  -force_lib devel/sage/sage/tests/benchmark.py # 4 doctests failed

So, no crash in Sage, but I guess many (or all?) of these errors could be explained by crashes in PARI/GP.

@simon-king-jena
Copy link
Member

comment:119

I have created a #13921 for the gp crash.

@simon-king-jena
Copy link
Member

Changed dependencies from #10352, #13832, #13865, #13867, #13868, #13876, #13878, #13882, #13883, #13889, #13896 to #10352, #13832, #13865, #13867, #13868, #13876, #13878, #13882, #13883, #13889, #13896, #13921

@jpflori
Copy link
Contributor Author

jpflori commented Jan 29, 2013

comment:120

Just wanted to point out that the original p4 spkg which actually sets pydebug when SAGE_DEBUG is set is not linked from anywhere anymore although the p5 spkg from #13867 is based on it, so let's just merge this second ticket quickly to actually be able to use Python in debug mode with a vanilla Sage.

@simon-king-jena

This comment has been minimized.

Sign in to view
@simon-king-jena
Copy link
Member

comment:121

With sage-5.7.beta2, another dependency is needed, namely #14059. Since several of the other dependencies were already merged, I changed the directions in the ticket description accordingly.

@simon-king-jena
Copy link
Member

Changed dependencies from #10352, #13832, #13865, #13867, #13868, #13876, #13878, #13882, #13883, #13889, #13896, #13921 to #10352, #13832, #13865, #13867, #13868, #13876, #13878, #13882, #13883, #13889, #13896, #13921, #14059

@simon-king-jena
Copy link
Member

comment:122

We are almost there. With sage-5.7.beta2 and the changes indicated in the ticket description and with MALLOC_CHECK_=3, make ptest yields:

        sage -t  -force_lib devel/sage/sage/graphs/generic_graph_pyx.pyx # Killed/crashed
        sage -t  -force_lib devel/sage/doc/en/bordeaux_2008/birds_other.rst # Time out

I don't know if the time-out is meaningful. Is there a ticket already for the generic graph problem?

@jpflori
Copy link
Contributor Author

jpflori commented Feb 6, 2013

comment:123

Not sure the crash is really related to the debug version.
See https://groups.google.com/d/msg/sage-release/nvm8f82Vs-U/CdhZmprersUJ
It might be a bug always present and triggered by MALLOC_CHECK_=3 even wihtout debug Python.

@simon-king-jena
Copy link
Member

Changed dependencies from #10352, #13832, #13865, #13867, #13868, #13876, #13878, #13882, #13883, #13889, #13896, #13921, #14059 to #10352, #13832, #13865, #13867, #13868, #13876, #13878, #13882, #13883, #13889, #13896, #13921, #14059, #14067

@simon-king-jena

This comment has been minimized.

Sign in to view
@simon-king-jena
Copy link
Member

comment:124

Even though in generic_graph is triggered my MALLOC_CHECK_=3 and not by the debug version, I think it won't hurt to add #14067 as a dependency, since running tests with MALLOC_CHECK_=3 is (at least for me) associated with debugging.

Moreover:

sage -t -force_lib "devel/sage/doc/en/bordeaux_2008/birds_other.rst"
         [318.2 s]
 
----------------------------------------------------------------------
All tests passed!

Hence, from my perspective, this ticket can be resolved as "fixed", as soon as all dependencies are merged. Since no patch will be merged from here, I suppose it has no author.

@simon-king-jena
Copy link
Member

Changed reviewer from Volker Braun to Volker Braun, Simon King

@simon-king-jena simon-king-jena added this to the sage-5.7 milestone Feb 6, 2013
@jdemeyer
Copy link
Contributor

jdemeyer commented Feb 9, 2013

Changed work issues from make sure Sage builds and starts to none

@jdemeyer
Copy link
Contributor

jdemeyer commented Feb 9, 2013

Changed reviewer from Volker Braun, Simon King to Volker Braun, Simon King, Jean-Pierre Flori

@jdemeyer
Copy link
Contributor

jdemeyer commented Feb 9, 2013

Changed author from Jean-Pierre Flori to none

@jdemeyer jdemeyer closed this as completed Feb 9, 2013
This was referenced Dec 30, 2012
Closed
Closed
Closed
Closed
@vbraun vbraun mentioned this issue Jan 31, 2014
Closed
This was referenced Apr 8, 2014
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jasongrout jasongrout

Labels
p: major / 3
Projects
None yet
Milestone
sage-5.7
Development

No branches or pull requests
6 participants
@jasongrout @vbraun @jpflori @nbruin @jdemeyer @simon-king-jena