Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coverage with Pkg.test causes infinite stacktrace on Appveyor #21289

Closed
omus opened this issue Apr 5, 2017 · 10 comments
Closed

Coverage with Pkg.test causes infinite stacktrace on Appveyor #21289

omus opened this issue Apr 5, 2017 · 10 comments
Labels
regression Regression in behavior compared to a previous version system:windows Affects only Windows
Milestone

Comments

@omus
Copy link
Member

omus commented Apr 5, 2017

I was looking into running coverage on Appveyor and got Julia 0.4 to work but on Julia 0.5.1 or the Julia nightly I ended up encountering the following stack trace:

[00:01:30] C:\projects\julia\bin\julia -e "Pkg.test(\"TimeZones\", coverage=true)"
[00:01:31] INFO: Testing TimeZones
[00:01:32] WARNING: Julia not started with `--compilecache=no`. Disabling tests that require Mocking
[00:01:34] INFO: Extracting tzdata archive
[00:01:55] fatal: error thrown and no exception handler available.
[00:01:55] ZNSt6localeaSERKS_ at C:\projects\julia\bin\libstdc++-6.dll (unknown line)
[00:01:55] jl_write_coverage_data at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\codegen.cpp:1492
[00:01:55] jl_atexit_hook at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\init.c:237
[00:01:55] wmain at /home/Administrator/buildbot/slave/package_win6_2-x86/build/ui\repl.c:244
[00:01:55] __tmainCRTStartup at /usr/src/debug/mingw64-i686-runtime-4.0.4-1/crt\crtexe.c:329
[00:01:55] BaseThreadInitThunk at C:\windows\SYSTEM32\KERNEL32.DLL (unknown line)
[00:01:55] RtlInitializeExceptionChain at C:\windows\SYSTEM32\ntdll.dll (unknown line)
[00:01:55] RtlInitializeExceptionChain at C:\windows\SYSTEM32\ntdll.dll (unknown line)
[00:01:55] ZNSt6localeaSERKS_ at C:\projects\julia\bin\libstdc++-6.dll (unknown line)
[00:01:55] jl_write_coverage_data at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\codegen.cpp:1492
[00:01:55] jl_atexit_hook at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\init.c:237
[00:01:55] jl_exit at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\jl_uv.c:550
[00:01:55] throw_internal at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\task.c:540
[00:01:55] jl_rethrow at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\task.c:558
[00:01:55] ZNSt6localeC2Ev at C:\projects\julia\bin\libstdc++-6.dll (unknown line)
[00:01:55] jl_write_coverage_data at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\codegen.cpp:1492
[00:01:55] jl_atexit_hook at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\init.c:237
[00:01:55] wmain at /home/Administrator/buildbot/slave/package_win6_2-x86/build/ui\repl.c:244
[00:01:55] __tmainCRTStartup at /usr/src/debug/mingw64-i686-runtime-4.0.4-1/crt\crtexe.c:329
[00:01:55] BaseThreadInitThunk at C:\windows\SYSTEM32\KERNEL32.DLL (unknown line)
[00:01:55] RtlInitializeExceptionChain at C:\windows\SYSTEM32\ntdll.dll (unknown line)
[00:01:55] RtlInitializeExceptionChain at C:\windows\SYSTEM32\ntdll.dll (unknown line)
[00:01:55] ZNSt6localeaSERKS_ at C:\projects\julia\bin\libstdc++-6.dll (unknown line)
[00:01:55] jl_write_coverage_data at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\codegen.cpp:1492
[00:01:55] jl_atexit_hook at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\init.c:237
[00:01:55] jl_exit at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\jl_uv.c:550
[00:01:55] throw_internal at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\task.c:540
[00:01:55] jl_rethrow at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\task.c:558
[00:01:55] ZNSt6localeC2Ev at C:\projects\julia\bin\libstdc++-6.dll (unknown line)
[00:01:55] jl_write_coverage_data at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\codegen.cpp:1492
[00:01:55] jl_atexit_hook at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\init.c:237
[00:01:55] jl_exit at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\jl_uv.c:550
[00:01:55] throw_internal at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\task.c:540
[00:01:55] jl_rethrow at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\task.c:558
[00:01:55] ZNSt6localeC2Ev at C:\projects\julia\bin\libstdc++-6.dll (unknown line)
[00:01:55] jl_write_coverage_data at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\codegen.cpp:1492
[00:01:55] jl_atexit_hook at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\init.c:237
[00:01:55] wmain at /home/Administrator/buildbot/slave/package_win6_2-x86/build/ui\repl.c:244
[00:01:55] __tmainCRTStartup at /usr/src/debug/mingw64-i686-runtime-4.0.4-1/crt\crtexe.c:329
[00:01:55] BaseThreadInitThunk at C:\windows\SYSTEM32\KERNEL32.DLL (unknown line)
[00:01:55] RtlInitializeExceptionChain at C:\windows\SYSTEM32\ntdll.dll (unknown line)
[00:01:55] RtlInitializeExceptionChain at C:\windows\SYSTEM32\ntdll.dll (unknown line)
[00:01:55] ZNSt6localeaSERKS_ at C:\projects\julia\bin\libstdc++-6.dll (unknown line)
[00:01:55] jl_write_coverage_data at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\codegen.cpp:1492
[00:01:55] jl_atexit_hook at /home/Administrator/buildbot/slave/package_win6_2-x86/build/src/home/Administrator/buildbot/slave/package_win6_2-x86/build/src\init.c:237
...

Full log for Julia 0.5.1 (Warning: very large log)

@omus omus added regression Regression in behavior compared to a previous version system:windows Affects only Windows labels Apr 5, 2017
@omus
Copy link
Member Author

omus commented Apr 5, 2017

Tried this against Julia 0.5.1 with URIParser.jl on a Windows VM with similar results (no infinite loop though):

julia> Pkg.test("URIParser", coverage=true)
INFO: Testing URIParser
WARNING: Use of the format user:password is deprecated (rfc3986)

signal (11): SIGSEGV
while loading no file, in expression starting on line 0
crt_sig_handler at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\signals-win.c:89
_gnu_exception_handler at /usr/src/debug/mingw64-x86_64-runtime-4.0.6-1/crt\crt_handler.c:223
_C_specific_handler at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
_chkstk at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
RtlWalkFrameChain at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
KiUserExceptionDispatcher at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
ZNSt6localeaSERKS_ at C:\Users\omus\AppData\Local\Julia-0.5.1\bin\libstdc++-6.dll (unknown line)
ZNSt8ios_base7_M_initEv at C:\Users\omus\AppData\Local\Julia-0.5.1\bin\libstdc++-6.dll (unknown line)
ZNSt9basic_iosIcSt11char_traitsIcEE4initEPSt15basic_streambufIcS1_E at C:\Users\omus\AppData\Local\Julia-0.5.1\bin\libstdc++-6.dll (unknown line)
basic_ifstream at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/usr/lib/gcc/x86_64-w64-mingw32/4.9.2/include/c++\fstream:473 [inlined]
write_log_data at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\codegen.cpp:1448
jl_write_coverage_data at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\codegen.cpp:1492
jl_atexit_hook at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\init.c:237
wmain at /home/Administrator/buildbot/slave/package_win6_2-x64/build/ui\repl.c:244
__tmainCRTStartup at /usr/src/debug/mingw64-x86_64-runtime-4.0.6-1/crt\crtexe.c:329
mainCRTStartup at /usr/src/debug/mingw64-x86_64-runtime-4.0.6-1/crt\crtexe.c:212
BaseThreadInitThunk at C:\Windows\system32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
Allocations: 1748127 (Pool: 1747192; Big: 935); GC: 0
==================================================[ ERROR: URIParser ]==================================================

failed process: Process(`'C:\Users\omus\AppData\Local\Julia-0.5.1\bin\julia' -Cx86-64 '-JC:\Users\omus\AppData\Local\Julia-0.5.1\lib\julia\sys.dll' --compile=yes --depwarn=yes --check-bounds=yes --code-coverage=user --inline=no --color=yes --compilecache=yes 'C:\Users\omus\.julia\v0.5\URIParser\test\runtests.jl'`, ProcessExited(3)) [3]

========================================================================================================================
ERROR: URIParser had test errors
 in #test#61(::Bool, ::Function, ::Array{AbstractString,1}) at .\pkg\entry.jl:749
 in (::Base.Pkg.Entry.#kw##test)(::Array{Any,1}, ::Base.Pkg.Entry.#test, ::Array{AbstractString,1}) at .\<missing>:0
 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#test,Tuple{Array{AbstractString,1}}})() at .\pkg\dir.jl:31
 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#test,Tuple{Array{AbstractString,1}}}, ::String) at .\file.jl:48
 in #cd#1(::Array{Any,1}, ::Function, ::Function, ::Array{AbstractString,1}, ::Vararg{Array{AbstractString,1},N}) at .\pkg\dir.jl:31
 in (::Base.Pkg.Dir.#kw##cd)(::Array{Any,1}, ::Base.Pkg.Dir.#cd, ::Function, ::Array{AbstractString,1}, ::Vararg{Array{AbstractString,1},N}) at .\<missing>:0
 in #test#3(::Bool, ::Function, ::String, ::Vararg{String,N}) at .\pkg\pkg.jl:258
 in (::Base.Pkg.#kw##test)(::Array{Any,1}, ::Base.Pkg.#test, ::String, ::Vararg{String,N}) at .\<missing>:0

@omus omus changed the title Coverage with Pkg.test causes infinite stacktrace on Windows Coverage with Pkg.test causes infinite stacktrace on Appveyor Apr 5, 2017
@christopher-dG
Copy link
Member

This appears to be fixed for 0.5.1, still broken for nightly.

https://ci.appveyor.com/project/christopher-dG/appveyortesting-jl/build/1.0.23

@tkelman
Copy link
Contributor

tkelman commented May 13, 2017

I was able to reproduce this locally. Happens as Julia is exiting, inside libstdc++ https://gist.github.com/028e9ade00a4470d2b1574c897f5cb0f

It doesn't happen in a basic source build unless you copy the dll's over from make win-extras (which we do in the binaries, replacing the cygwin-to-mingw cross-compiler's gcc libraries with the ones from opensuse for winrpm package compatibility). This is probably because our windows buildbots have been intentionally holding back to GCC 4.9 to avoid a bug in GCC 5.1-5.4 and 6.1-6.3 (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77333) that was affecting the build of LLVM. We're actually kind of lucky it's been working at all and this is the most visible place it's gone wrong recently.

The cygwin cross-compiler maintainer did rebuild a version of 5.4.0 including an early version of the GCC patch (not the final one that got committed upstream though), so it might be safe to upgrade the buildbots to that now. I'm going to do some tests in some from-scratch VM's first though, since it's a bit complicated to downgrade back to cygwin package versions other than the most recent 2 builds.

@tkelman
Copy link
Contributor

tkelman commented May 14, 2017

Building with the latest GCC 5.4 cross-compiler from Cygwin seems to work in basic testing, and doesn't appear to segfault on code coverage even if the opensuse dll's are copied from dist-extras into usr/bin. So maybe we should try upgrading them on the Windows buildbots. I'm a little hesitant because I'm not sure how we could reverse that.

@staticfloat did you try at any point to set the cygwin install provisioning scripts to point to cygwin time machine, is that possible? That should allow us to get back to 4.9 if something goes horribly wrong with 5.4.

@staticfloat
Copy link
Member

Every time I tried to use the cygwin time machine, the network connection of that server has been so flaky that it took me days to get the packages I needed. In theory it works, in practice, we should maybe just download the packages and shove them onto S3 somewhere for manual installation if worst comes to worst.

The new buildbots will be running 5.4.

@tkelman
Copy link
Contributor

tkelman commented May 14, 2017

How long ago was the last time you tried? They may have upgraded the server and bandwidth recently.

I have copies of 4.9 backed up locally, but it's difficult to install them "properly" without messing with the setup.ini file that cygwin's package manager uses.

I wonder if we could upload nightlies from the new buildbots so packages could test 5.4-built windows binaries? Might take a while for Windows issues to get noticed by people on appveyor for lower-traffic packages.

@tkelman
Copy link
Contributor

tkelman commented Jun 7, 2017

We needed to upgrade to gcc 5.4 on the old buildbots for #22197 to not break winrpm packages, so I did that. Seems to be working okay on nightlies I think, and when 0.6.0-rc3 goes up later today it will be built using gcc 5.4 so hopefully shouldn't have this problem.

@omus
Copy link
Member Author

omus commented Jun 7, 2017

The nightly no longer has this issue. I'll verify that 0.6.0-rc3 is also fixed when it comes out.

@tkelman
Copy link
Contributor

tkelman commented Jun 7, 2017

optimistically closing, but reopen if I'm wrong

(this is also complicated to test on this repo's appveyor, since it does a source build in a configuration where this wouldn't show up directly)

@omus
Copy link
Member Author

omus commented Jun 8, 2017

Julia 0.6.0-rc3.0 is also fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
regression Regression in behavior compared to a previous version system:windows Affects only Windows
Projects
None yet
Development

No branches or pull requests

5 participants