Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enzyme broken for Windows on Julia v1.10 #1236

Closed
jeremiedb opened this issue Jan 3, 2024 · 13 comments
Closed

Enzyme broken for Windows on Julia v1.10 #1236

jeremiedb opened this issue Jan 3, 2024 · 13 comments
Labels
bug Something isn't working help wanted Extra attention is needed more information needed windows

Comments

@jeremiedb
Copy link

The following minimal script hangs forever when launched on Windows machine with Julia 1.10. It works fine with Julia 1.9.4 and 1.8.3:

using Enzyme
@info "starting"
f1(x) = x*x
autodiff(Reverse, f1, Active(1.0))
@info "done"
PS C:\Users\jerem\OneDrive\github\ADTests.jl> julia --project=@. --threads=1 .\experiments\debug.jl
[ Info: starting

I noticed some work related to Julia v1.10 (#1231) and that Windows CI on Julia 1.10 is failing, so perhaps this was already on the radar?

@leerosenthalj
Copy link

I am encountering the same issue, also with Julia 1.10 on Windows, with a similar MWE.

@vchuravy vchuravy added bug Something isn't working help wanted Extra attention is needed windows labels Jan 25, 2024
@vchuravy
Copy link
Member

If you can get a backtrace from where the hang is coming from there might be something we can intuit from it.

@ArnoStrouwen
Copy link

Interrupting Enzyme with Ctrl+C is not working.
I created a dump file of the hanging Julia process on Windows, which might contain something useful:


************* Preparing the environment for Debugger Extensions Gallery repositories **************
   ExtensionRepository : Implicit
   UseExperimentalFeatureForNugetShare : true
   AllowNugetExeUpdate : true
   AllowNugetMSCredentialProviderInstall : true
   AllowParallelInitializationOfLocalRepositories : true

   -- Configuring repositories
      ----> Repository : LocalInstalled, Enabled: true
      ----> Repository : UserExtensions, Enabled: true

>>>>>>>>>>>>> Preparing the environment for Debugger Extensions Gallery repositories completed, duration 0.000 seconds

************* Waiting for Debugger Extensions Gallery to Initialize **************

>>>>>>>>>>>>> Waiting for Debugger Extensions Gallery to Initialize completed, duration 0.016 seconds
   ----> Repository : UserExtensions, Enabled: true, Packages count: 0
   ----> Repository : LocalInstalled, Enabled: true, Packages count: 36

Microsoft (R) Windows Debugger Version 10.0.25921.1001 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Users\arno\AppData\Local\Temp\julia (2).DMP]
User Mini Dump File with Full Memory: Only application data is available


************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*
Symbol search path is: srv*
Executable search path is: 
Windows 10 Version 19045 MP (24 procs) Free x64
Product: WinNt, suite: SingleUserTS
Edition build lab: 19041.1.amd64fre.vb_release.191206-1406
Debug session time: Sat Jan 27 19:42:09.000 2024 (UTC + 1:00)
System Uptime: 0 days 1:14:15.392
Process Uptime: 0 days 0:01:26.000
................................................................
..................

+------------------------------------------------------------------------+
| This target supports Hardware-enforced Stack Protection. A HW based    |
| "Shadow Stack" may be available to assist in debugging and analysis.   |
| See aka.ms/userhsp for more info.                                      |
|                                                                        |
| dps @ssp                                                               |
|                                                                        |
+------------------------------------------------------------------------+

For analysis of this file, run !analyze -v
ntdll!NtWaitForSingleObject+0x14:
00007fff`a6e8d064 c3              ret
0:000> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************


KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 1671

    Key  : Analysis.Elapsed.mSec
    Value: 1665

    Key  : Analysis.IO.Other.Mb
    Value: 0

    Key  : Analysis.IO.Read.Mb
    Value: 0

    Key  : Analysis.IO.Write.Mb
    Value: 0

    Key  : Analysis.Init.CPU.mSec
    Value: 342

    Key  : Analysis.Init.Elapsed.mSec
    Value: 110180

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 118

    Key  : Failure.Bucket
    Value: BREAKPOINT_80000003_libwinpthread-1.dll!Unknown

    Key  : Failure.Hash
    Value: {656a880b-7201-663a-ee71-fadf484d9a93}

    Key  : Timeline.OS.Boot.DeltaSec
    Value: 4455

    Key  : Timeline.Process.Start.DeltaSec
    Value: 86

    Key  : WER.OS.Branch
    Value: vb_release

    Key  : WER.OS.Version
    Value: 10.0.19041.1

    Key  : WER.Process.Version
    Value: 1.10.0.0


FILE_IN_CAB:  julia (2).DMP

NTGLOBALFLAG:  0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS:  0

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 0000000000000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

FAULTING_THREAD:  00000920

PROCESS_NAME:  julia.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

EXCEPTION_CODE_STR:  80000003

STACK_TEXT:  
00000099`0b9faca8 00007fff`a47330ce     : 00000289`ce550000 00007fff`a6e147b1 00000289`fd438700 00007fff`a6e15ba1 : ntdll!NtWaitForSingleObject+0x14
00000099`0b9facb0 00007fff`89632e49     : 00000289`d4fb6290 00000000`00000001 00000000`00000000 00000000`00000340 : KERNELBASE!WaitForSingleObjectEx+0x8e
00000099`0b9fad50 00007fff`356a3244     : 00000289`d08eacf0 00000000`00000000 00000000`00000000 00007fff`89633129 : libwinpthread_1!pthread_mutex_lock+0x99
00000099`0b9fada0 00000000`6f51b593     : 00000289`fd4386f0 00000289`fcf0da60 00000289`fd438708 00000099`0b9faf00 : libjulia_codegen!LLVMExtraLPMAddLowerSIMDLoopPass_impl+0x28fb4
00000099`0b9fae00 00000000`6f52cd85     : 00000289`fd438708 00000289`c0103e48 00007fff`356bf5c1 00000000`00000018 : libLLVM_15jl!ZN4llvm3orc14IRCompileLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE+0x243
00000099`0b9faea0 00007fff`355e28a7     : 00000289`bf8fcaa0 00000289`d08e75c0 00000289`bf8fc560 00007fff`a6e147b1 : libLLVM_15jl!ZN4llvm3orc16IRTransformLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE+0x185
00000099`0b9faf60 00000000`6f57200c     : 00000289`d08e7878 00000289`00000000 00000289`d08ea690 00000289`c055d370 : libjulia_codegen!jl_jit_total_bytes_impl+0xe47
00000099`0b9fb030 00000000`6f53a48c     : 00000000`00000018 00000289`d08e7610 00000000`715b6d00 00007fff`a5c99d40 : libLLVM_15jl!ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EE+0xac
00000099`0b9fb110 00000000`6f784fbd     : 00000099`0b9fb1b8 00007fff`3bc5e21c 00000000`00000018 00000289`d08e7610 : libLLVM_15jl!ZN4llvm3orc19MaterializationTask3runEv+0x2c
00000099`0b9fb150 00000000`6f52ab3c     : 00000289`d08f32b0 00000289`bf8fcaa0 00000289`bfa52d88 00000000`00000000 : libLLVM_15jl!ZN4llvm6detail18UniqueFunctionBaseIvJSt10unique_ptrINS_3orc4TaskESt14default_deleteIS4_EEEE8CallImplIPFvS7_EEEvPvRS7_+0x1d
00000099`0b9fb190 00000000`6f52842e     : 00000289`ce552ac0 00000289`caa69410 00000289`d09757a0 00007fff`a6e1b86b : libLLVM_15jl!ZN4llvm3orc16ExecutionSession22dispatchOutstandingMUsEv+0x9c
00000099`0b9fb200 00000000`6f55bc8d     : 00000289`d08e7580 00000099`0b9fb3a8 00000099`0b9fb3b0 00000099`0b9fb3c0 : libLLVM_15jl!ZN4llvm3orc16ExecutionSession17OL_completeLookupESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EESt10shared_ptrINS0_23AsynchronousSymbolQueryEESt8functionIFvRKNS_8DenseMapIPNS0_8JITDylibENS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISF_vEEEENSG_ISD_vEENS_6detail12DenseMapPairISD_SI_EEEEEE+0xb2e
00000099`0b9fb380 00000000`6f529999     : 00000000`00000000 00007fff`a6e10000 00000002`ca0959e8 00007fff`3bc5e21c : libLLVM_15jl!ZN4llvm3orc25InProgressFullLookupState8completeESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EE+0xdd
00000099`0b9fb400 00000000`6f52bd44     : 00000000`015c0657 00000000`000000e2 00000289`fcf238f0 00000000`6f5899dc : libLLVM_15jl!ZN4llvm3orc16ExecutionSession19OL_applyQueryPhase1ESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EENS_5ErrorE+0xb19
00000099`0b9fb550 00000000`6f52c297     : 00000289`d08e7878 00000000`00000000 00000289`d4fa9880 00000289`d4fa9c30 : libLLVM_15jl!ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS8_EENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISI_vEENS_6detail12DenseMapPairISI_SJ_EEEEEEEEESt8functionIFvRKNSH_IS6_NS_8DenseSetISI_SL_EENSK_IS6_vEENSN_IS6_SV_EEEEEE+0x3a4
00000099`0b9fb660 00007fff`355e4cd6     : 00000289`ce540000 00000289`d08e7580 00000099`0b9fb8e0 00000000`00000011 : libLLVM_15jl!ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_15SymbolLookupSetENS0_10LookupKindENS0_11SymbolStateESt8functionIFvRKNS_8DenseMapIS5_NS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISI_vEEEENSJ_IS5_vEENS_6detail12DenseMapPairIS5_SL_EEEEEE+0x1d7
00000099`0b9fb780 00007fff`355eb56a     : 00000289`d08e9708 00000000`00000000 00000000`00000000 00000000`00000030 : libjulia_codegen!jl_jit_total_bytes_impl+0x3276
00000099`0b9fb9f0 00007fff`355ec879     : 00000289`c919a1f8 00000099`0b9fbcf0 00000099`0b9fbd30 00000099`0b9fbd90 : libjulia_codegen!jl_compile_extern_c_impl+0x234a
00000099`0b9fbbb0 00007fff`355ed7eb     : 00000289`d3ff65b0 00000289`bf4d8d00 ffffffff`ffffffff 00000289`c292d1e0 : libjulia_codegen!jl_compile_extern_c_impl+0x3659
00000099`0b9fc3d0 00007fff`3cfb797f     : 00007fff`2d3d66f0 00000289`d07769b8 00000000`0000026b 00007fff`a4760fbb : libjulia_codegen!jl_generate_fptr_impl+0x2ab
00000099`0b9fc4c0 00007fff`3cfbb15f     : 00007fff`7ac64200 00007fff`3cfbb10e 00007fff`00000102 00000000`00000002 : libjulia_internal!jl_normalize_to_compilable_mi+0x1cf
00000099`0b9fc570 00007fff`3cfd626b     : 00000099`0b9fc650 00007fff`3cfd626b 00000000`00000001 00000000`00000030 : libjulia_internal!ijl_apply_generic+0x14f
00000099`0b9fc600 00007fff`3cfd5d9d     : 00000289`fdca8608 00007fff`3d00ce84 00000000`00000002 00000289`d0b00080 : libjulia_internal!ijl_init_restored_module+0x9ab
00000099`0b9fc690 00007fff`3cfd6929     : 00000289`c39f2770 00000099`0b9fc950 00000000`00000302 00000000`00000003 : libjulia_internal!ijl_init_restored_module+0x4dd
00000099`0b9fc730 00007fff`3cfd772f     : 00000289`c39f2770 00000099`0b9fc950 00000289`fd726e86 00000000`00000001 : libjulia_internal!ijl_init_restored_module+0x1069
00000099`0b9fc930 00007fff`3cff306c     : 00000289`c1de4190 00007fff`3d00ce84 00000000`00000038 00000001`00000000 : libjulia_internal!jl_interpret_toplevel_thunk+0xef
00000099`0b9fca10 00007fff`3cff39df     : 00000000`00000005 00000099`0b9fcbe0 00000099`0b9fcb48 00000001`00000010 : libjulia_internal!jl_toplevel_eval_flex+0x35c
00000099`0b9fcb50 00007fff`3cff4a00     : 00000289`c1d97eb0 00000289`c40277f0 00000289`c40277f0 00000289`c1de40b0 : libjulia_internal!jl_toplevel_eval_flex+0xccf
00000099`0b9fcc90 00007fff`287a6e44     : 00007fff`2d3d66f0 00000289`c1de40b0 00007fff`276c7835 00000000`00000000 : libjulia_internal!ijl_toplevel_eval_in+0x90
00000099`0b9fce20 00007fff`292d8ce0     : 00007fff`293bd270 00007fff`2c28e1a0 00000000`00000eb8 00000000`00000002 : sys+0x1286e44
00000099`0b9fd180 00007fff`28054ec4     : 00000099`0b9fd400 00000289`d0b00080 00000289`d0b00080 00000099`0b9fd340 : sys+0x1db8ce0
00000099`0b9fd2c0 00007fff`287a491d     : 00007fff`29314650 00000289`d0b00080 00000289`d45843a0 00000000`00007af8 : sys+0xb34ec4
00000099`0b9fd320 00007fff`28053c8d     : 00007fff`2d86ffd0 00007fff`3d00ce84 00000000`00000001 00007fff`2d8705c0 : sys+0x128491d
00000099`0b9fd3e0 00007fff`27c9c0e3     : 00000000`00000000 00000000`00000000 00000000`00007af8 00000000`00000000 : sys+0xb33c8d
00000099`0b9fd540 00007fff`28f3dcc5     : 00000099`0b9fd6c8 00007fff`2c754dd0 00007fff`319c9850 00007fff`2c8ed8c0 : sys+0x77c0e3
00000099`0b9fd660 00007fff`28f17a08     : 00000099`0b9fd8e0 00007fff`3d280440 00007fff`2a21f670 00000289`d49040a0 : sys+0x1a1dcc5
00000099`0b9fd6a0 00007fff`278b4e8b     : 00000000`00000000 00000289`d0b00080 00000099`0b9fd8d8 00000000`00000002 : sys+0x19f7a08
00000099`0b9fd800 00007fff`3cfc695b     : 00000000`00007af8 00000289`00000006 00000000`00000000 00000000`00000001 : sys+0x394e8b
00000099`0b9fd850 00007fff`292d3f13     : 00000000`00000081 00000000`00000000 00000000`00000080 00000000`00000000 : libjulia_internal!jl_f__call_latest+0x3b
00000099`0b9fd8a0 00007fff`292c5e87     : 00000289`0000000f 00000000`00000000 00000000`00000004 00000000`00000000 : sys+0x1db3f13
00000099`0b9fdc00 00007fff`28f3d6b4     : 00000000`00000000 00000099`0b9ff060 00000099`0b9fefb8 00000099`0b9ff050 : sys+0x1da5e87
00000099`0b9ff060 00007fff`283f9f7d     : 00000289`ce364d18 00000289`d0b00010 00000000`0000004a 00007fff`a4e65ec0 : sys+0x1a1d6b4
00000099`0b9ff370 00007fff`3d02759a     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : sys+0xed9f7d
00000099`0b9ff3b0 00007fff`3d028059     : 00007fff`965e4700 00000099`0b9ff6f8 00000000`00000010 00007fff`965d1d74 : libjulia_internal!ijl_call2+0x36a
00000099`0b9ff6d0 00007ff7`91d11ab2     : 00000289`00000000 00000289`00000000 00000289`ce3541d0 00000099`0b9f0000 : libjulia_internal!jl_repl_entrypoint+0x79
00000099`0b9ff730 00007fff`a4e67344     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : julia+0x1ab2
00000099`0b9ff7f0 00007fff`a6e426b1     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
00000099`0b9ff820 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21


SYMBOL_NAME:  libwinpthread_1+99

MODULE_NAME: libwinpthread_1

IMAGE_NAME:  libwinpthread-1.dll

STACK_COMMAND:  dt ntdll!LdrpLastDllInitializer BaseDllName ; dt ntdll!LdrpFailureData ; ~0s; .ecxr ; kb

FAILURE_BUCKET_ID:  BREAKPOINT_80000003_libwinpthread-1.dll!Unknown

OS_VERSION:  10.0.19041.1

BUILDLAB_STR:  vb_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

IMAGE_VERSION:  1.0.0.0

FAILURE_ID_HASH:  {656a880b-7201-663a-ee71-fadf484d9a93}

Followup:     MachineOwner
---------

@wsmoses
Copy link
Member

wsmoses commented Jan 27, 2024 via email

@Larbino1
Copy link

Ahhh, I must be experiencing the same issue. Hangs on differentiating a simple function on my windows 10 PC, Ctrl+C doesn't work, recently updated to Julia 1.10, before which it was working.

@salbalkus
Copy link

I just started using Enzyme.jl with Julia and I'm experiencing the same issue as well.

@RGonTheNoble
Copy link

RGonTheNoble commented Feb 25, 2024

I tried using it on version 1.10.0 and 1.10.1, on Windows 11 and in both instances it just froze. I was running a simple script used in the example:

f1(x) = x*x
autodiff(Reverse, f1, Active(1.0))

Note that when running it on Kubuntu 22.04 it worked.

@wsmoses
Copy link
Member

wsmoses commented Feb 25, 2024 via email

@wsmoses
Copy link
Member

wsmoses commented Mar 13, 2024

Reduce to a pure bug in LLVM.jl.

Unfortunately nothing we can do here in Enzyme to fix, but feel free to bump LLVM.jl about a fix:

 using LLVM

function cpu_name()
    ccall(:jl_get_cpu_name, String, ())
end

function cpu_features()
    return ccall(:jl_get_cpu_features, String, ())
end

const modstr = """
; ModuleID = 'start'
source_filename = "start"
target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-w64-mingw32"

; Function Attrs: inaccessiblemem_or_argmemonly nofree
declare void @julia.safepoint(i64*) local_unnamed_addr #0

; Function Attrs: alwaysinline nofree
define [1 x double] @fwddiffejulia_identity_1243wrap(double %0, double %1) #1 !dbg !4 {
entry:
  %2 = call {}*** inttoptr (i64 140708411134816 to {}*** ()*)() #5
  %ptls_field3.i = getelementptr inbounds {}**, {}*** %2, i64 2
  %3 = bitcast {}*** %ptls_field3.i to i64***
  %ptls_load45.i = load i64**, i64*** %3, align 8, !tbaa !8, !alias.scope !12, !noalias !15
  %4 = getelementptr inbounds i64*, i64** %ptls_load45.i, i64 2
  %safepoint.i = load i64*, i64** %4, align 8, !tbaa !17, !alias.scope !19, !noalias !22
  fence syncscope("singlethread") seq_cst
  %5 = load volatile i64, i64* %safepoint.i, align 8, !dbg !24
  fence syncscope("singlethread") seq_cst
  %6 = insertvalue [1 x double] zeroinitializer, double %1, 0
  ret [1 x double] %6
}

; Function Attrs: inaccessiblemem_or_argmemonly
declare void @ijl_gc_queue_root({} addrspace(10)*) #2

; Function Attrs: allocsize(2)
declare noalias nonnull {} addrspace(10)* @ijl_gc_pool_alloc(i8*, i32, i32) #3

; Function Attrs: allocsize(1)
declare noalias nonnull {} addrspace(10)* @ijl_gc_big_alloc(i8*, i64) #4

; Function Attrs: allocsize(1)
declare noalias nonnull {} addrspace(10)* @ijl_gc_alloc_typed(i8*, i64, i8*) #4

attributes #0 = { inaccessiblemem_or_argmemonly nofree "enzyme_inactive" "enzymejl_world"="31504" }
attributes #1 = { alwaysinline nofree }
attributes #2 = { inaccessiblemem_or_argmemonly }
attributes #3 = { allocsize(2) }
attributes #4 = { allocsize(1) }
attributes #5 = { nounwind readnone }

!llvm.module.flags = !{!0, !1}
!llvm.dbg.cu = !{!2}

!0 = !{i32 2, !"Dwarf Version", i32 4}
!1 = !{i32 2, !"Debug Info Version", i32 3}
!2 = distinct !DICompileUnit(language: DW_LANG_Julia, file: !3, producer: "julia", isOptimized: true, runtimeVersion: 0, emissionKind: LineTablesOnly, nameTableKind: None)
!3 = !DIFile(filename: "julia", directory: ".")
!4 = distinct !DISubprogram(name: "fwddiffejulia_identity_1243wrap", linkageName: "fwddiffejulia_identity_1243wrap", scope: null, file: !5, type: !6, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition | DISPFlagOptimized, unit: !2, retainedNodes: !7)
!5 = !DIFile(filename: "operators.jl", directory: ".")
!6 = !DISubroutineType(types: !7)
!7 = !{}
!8 = !{!9, !9, i64 0}
!9 = !{!"jtbaa_gcframe", !10, i64 0}
!10 = !{!"jtbaa", !11, i64 0}
!11 = !{!"jtbaa"}
!12 = !{!13}
!13 = distinct !{!13, !14, !"primal"}
!14 = distinct !{!14, !" diff: %"}
!15 = !{!16}
!16 = distinct !{!16, !14, !"shadow_0"}
!17 = !{!18, !18, i64 0}
!18 = !{!"jtbaa_const", !10, i64 0}
!19 = !{!20}
!20 = distinct !{!20, !21, !"primal"}
!21 = distinct !{!21, !" diff: %ptls_load45"}
!22 = !{!23}
!23 = distinct !{!23, !21, !"shadow_0"}
!24 = !DILocation(line: 522, scope: !25, inlinedAt: !26)
!25 = distinct !DISubprogram(name: "identity", linkageName: "julia_identity_1243", scope: null, file: !5, line: 522, type: !6, scopeLine: 522, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2, retainedNodes: !7)
!26 = distinct !DILocation(line: 0, scope: !4)
"""

function move_to_threadsafe(ir)
    LLVM.verify(ir) # try to catch broken modules

    # So 1. serialize the module
    buf = convert(MemoryBuffer, ir)

    # 2. deserialize and wrap by a ThreadSafeModule
    return ThreadSafeContext() do ctx
        mod = parse(LLVM.Module, buf)
        ThreadSafeModule(mod)
    end
end

optlevel = LLVM.API.LLVMCodeGenLevelNone

tempTM = LLVM.JITTargetMachine(LLVM.triple(), cpu_name(), cpu_features(); optlevel)
LLVM.asm_verbosity!(tempTM, true)

lljit = JuliaOJIT()

jd_main = JITDylib(lljit)

prefix = LLVM.get_prefix(lljit)
dg = LLVM.CreateDynamicLibrarySearchGeneratorForProcess(prefix)
LLVM.add!(jd_main, dg)


function absolute_symbol_materialization(name, ptr)
	address = LLVM.API.LLVMOrcJITTargetAddress(reinterpret(UInt, ptr))
	flags = LLVM.API.LLVMJITSymbolFlags(LLVM.API.LLVMJITSymbolGenericFlagsExported, 0)
	symbol = LLVM.API.LLVMJITEvaluatedSymbol(address, flags)
	gv = if LLVM.version() >= v"15"
		LLVM.API.LLVMOrcCSymbolMapPair(name, symbol)
	else
		LLVM.API.LLVMJITCSymbolMapPair(name, symbol)
	end
	return LLVM.absolute_symbols(Ref(gv))
end

function define_absolute_symbol(jd, name)
	ptr = LLVM.find_symbol(name)
	if ptr !== C_NULL
		LLVM.define(jd, absolute_symbol_materialization(name, ptr))
		return true
	end
	return false
end

if Sys.iswindows() && Int === Int64
	# TODO can we check isGNU?
	define_absolute_symbol(jd_main, mangle(lljit, "___chkstk_ms"))
end

es = ExecutionSession(lljit)
lctm = LLVM.LocalLazyCallThroughManager(triple(lljit), es)
ism = LLVM.LocalIndirectStubsManager(triple(lljit))

activate(LLVM.Context())

mod = parse(LLVM.Module, modstr)
adjoint_name = "fwddiffejulia_identity_1243wrap"

jd = LLVM.JITDylib(lljit)

tsm = move_to_threadsafe(mod)
LLVM.add!(lljit, jd, tsm)

LLVM.lookup(lljit, adjoint_name)

@wsmoses
Copy link
Member

wsmoses commented Mar 13, 2024

Issue on LLVM.jl here: maleadt/LLVM.jl#393

@ChrisRackauckas
Copy link
Contributor

Thanks for taking the time to track it down!

@leerosenthalj
Copy link

leerosenthalj commented Mar 14, 2024 via email

@wsmoses
Copy link
Member

wsmoses commented Mar 14, 2024

This PR #1342 hopefully should provide a workaround, which I've also just backported to 0.11 [and hopefully julia will register shortly].

Unfortuantely the way the backport works is to bypass the Julia Ojit @gbaraldi introduced in 1.10, so no nice backtraces/etc. But at least it won't lock up....

Closing for now (and the ojit can get reenabled once the issue on llvm is resolved)

@wsmoses wsmoses closed this as completed Mar 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed more information needed windows
Projects
None yet
Development

No branches or pull requests

9 participants