Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Turing.jl error with AdvancedHMC #1696

Closed
sunxd3 opened this issue Aug 2, 2024 · 14 comments
Closed

Turing.jl error with AdvancedHMC #1696

sunxd3 opened this issue Aug 2, 2024 · 14 comments
Labels
more information needed

Comments

@sunxd3
Copy link

sunxd3 commented Aug 2, 2024 

using Turing, Enzyme

@model function hmcmatrixsup()
    return v ~ Wishart(7, [1 0.5; 0.5 1])
end

model_f = hmcmatrixsup()
rng = Turing.Random.default_rng()
alg = HMC(0.15, 7; adtype=Turing.ADTypes.AutoEnzyme())
spl = Turing.Sampler(alg, model_f)
vi = Turing.DynamicPPL.default_varinfo(rng, model_f, spl)

DynamicPPL.initialstep(rng, model_f, spl, vi)

fails on https://github.com/TuringLang/Turing.jl/blob/4766fdde20ca9502472afcb991bd05f984d3ad35/src/mcmc/hmc.jl#L184

error msg

ERROR: Enzyme execution failed.
Mismatched activity for:   store {} addrspace(10)* %8, {} addrspace(10)** %.repack, align 8, !dbg !285, !noalias !234 const val:   %8 = load atomic {} addrspace(10)*, {} addrspace(10)* addrspace(11)* %7 unordered, align 8, !dbg !193, !tbaa !200, !nonnull !4, !dereferenceable !204, !align !191
Type tree: {[-1]:Pointer, [-1,0]:Pointer, [-1,0,0]:Pointer, [-1,0,0,-1]:Integer, [-1,0,8]:Integer, [-1,0,9]:Integer, [-1,0,10]:Integer, [-1,0,11]:Integer, [-1,0,12]:Integer, [-1,0,13]:Integer, [-1,0,14]:Integer, [-1,0,15]:Integer, [-1,0,16]:Integer, [-1,0,17]:Integer, [-1,0,18]:Integer, [-1,0,19]:Integer, [-1,0,20]:Integer, [-1,0,21]:Integer, [-1,0,22]:Integer, [-1,0,23]:Integer, [-1,0,24]:Integer, [-1,0,25]:Integer, [-1,0,26]:Integer, [-1,0,27]:Integer, [-1,0,28]:Integer, [-1,0,29]:Integer, [-1,0,30]:Integer, [-1,0,31]:Integer, [-1,0,32]:Integer, [-1,0,33]:Integer, [-1,0,34]:Integer, [-1,0,35]:Integer, [-1,0,36]:Integer, [-1,0,37]:Integer, [-1,0,38]:Integer, [-1,0,39]:Integer, [-1,8]:Pointer, [-1,8,0]:Pointer, [-1,8,8]:Integer, [-1,8,9]:Integer, [-1,8,10]:Integer, [-1,8,11]:Integer, [-1,8,12]:Integer, [-1,8,13]:Integer, [-1,8,14]:Integer, [-1,8,15]:Integer, [-1,8,16]:Integer, [-1,8,17]:Integer, [-1,8,18]:Integer, [-1,8,19]:Integer, [-1,8,20]:Integer, [-1,8,21]:Integer, [-1,8,22]:Integer, [-1,8,23]:Integer, [-1,8,24]:Integer, [-1,8,25]:Integer, [-1,8,26]:Integer, [-1,8,27]:Integer, [-1,8,28]:Integer, [-1,8,29]:Integer, [-1,8,30]:Integer, [-1,8,31]:Integer, [-1,8,32]:Integer, [-1,8,33]:Integer, [-1,8,34]:Integer, [-1,8,35]:Integer, [-1,8,36]:Integer, [-1,8,37]:Integer, [-1,8,38]:Integer, [-1,8,39]:Integer, [-1,16]:Pointer, [-1,16,0]:Pointer, [-1,16,0,-1]:Integer, [-1,16,8]:Integer, [-1,16,9]:Integer, [-1,16,10]:Integer, [-1,16,11]:Integer, [-1,16,12]:Integer, [-1,16,13]:Integer, [-1,16,14]:Integer, [-1,16,15]:Integer, [-1,16,16]:Integer, [-1,16,17]:Integer, [-1,16,18]:Integer, [-1,16,19]:Integer, [-1,16,20]:Integer, [-1,16,21]:Integer, [-1,16,22]:Integer, [-1,16,23]:Integer, [-1,16,24]:Integer, [-1,16,25]:Integer, [-1,16,26]:Integer, [-1,16,27]:Integer, [-1,16,28]:Integer, [-1,16,29]:Integer, [-1,16,30]:Integer, [-1,16,31]:Integer, [-1,16,32]:Integer, [-1,16,33]:Integer, [-1,16,34]:Integer, [-1,16,35]:Integer, [-1,16,36]:Integer, [-1,16,37]:Integer, [-1,16,38]:Integer, [-1,16,39]:Integer, [-1,24]:Integer, [-1,25]:Integer, [-1,26]:Integer, [-1,27]:Integer, [-1,28]:Integer, [-1,29]:Integer, [-1,30]:Integer, [-1,31]:Integer, [-1,32]:Integer, [-1,33]:Integer, [-1,34]:Integer, [-1,35]:Integer, [-1,36]:Integer, [-1,37]:Integer, [-1,38]:Integer, [-1,39]:Integer, [-1,40]:Integer, [-1,41]:Integer, [-1,42]:Integer, [-1,43]:Integer, [-1,44]:Integer, [-1,45]:Integer, [-1,46]:Integer, [-1,47]:Integer, [-1,48]:Integer, [-1,49]:Integer, [-1,50]:Integer, [-1,51]:Integer, [-1,52]:Integer, [-1,53]:Integer, [-1,54]:Integer, [-1,55]:Integer, [-1,56]:Integer, [-1,57]:Integer, [-1,58]:Integer, [-1,59]:Integer, [-1,60]:Integer, [-1,61]:Integer, [-1,62]:Integer, [-1,63]:Integer}
 llvalue=  %8 = load atomic {} addrspace(10)*, {} addrspace(10)* addrspace(11)* %7 unordered, align 8, !dbg !193, !tbaa !200, !nonnull !4, !dereferenceable !204, !align !191
You may be using a constant variable as temporary storage for active memory (https://enzyme.mit.edu/julia/stable/faq/#Activity-of-temporary-storage). If not, please open an issue, and either rewrite this variable to not be conditionally active or use Enzyme.API.runtimeActivity!(true) as a workaround for now

Stacktrace:
 [1] VarInfo
   @ ~/.julia/packages/DynamicPPL/93t4P/src/varinfo.jl:117

Stacktrace:
 [1] throwerr(cstr::Cstring)
   @ Enzyme.Compiler ~/.julia/packages/Enzyme/r8mFE/src/compiler.jl:1797

versions:
Julia Version 1.7.3
[7da242da] Enzyme v0.12.26
[fce5fe82] Turing v0.33.3
@sunxd3 sunxd3 mentioned this issue Aug 2, 2024
Draft
@wsmoses
Copy link
Member

wsmoses commented Aug 2, 2024

What happens if you set Enzyme.API.runtimeActivity!(true) as the error message says?

Turing already does this automatically so this isn't the cause of the issue at hand.

@sunxd3
Copy link
Author

sunxd3 commented Aug 2, 2024

It hangs and crushes Julia, shown something like

signal (11): Segmentation fault: 11
in expression starting at REPL[11]:1
unknown function (ip: 0x123e2497a)
Allocations: 277657897 (Pool: 277514439; Big: 143458); GC: 107

@yebai
Copy link

yebai commented Aug 2, 2024

Likely related: #1649 #1081

@yebai
Copy link

yebai commented Aug 2, 2024

I mentioned #1649 (comment): a test on both Wishart and Chol should be added to the test suite on the Julia side.

@yebai yebai mentioned this issue Aug 2, 2024
Closed
@wsmoses
Copy link
Member

wsmoses commented Aug 2, 2024 via email

@sunxd3
Copy link
Author

sunxd3 commented Aug 2, 2024 •
edited
Loading

On 1.10.4, it also fails (on my machine MacOS) (edited, never mind, it works!!)

@wsmoses
Copy link
Member

wsmoses commented Aug 2, 2024 via email

@sunxd3
Copy link
Author

sunxd3 commented Aug 2, 2024

Got it, it runes well on 1.10.4, which is a relief

@sunxd3
Copy link
Author

sunxd3 commented Aug 2, 2024

A much smaller MWE is the way to go, will try to produce one when get time. (Maybe @mhauru can take over when back?)

@wsmoses
Copy link
Member

wsmoses commented Aug 21, 2024

bump @mhauru @sunxd3 and or @yebai

@mhauru
Copy link
Contributor

mhauru commented Aug 21, 2024

On my todo list, just haven't gotten to it yet, sorry. May still take a week or two to find time.

@wsmoses
Copy link
Member

wsmoses commented Sep 18, 2024

bump here @mhauru

@mhauru
Copy link
Contributor

mhauru commented Sep 18, 2024

#1649 seems similar and also works on 1.10 but crashes on 1.7. I'll wait for that to be sorted out before putting in the work to minimise this.

@wsmoses
Copy link
Member

wsmoses commented Sep 21, 2024

@mhauru we've dropped Julia before 1.10, so I presume this error is moot and no longer applies. Reopen with a MWE if there's still an issue.

@wsmoses wsmoses closed this as completed Sep 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
more information needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@wsmoses @yebai @mhauru @sunxd3