with for deterministic destruction

[klaufir]

Deterministic destruction
Deterministic destruction is a guarantee that some specified resources will be released when exiting from the enclosing block even if we exit from the block by the means of exceptions.

Example
In python deterministic destruction is done using the with statement.

with open("myfile","w") as f:
    f.write("Hello")

Considering the code above we don't have to worry about closing a file explicitly. After the with block, the file will be closed. Even when something throws an exception inside the with block, the resources handled by the with statement will be released (in this case closed).

Other languages

• C++ always had guaranteed destructor calls at scope exit for stack objects - even in the face of exceptions (see RAII).
• In C# there is the using statement for this same purpose.
• Java has try-with-resources since Java7.

Julia?

It is my firm belief that Julia also needs to have a feature supporting deterministic destruction. We have countless cases when a resource needs to be closed at soon as possible: serial ports, database connections, some external api handles, files, etc. In cases like these deterministic destruction would mean cleaner code, no explicit close() / release() calls and no sandwiching the code in try .. catch .. finally blocks.

[prcastro]

The do syntax doesn't handle this?

[jakebolewski]

You can do the same things with julias do notation anonymous function syntax and with macros. Try finally also lends itself to deterministic destruction.

[JeffBezanson]

I brought this up briefly in #4664. I think with and finalize would be good.

[klaufir]

I know it can be done in a custom way by hacking some macros, but it would be much more beneficial if we had this feature in combination with support from the standard library - as it is done in python. Having a standard way of doing it would make new libraries conform to this standard way. Then the users don't have to worry about the method name of the finalizer (Is it close(), release(), free()?).

[timholy]

@klaufir, it's not hacking some macros. In Julia your example is

open("myfile", "w") do file
    write(file, "Hello")
end

and the file is automatically closed for you. See? No macros 😄.

But there are places where one might want to use finalizers in most circumstances yet be able to force them to run in specific situations. So there might be room for some new features, but it's not like there isn't already a way to do this kind of thing.

[klaufir]

Oh, I see there is a standard way already. Sorry for the noise.

[StefanKarpinski]

That is the standard idiom – writing code to ensure that some code is always called upon exit is still manual though. I've often wanted something like Go's defer or D's scope exit clauses. Note that these are more general in a way because you can ensure that any expression is executed on scope exit.

[JeffBezanson]

I think with would be better than what we do now, since different finalizeable types wouldn't need to re-implement the pattern used by open.

[timholy]

I agree that implementing that pattern usually requires a trip to the manual, and occasionally interferes with some other desirable call syntax. Ran into the latter with CUDArt.

[awf]

Just checking: the general feeling is that "do" doesn't do it, and "with" is still desirable?

[StefanKarpinski]

I'm convinced at this point that do syntax isn't sufficient, but I also think that Python's with syntax doesn't quite cut it either. What seems to be needed is automatic insertion of a finalize on the result of a function call when the appropriate scope exits. One problem with both do and with is that they require nesting/indentation. It's common to do a bunch of setup and then do the main work and then do all the corresponding teardown. Even using the with construct, we'd have to write something like this:

with open("input") as r
    with open("output", "w") as w
        # do actual work
    end
end

Another problem with both syntaxes is that they cannot be used inline, making them unhelpful, e.g. for the common problem of wanting to open a file only to pass it to a function and then close it when that call returns (see here for example).

I noticed that the syntax f(x...)! is available so I'm going to throw this out there as syntax for doing y = f(x...) and inserting finalize(y) in at the point where the returned value y goes out of scope. This would allow us to write the above examples as:

r = open("input")!
w = open("output", "w")!
# do actual work

Calls to finalize(w) and finalize(r) are inserted automatically when r and w go out of scope. Actually, it's more than that since the finalize calls are guaranteed no matter how the stack unwinds. You can also use this effectively without binding to a local variable, just passing it to a function call:

write(open("file","w")!, data)

Since the result of the open call goes out of scope after this line, this becomes something like this:

try f = open("file","w")
    write(f, data)
finally
    finalize(f)
end

So that addresses #14546 in a systematic way without adding any new methods and can eliminate many of the functions littering #14608.

[JeffBezanson]

Wow, I kind of like that idea. It would be excellent if resource-backed objects only needed to implement close and one form of open, and we hardly ever used gc finalizers (or eliminated them entirely).

[awf]

Just for a second, can we imagine inverting the syntax: so v = ...! covers the unusual case where I do want to wait for GC to finalize v? How much existing code depends on that? Which bugs are worse: referencing prematurely-finalized objects, or leaking resources? The former are pretty easy to detect, at least at the cost of the finalizer explicitly putting the object into an invalid state.

[JeffBezanson]

See also #11207

[StefanKarpinski]

@awf you don't want to translate every function call into a try/catch with a corresponding finalize call.

[kmsquire]

Sounds somewhat like golang's defer (also mentioned by @quinnj in #11207), except that defer is more general (and more specific about what exactly is going to happen).

[JeffBezanson]

I think we can have defer and have ! be syntax for defer close(x) since that's what you need 99.9% of the time. The verbosity reduction is huge:

x = open(file); defer close(x)
write(x, data)

vs

write(open(file)!, data)

[kmsquire]

Nice. +1

[tkelman]

I like the idea, but not the syntax. Sigils for resource management starts getting into rust levels of "what is going on" for newcomers unfamiliar with what a trailing ! might mean. Can this be done with a macro or function/type wrapper? If we want deterministic destruction, the question of when it occurs should be easy to answer at a glance. It can be hard enough as is to explain how julia's scoping works sometimes.

[StefanKarpinski]

@JeffBezanson: shouldn't we be having this call finalize(x) rather than close(x)? Or do you feel that close is a general enough as a concept to be considered the generic name for "finalize me"? It feels kind of I/O-specific to me. Keep in mind that we can just define finalize(io::IO) = close(io) in Base and then all you ever need to define for any IO object is open and close.

[eschnett]

Next step is an RC{T} class that wraps objects of type T, adding reference counting. This would be quite useful if the resource might be stored in a data structure or might be passed to another thread, but we still want early finalization. I'm thinking e.g. of large arrays or matrices for numerical calculations.

Although the implementation will be quite different, I hope that the syntax to use it can be made similar to the one discussed here. A macro might work:

dothework(@fin open("file"))
dothework(@rc open("file"))

[StefanKarpinski]

@tkelman: Given the potential ubiquity of this, a very lightweight syntax is essential. Since you don't like the syntax, please propose alternatives. This cannot be done with a function since it interacts with the surrounding syntax; if we had defer then it could be done with a macro generating defer. I very much like that f(x)! almost looks like f(x) since that's what you would write if you just let GC finalize x. Inserting a macro call would make this a lot less pleasant.

[JeffBezanson]

I kind of like the idea of combining finalize and close into one function, but it's no big deal.

Next step is an RC{T} class that wraps objects of type T, adding reference counting

-100. I would be amazed if there is any reasonable way to make that work. The right way to handle this case is to have the compiler insert speculative early-free calls.

[tkelman]

I don't mind the indentation of the do block form. I think readable and intuitive syntax should trump saving keystrokes especially for subtle sources of bugs like resource management. Defer with a macro would be easier to explain the rules for than a sigil handled at the parser level.

[StefanKarpinski]

I've watched a lot of people write code like this over and over:

f = open("file")
# do work
close(f)

I cannot get them to use the do-block form, even though I've explained many times that it's the right way to express this since it prevents unclosed file handles in case of exceptions. Of course, the file handles do eventually get closed on gc, so it's not dire, but in general, if people want to do something one way, it's better to make the thing they want to do work right rather than lecturing them about how some other way to do it is better. I doubt we'd have more luck with getting people to use the with form than the do block form. But I'm pretty optimistic that I could get the same people to just write f = open("file")! and omit the close(f) entirely. In fact, I think people would love this (it's way easier and requires less code), and I don't think that explaining what the ! means would be hard at all.

Longer term, this syntax would entirely eliminate the need for having do-block versions of all functions to do cleanup. That's a big win. But the real question is whether it's important enough to have its own syntax. I would argue that the pattern of doing setup, then doing work, then doing some cleanup (regardless of how the stack unwinds) is ubiquitous. It's also annoying to get right without syntactic support and as above, people usually just don't bother. So in my view, having syntactic support for doing this pattern correctly is a no-brainer. Whether the f(x)! syntax is the best one or not is another question, but I can't think of anything better. It makes some mnemonic sense too:

• f!(x) means that the caller is passing a value to the callee, which will do some work with it and return it to the caller.
• f(x)! means that the callee is returning a value to the caller, which will do some work with it and return it to the callee.

Makes sense to me. I suspect it will make sense to other people too and not be terribly hard to explain.

[c42f]

Right, having something like @manage makes sense 👍

But I think this is orthogonal to the point I'm trying to make here. It's the following dichotomy which is bothering me:

• For the resource author the "do-block" form of open() is flexible and convenient
• For the resource user, syntax like the proposed user_func(open(config)!) is desirable compared to open(user_func, config)

Hah, with the transformation user_func(open(config)!) => open(user_func, config) in mind, this is reminding me strikingly of delimited continuations. Is shift/reset an alternative way to think about the desired functioning of the ! syntax?

[o314]

Sorry, but i disagree with the proposal of merging the management of close, lock and sync

@lock is super simple, ~ 10 sloc

macro lock(l, expr)
    quote
        temp = $(esc(l))
        lock(temp)
        try
            $(esc(expr))
        finally
            unlock(temp)
        end
    end
end

while @sync is out of scope. one has to deal with :

macro sync(block) # ~ 10 sloc @ https://github.com/JuliaLang/julia/blob/v1.5.2/base/task.jl
    var = esc(sync_varname)
    quote
        let $var = Channel(Inf)
            v = $(esc(block))
            sync_end($var)
            v
        end
    end
end


# and
function sync_end(c::Channel{Any}) # ~ 30 sloc
    local c_ex
    while isready(c)
        r = take!(c)
        if isa(r, Task)
            _wait(r)
            if istaskfailed(r)
                if !@isdefined(c_ex)
                    c_ex = CompositeException()
                end
                push!(c_ex, TaskFailedException(r))
            end
        else
            try
                wait(r)
            catch e
                if !@isdefined(c_ex)
                    c_ex = CompositeException()
                end
                push!(c_ex, e)
            end
        end
    end
    close(c)
    if @isdefined(c_ex)
        throw(c_ex)
    end
    nothing
end

then

# https://github.com/JuliaLang/julia/blob/v1.5.2/base/channels.jl#L381
take!(c::Channel)) # ~ 30 sloc
take_buffered(c::Channel)
take_unbuffered(c::Channel{T}) where {T}

nearly 100 sloc. to use @sync. that's way too much in regard to the first code
one has to separate more clearly the fork/join and the failure compensation for sync.

close and lock seem to be good companion however.
but lock can become quickly tricky. it may be wise to tackle a broader scope with close than lock

---

EDIT to let the common forms speak by themselves :

# open @ https://github.com/JuliaLang/julia/blob/v1.5.2/base/io.jl
function open(f::Function, args...; kwargs...)
    io = open(args...; kwargs...)
    try
        f(io)
    finally
        close(io)
    end
end

# Base.@lock @ https://github.com/JuliaLang/julia/blob/v1.5.2/base/lock.jl
macro lock(l, expr)
    quote
        temp = $(esc(l))
        lock(temp)
        try
            $(esc(expr))
        finally
            unlock(temp)
        end
    end
end

[stevengj]

Rather than annotating individual objects, would it be possible to annotate a type for "aggressive" memory management, so that the compiler is encouraged to call finalize on the object immediately when escape analysis or similar indicates that it is no longer referenced?

In all the examples given so far, it seems like we want to perform deterministic destruction of essentially every object of certain types, e.g. file handles. In that case, it would be better not to rely on the programmer remembering to include call-site annotations. And ordinary garbage collection will still operate as a fallback when static analysis fails.

See also #34836. I realize that we don't want reference-counting-like semantics for every type because of the performance implications, but it seems like it would be helpful to have it for some types. And since this is essentially just a performance hint it will be backwards compatible (and will automatically improve old code using these types).

[vtjnash]

If we had escape analysis, the compiler would just do that for every object. There's no need to opt into it.

[saolof]

So, when interfacing with C, finalizers are a very common source of bugs, and were incredibly painful to deal with when trying to use Vulkan.jl .

It'd definitely be great if there were an abstraction that let you deal with values that should never be GCed, but should be disposed of by scope at a specified point with a destructor.

@vtjnash I strongly support escape analysis for exactly that reason (inserted frees whenever something does not escape). It'd be great if there were a complementary syntax that also guarentees an object never escapes though.

[KristofferC]

It'd definitely be great if there were an abstraction that let you deal with values that should never be GCed, but should be disposed of by scope at a specified point with a destructor.

I use https://github.com/adambrewster/Defer.jl for that. Works nice. Here is an example: https://github.com/JuliaGPU/AMGX.jl/blob/e7ac5aaf3d141425dcb1f5ebd2b203901f6e5856/test/test_solver.jl#L9-L15.

[saolof]

It'd definitely be great if there were an abstraction that let you deal with values that should never be GCed, but should be disposed of by scope at a specified point with a destructor.

I use https://github.com/adambrewster/Defer.jl for that. Works nice. Here is an example: https://github.com/JuliaGPU/AMGX.jl/blob/e7ac5aaf3d141425dcb1f5ebd2b203901f6e5856/test/test_solver.jl#L9-L15.

Okay, that looks pretty close to what I was looking for, with the @! macro. Does it also prevent the value from being GCed?

[StefanKarpinski]

@saolof, I've found the opposite to be true: finalizers are a godsend when interfacing with C code as they provide a simple means of ensuring that C-allocated objects are finalized and freed when they are no longer reachable. What kinds of issues did you encounter?

[KristofferC]

FWIW, I tried for a long time to use finalizers in AMGX.jl but it just want possible to uphold e.g the ordering of destruction needed. So in the end I just recommend the Defer.jl approach for the package.

[c42f]

Note that Defer.jl uses mutable global state to manage the scopes so it's more a useful behavioral prototype than a production-ready package.

Related to Defer.jl I've been thinking about this problem again in the last few days, and have just started a new package https://github.com/c42f/Contexts.jl.

To quote some of the readme:

This package uses the macro @! as a proxy for the proposed postfix ! syntax and adds some new ideas:

The user should not be able to "forget the !". We prevent this by introducing a new context calling convention for resource creation functions where the current AbstractContext is passed as the first argument. The @context macro creates a new context in lexical scope and the @! macro is syntactic sugar for calling with the current context.

Resource creation functions should be able to compose any desired object return type with arbitrary resources. This preserves the composability of the do block form
by rejecting the conflation of the returned object and its backing resources. This is a break with some familiar APIs such as the standard file handles returned by open(filename) which are both a stream interface and a resource in need of cleanup.

Some examples:

function f()
    @context readlines(@!(open("tmp/hi.txt", "r")))
end

Create a temporary file and ensure it's cleaned up afterward

@context function f()
    path, io = @! mktemp()
    write(io, "content")
    flush(io)
    @info "mktemp output" path ispath(path) isopen(io) read(path, String)
end

Defer shredding of a secretbuffer until scope exit

@context function f()
    buf = SecretBuffer()
    @defer shred!(buf)
    secret_computation(buf)
end

Superficially this is similar to Defer.jl, but the internal context-passing design resolves the problems with composability I was trying to describe in #7721 (comment).

It's also compatible with existing verbs like Base.open() by simply adding new overloads, and can interoperate with "do-block-based" resource handling via an async shim.

[saolof]

@saolof, I've found the opposite to be true: finalizers are a godsend when interfacing with C code as they provide a simple means of ensuring that C-allocated objects are finalized and freed when they are no longer reachable. What kinds of issues did you encounter?

Finalizers can cause double frees if you try to call the disposal function directly, so if you use them you implicitly give up the ability to manually destroy something before the GC removes it, and sometime you run into the opposite issue of it being run while C code is supposed to hold onto the data. When finalization order matters and you want it to happen at the end of some scope, that is incredibly annoying.

If the compiler had good escape analysis and there were some statement that ensured that an object doesn't escape a scope or live past a certain point but keeps it alive until then, it'd be a different issue (arguably this is an overlap of several possible features in the language design space from pinning to linear types)

[eschnett]

I might be missing the point... would this not work? Recording the fact that the disposal function has been called in the Julia wrapper object?

mutable struct Wrapper
    ptr::CPtr
    Wrapper() = (...; register finalizer)
end

free(wrapper::Wrapper) = if wrapper.ptr c_free(wapper.ptr); wrapper.ptr=null end

finalize(wrapper::Wrapper) = if wrapper.ptr c_free(wrapper.ptr) end

[KristofferC]

Yes, you can quite easily make finalizers idempotent but that's just a small part of the crux.

[c42f]

When finalization order matters and you want it to happen at the end of some scope, that is incredibly annoying.

Exactly 👍

To get the ordering correct when using finalizers, you'd have to ensure that each wrapper also held a Julia reference to any parent resources it wants to keep alive. This is possible but it bulks up the wrappers on the Julia side and is generally just annoying.

I think Contexts.jl solves these problems in a pleasant way, and I'm excited because it also solves a bunch of problems I didn't expect. For example, have you ever wanted to return a raw Ptr to a temporary buffer for some low level work? Well now you can:

using Contexts

@! function raw_buffer(len)
    buf = Vector{UInt8}(undef, len)
    @defer GC.@preserve buf nothing
    pointer(buf)
end

@context begin
    len = 1_000_000_000
    ptr = @! raw_buffer(len)
    GC.gc() # `buf` is preserved!
    unsafe_store!(ptr, 0xff)
end

[saolof]

When finalization order matters and you want it to happen at the end of some scope, that is incredibly annoying.

Exactly 👍

To get the ordering correct when using finalizers, you'd have to ensure that each wrapper also held a Julia reference to any parent resources it wants to keep alive. This is possible but it bulks up the wrappers on the Julia side and is generally just annoying.

I think Contexts.jl solves these problems in a pleasant way, and I'm excited because it also solves a bunch of problems I didn't expect. For example, have you ever wanted to return a raw Ptr to a temporary buffer for some low level work? Well now you can:

using Contexts

@! function raw_buffer(len)
    buf = Vector{UInt8}(undef, len)
    @defer GC.@preserve buf nothing
    pointer(buf)
end

@context begin
    len = 1_000_000_000
    ptr = @! raw_buffer(len)
    GC.gc() # `buf` is preserved!
    unsafe_store!(ptr, 0xff)
end

Starred the repo. This is exactly what I need

[KristofferC]

@c42f Is there overlap with #35833?

[c42f]

Starred the repo. This is exactly what I need

Excellent. I've got a registration PR for this at JuliaRegistries/General#36658 so you should be able to use it conveniently soon. However, I'm considering using a less generic name for the package so I've put the merge on hold for now. In the meantime if you try the package out I'd appreciate any feedback.

Is there overlap with #35833?

Good question. There's not a lot of overlap:

• RFC: Context variables #35833 is about adding better task-local storage; essentially a "better global variable":
  • it's passed implicitly down the call chain
  • doesn't affect dispatch
  • doesn't provide any scope management
• On the other hand, Contexts.jl is for resource management:
  • The context is passed an argument but doesn't propagate implicitly — it's a purely local change to the call chain which is marked syntactically.
  • Put another way, the context affects dispatch
  • It includes a notion of scope and scope exit for cleanup

The use cases for these two different types of contexts are fairly different.

[c42f]

Ok, I've renamed Contexts.jl as ResourceContexts.jl and it's now registered in the General registry.

A long while back (years ago now!) @JeffBezanson wrote

I think we can have defer and have ! be syntax for defer close(x) since that's what you need 99.9% of the time.

I think it's interesting to compare the implicit defer close() with a context-passing design where foo!(x) means foo(ctx, x). Here ctx::AbstractContext is a context object which should be defined and cleaned up in the caller based on language scope rules.

There's some benefits to passing a context:

• Resources created within the call result = foo()! are attached to the context for cleanup, rather than being rooted in the result. This allows result to have a type and API the user cares about, regardless of the number or type of underlying resources backing that object and decouples resource management from the main flow of values within the program. result may even be immutable.
• The implementer of foo(ctx) uses defer to attach arbitrary cleanup code to ctx which will happen in the caller. There's no need for result to implement close or be involved in resource management in any way.

Clearly context passing introduces a form of colored functions (colored by their calling convention). It's possible there's some drawbacks for this, as it makes context passing an official part of the API of a function — in a sense, it bifurcates method tables into two based on the syntax used at the callsite. There's a big potential benefit for this, however — users will get a method error for calling foo() when they only implemented foo()!, so forgetting to close resources is much less of a problem.

[o314]

FWIW here is the with function of LibGit2 @ julia v1.6

function with(f::Function, obj)
    try
        f(obj)
    finally
        close(obj)
    end
end

May clearly be promoted to a more generic "module host".

It may be enhanced too by handling a return for f

function with(f::Function, obj)
    local r
    try
        r = f(obj)
    finally
        close(obj)
    end
    r
end

tests :

mutable struct A
    v
    step # consumption
    A(v) = new(v, 10) # big step
 end
f(a::A) = a.v * a.step
close(a::A) = a.step = 1 # small step after close

using Test
a = A(1)
@test with(f, a) == 10
@test f(a) == 1 # no it's closed

[simonbyrne]

How about defining a generic with function along the lines of:

function with(fn, args...)
    try
        fn(args...)
    finally
        for arg in args
            finalize(arg)
        end
    end
end

(from JuliaIO/HDF5.jl#990 (comment))

[o314]

May be good to insert a return nothing at the end of the finally block , this will enable to get a success/failure triage by using Some on the happy path. Otherwise it's lost

solved by #7721 (comment)

[simonbyrne]

There's no catch, so it should propagate the error, and not return

[stevengj]

With eager finalization (#45272), couldn't destruction be guaranteed for something as simple as a let block, or any local scope that doesn't let the object escape?

[simonbyrne]

That would be ideal: @Keno @aviatesk are there any such guarantees, and if so could we document it?

[Keno]

There are no guarantees, but I did have a sketch of a design that would add those guarantees based on the same mechanism that I had discussed with @JeffBezanson and @StefanKarpinski at some point.

[quinnj]

Can someone help me understand how the eager finalization works? Is it just based on whether the compiler can figure out the lifetime of the object and infer the finalizer? Are there other ways/requirements to "opt in" to eager finalization?

[Keno]

Can someone help me understand how the eager finalization works? Is it just based on whether the compiler can figure out the lifetime of the object and infer the finalize?

Pretty much

Are there other ways/requirements to "opt in" to eager finalization?

The compiler needs to be able to prove :nothrow and :notaskstate on the actual finalizer itself.