Comment on method addition change from 0.4 to 0.5

[jiahao]

The Julia analogue of this Python snippet changes behavior going from 0.4 to 0.5:

#Program A
xs = []
for i in 1:3
    f() = i #which value of i gets bound?
    push!(xs, f)
end
println(xs[1]())

In Julia v0.4, the program produces the result 3 (as do the equivalent Python 2.x and 3.x programs). However in Julia v0.5, the program produces the result 1.

The difference here is that Julia v0.5 produces a new function f at each iteration

pointer_from_objref(f) = Ptr{Void} @0x000000010e676c50
pointer_from_objref(f) = Ptr{Void} @0x000000010e677cd0
pointer_from_objref(f) = Ptr{Void} @0x000000010e677d90

but Julia v0.4 does not:

pointer_from_objref(f) = Ptr{Void} @0x000000010ae71ff0
pointer_from_objref(f) = Ptr{Void} @0x000000010ae71ff0
pointer_from_objref(f) = Ptr{Void} @0x000000010ae71ff0

According to @JeffBezanson, the change was unintentional, and so the issue here is whether or not this change in behavior is desirable.

Related Python resources: PEP 289 and this blog post.

---

Catalogued below are illustrative variant programs.

Workaround using default arguments

In the Python world, the behavior above is termed "late binding" (see, e.g. here and here and here, not to be confused with "late binding" in the context of dynamic dispatch of a generic function). An apparently standard hack is to use default arguments to get lexical scoping of i. In Julia, the equivalent code is

#Program B
xs = []
for i in 1:3
    f(i=i) = i
    push!(xs, f)
end
println(xs[1]())

Program B outputs 1 on both 0.4 and 0.5. Unlike in Program A, 0.4 appears to generate a new function at each iteration in Program B:

pointer_from_objref(f) = Ptr{Void} @0x000000010e7dd340
pointer_from_objref(f) = Ptr{Void} @0x000000010e7dd410
pointer_from_objref(f) = Ptr{Void} @0x000000010e7dd4e0

Anonymous functions (from @JeffBezanson)

#Program C
xs = cell(3)
for i in 1:3
    xs[i] = ()->i
end
println(xs[1]())

On both 0.4 and 0.5, the output of Program C is 1, and a new function is created with each iteration of the loop.

Anonymous functions referencing i in global scope (from @mbauman)

#Program D
xs = cell(3)
i = 0 #i now exists in global scope
for i in 1:3
    xs[i] = ()->i #global scope!
end
println(xs[1]())

Program D outputs 3 on both 0.4 and 0.5. In both versions, the function generated does not change in each iteration:

pointer_from_objref(xs[i]) = Ptr{Void} @0x000000010a0937b0
pointer_from_objref(xs[i]) = Ptr{Void} @0x000000010a0937b0
pointer_from_objref(xs[i]) = Ptr{Void} @0x000000010a0937b0

According to @yuyichao, #14948 is a related issue.

Named functions referencing i in global scope

#Program E
xs = cell(3)
i = 0 #i now exists in global scope
for i in 1:3
    f() = i
    xs[i] = f
end
println(xs[1]())

The behavior of Program E is identical to that of Program D.

Thanks to @JeffBezanson @mbauman @yuyichao for participating in the discussion of this issue prior to filing.

[Keno]

I like the 0.5 behavior.

[staticfloat]

I think semantically this boils down to the answer to the following two questions: "Is the scope of the inside of a for loop independent from other iterations?", and "is the iteration variable an independent realization across loop iterations?". I think the only sane answer to these questions is "Yes", and "Yes", and so I see no alternative than to adopt the v0.5 behavior, as the i variable should be independent across for loop boundaries, as should the f function. Anything otherwise is just inviting scope contamination and weird bugs.

[mbauman]

To my reading, 0.5 matches the documentation: http://docs.julialang.org/en/latest/manual/variables-and-scoping/#for-loops-and-comprehensions

for loops and comprehensions have the following behavior: any new variables introduced in their body scopes are freshly allocated for each loop iteration.

[staticfloat]

@mbauman I thought that at first as well, but I think the crux of the discussion is the scoping of i, the iteration variable. Is that variable "introduced in the body scope" or not?

[JeffBezanson]

Wow, I really take issue with those sources' use of the term "late binding". This is almost as bad as python calling arrays lists. This does not have to do with when variables are looked up (there is no dynamic scoping here either; this is all lexical scope), but with when new variables are allocated. What that comes down to is how a high-level construct like julia or python's for is lowered to lambdas. Here are 2 choices expressed in scheme:

Choice 1: allocate variable "outside" the loop, ala python

(define (f)
  (let ((xs ()))
    (let ((i 1))
      (let loop ()
        (if (> i 3)
            xs
            (begin (set! xs (cons (lambda () i) xs))
                   (set! i (+ i 1))
                   (loop)))))))

Choice 2: allocate variable "inside" the loop, ala julia:

(define (f)
  (let ((xs ()))
    (let loop ((i 1))
      (if (> i 3)
          xs
          (begin (set! xs (cons (lambda () i) xs))
                 (loop (+ i 1)))))))

We picked the second one in 0.4 or earlier. I think what really changed here has to do with when f() = i creates a new function f versus adding or replacing a method of an existing function f. The version of the code that returns 3 is mutating f to contain a method that returns a different value. In any case the 0.5 behavior should be simpler and more consistent.

Long story short, this is a method-addition change and not a scoping change.

[StefanKarpinski]

I would argue that the old behavior was actually a bug since f is local to the for loop.

[StefanKarpinski]

So the conclusion here is that the new behavior is preferable and the old behavior was basically a bug.