def vs setv

[gilch]

A subissue from #240.

These do exactly the same thing as far as I can tell. We don't need both. def is a Python keyword, so it gets priority in my mind. However, it's currently defined to do something very different than the Python def, so I'm less certain.

As pointed out in the discussion for #899, we don't actually have to reserve Python keywords in Hy. It might be problematic for Hy/Python interop not to, but there are workarounds. So the opposite is also possible. We can just keep setv and treat def like any other symbol.

I've heard arguments that we should keep both, the gist being because def feels more like a global constant definition, and setv feels more like a variable change. I don't exactly buy this because,

• they're doing exactly the same thing;
• we can have nested defn, defclass, and defmacro--meaning the def- prefix does not really mean global nor constant elsewhere in Hy;
• global "constants" already have earmuffs/caps to distinguish them from normal variables and we don't need another convention on top of that.

But if this feeling is still that important, there are some alternative ways to deal with it.

[gilch]

Alternative proposal 1:
get rid of BOTH def and setv

And replace with := (a la ALGOL). Now they both feel like what they are--an assignment to a variable.

Why :=? Well = was already taken to mean ==. Confusing the two is a common source of errors in C. (Not using ALGOL's := is widely considered one of C's historical design flaws.)

Python fixed this by not counting = statements as expressions, so they can't accidentally be used as one. Hy has unfixed this since everything works as an expression, but mostly made up for this by having setv/def for assignment instead.

Edit: We should probably rename Hy's = to == in this case, just so the problematic default typo of = when either := or == was intended isn't allowed.

[gilch]

Alternative proposal 2:
Make def and setv different enough to be worth another keyword.

Like I said before, filling keyword quotas is poor language design.

But there have been proposals to kind of enforce the existing convention for def, so perhaps we could try to enforce that better.

The best I've come up with is to make def a macro that asserts the target hasn't been assigned to yet. This is what you want for a constant. So you can only def any given symbol once. At least not without using an explicit del on it in between redefs. You can still setv a defed variable though.

Many languages distinguish between delcaring a variable and assigning to it, and also variables vs constants--both points Python kind of glosses over. You could use the new asserting def for either purpose.

We could have our linter (hydiomatic) emit a warning whenever an earmuffed symbol was setv'd, but not when defed.

We could also have a convention that variables are declared with def and altered with setv, though this seems terribly unPythonic to me.

One question I still have is

(def foo "foo!")
(defn foonorf []
  (def foo "footwo!"))

Should the above case trigger the assertion or not? You're not overwriting the global, so I'm inclined to answer "not". But this assertion would be a bit more difficult to implement in the def macro. We also have nonlocal and attributes to worry about.

[refi64]

The assertion in proposal 2 should be a runtime assertion, IMO. It could be compiled to:

try:
    foo
except NameError:
    foo = 'footwo!'
else:
    # error

Also, what about the proposal for def always making the target global?

[gilch]

I suppose runtime could work. We could even have a def macro expand into the equivalent Hy, if that's easier. But remember that def also works on attributes when the target symbol contains a dot. So you also need to catch AttributeErrors.

I didn't understand how this works in nested scopes at first, but it does seem to work. I was expecting UnboundLocalError rather than NameError if anything, but it turns out that UnboundLocalError is a NameError. So I'm not confident I can predict how this will work in other cases, like nonlocals or class definitions. It would need more testing at least. (AttributeErrors, don't count as NameErrors, btw.)

Also, what about the proposal for def always making the target global?

I'm including that in the "But there have been proposals to kind of enforce the existing convention for def...", from the OP. The reason I'm not considering this as an alternative is that I don't approve.

Global variables are considered harmful. It scatters state dependency throughout the program. This is a problem OO encapsulation was made to address, though (in practice) mutable fields can be almost as bad if you aren't careful. We really don't want to encourage this in Hy by giving it dedicated syntax. We already have global, and that's (bad) enough.

Global constants, on the other hand, are not so bad. Every top level class, function, and macro name is effectvely used as a global constant anyway. We can manage these pretty well with namespaces. We don't need global or a global def syntax to use read-only global constants, since Python already allows you to read from enclosing scopes without any special syntax. You just can't assign to them.

[gilch]

A variation on alternative 2:

def creates a local (locals are still global at toplevel)
setv can't create a new variable. It assigns to a nonlocal. It would macroexpand into a nonlocal statement followed by an assignment, e.g.

(setv a 1 b 2)
;; expands to
(do
  (nonlocal a b)
  (def a 1 b 2))

nonlocal a, b
a = 1
b = 2

I don't think multiple nonlocal statements with the same variable are allowed in Python, but we could allow it in Hy by filtering duplicates from the AST. So multiple setvs on the same variable will still work.

And, of course, nonlocal doesn't work in Python2. We could simply disallow setv the same way we disallow nonlocal. If we somehow port nonlocal to Hy in Py2 before Python2 becomes too obsolete to bother, we can enable it again.

[tx46]

So what's the status here? Why can't we bring back let for some sane scoping? Couldn't you just del varname at the end of the scope?

[refi64]

I had proposed that already, but it broke in the case of closures, e.g.:

(defn f []
  (let [x 1]
    (fn [] x)))

((f))  ; ERROR: x was deleted when f ended

[Kodiologist]

So what's the status here?

You mean @gilch's proposal 2? Nobody's been working on it that I know of, but not for any special reason. I like it, for what that's worth. Or at least, I'd appreciate an optional feature that requires you to declare variables before you use them and checks this at compile-time, like Perl's use strict 'vars'.

[tx46]

I usually use def for constants and setv for variables. Could we change def to prevent further assignments? Absolutely breaking change but we're still pretty early.

[Kodiologist]

I believe that that's the proposal 2 I was just talking about.

[gilch]

Update

After my original proposals, Python has added a new option to declare a variable before assignment. See PEP 526 -- Syntax for Variable Annotations These do have runtime effects, so Hy will need to support them anyway for compatibility.

I think it's likely that Python's new optional static type checker, mypy, will be able to declare constants python/mypy#1214. This would have no runtime effect, but the static type checker could catch a violation in most cases. Once we've added variable and function annotation syntax to Hy I'd like to see if mypy is compatible. I think it can check the AST in Python 3 instead of the text, so it should work.

These annotations can also be used by metaclasses and decorators, and are already being used in the standard library--see the workaround for NamedTuple I had to use to answer this StackOverflow question. So Hy needs a syntax for this to fully support the standard library, not just for the linter.

So rather than expand to some awkward runtime assertion for constants, let's just do it Python's way when it's available.

Then how should an advance type declaration look in Hy? From the PEP, we can see that it can be done either in an assignment statement, or in advance of it. So it might make sense to put it in the def form somehow. (And perhaps not in setv.) So (def foo str) would compile like the foo: str declaration in Python, and (def foo str "Foo!") would compile like the declaration with assignment foo: str = "Foo!". This means it can't take multiple pairs like setv anymore.

But mypy uses both variable and function annotations. They should have a consistent syntax. We should seriously consider adopting Clojure's ^ metadata syntax to mean Python's annotations in both cases. We'd want to rename the bitwise operators to free up ^ for the metadata syntax. And also | for symbol quoting #1117, and & instead of &rest (like Clojure), and we can make the grammar for ~ more consistent. (It currently depends on whitespace to distinguish an unquote from a bitwise-not.)

We're only using def in the first place to be like Clojure. def starts a function declaration in Python, which is a completely different meaning. If we want to support dynamic/special variables (#1089), we should probably reserve def for that purpose, since Clojure's special variable equivalent is its "thread-local var" ref type, which it defines using def.

[gilch]

@kirbyfan64 do you know what happened after python/mypy#590? Could mypy be made to work on Hy on Python 3.6+ if we put the type annotations in the AST? Would stub files work now?

[gilch]

@hylang/core @vodik I'm uncomfortable dictating new syntax without feedback. Variable annotations could be coming in #1475.

The current proposal.

(def foo str)  ; foo: str
(def foo str "Foo!")  ; foo: str = "Foo!". 

That means def can't do multiple pairs like setv anymore. If you want to declare multiple pairs we either have to use multiple defs or distinguish the type from the variable name somehow.

Option A

(current proposal--multiple defs)

(def foo str)
(def bar int 42)

Option B

(Clojure-like--use ^ to distinguish type from variable name)

(def ^str foo
     ^int bar 42)

B may be more consistent with function annotation proposals. #656, which follow Clojure syntax for this kind of thing (like we're already doing for most of Hy). It would be nice if these were consistent.

But Option A gives us an excuse to keep both def and setv.

Option C

make def work as in option A, and make setv a macro using the ^ annotations that expands to def(s).

(setv ^str foo
      ^int bar 42)

expands to

(do
  (def foo str)
  (def bar int 42))

[gilch]

Wait, def in Option A has no way of assigning a variable without a type, so we can't expand setv to that as in Option C. It still has to be a special form.

[Kodiologist]

I guess I like Option A the best. Using sigils in Lisp is always a bit weird.

[vodik]

I guess another option could be to use a keyword like how import uses :as:

(def foo 42 :as int)

I also played around taking a symbol as an annotation: (def foo 42 'int) but that feels a little weird.

I do kinda like how racket does it with (: ...) but that's not really workable here as it collides with keywords. The Racket syntax is really close to Pythons.

That said, something worth noting is how painful type annotations are possible going to be as they've been implemented in the typing library. For example, consider the following Python stub:

def get_users(id: int) -> Generator[User, None, None]: ...

That Generator expression is effectively (get Generator (, User None None)) which is going to get unwieldy fast. This was the biggest reason for in #640 I was musing if it was possible to completely separate annotations and functions, but I don't think its going to be workable - as far as I can tell.

But this problem makes me wonder if Option B is best, and instead provided a mechanism for defining new type aliases and types as a practical solution - and one admittedly probably too tied to the current Python implementation of types. I do like how Clojure offers ^ints for a list of ints, for example, instead of (get List int), I wonder if we could do something like (defalias ints List [int])...

Another option is to only support string annotations, in which case the annotation would litterally be "List[int]". Maybe that's the best/least surprising. Especially as deferred evaluation is around the corner anyways. The catch here would be that List still needs to be imported from typing though, as I understand it, and maybe that's going to be surprising.

[gilch]

Using sigils in Lisp is always a bit weird.

Like the ' ` ~ ~@ : "sigils" we already use? Clojure does it with ^.

That Generator expression is effectively (get Generator (, User None None)) which is going to get unwieldy fast.

Does (. Generator [(, User None None)]) look any better? It's equivalent in Hy. Tuples inside [] come up a lot in Python, especially with numpy slices. Maybe Hy just needs better syntax for this. But how would you want it to look in general? Hy's tuple sytnax has an overhead of (,) three characters, but then we never need a comma between elements. For four or more elements, Hy's syntax is more compact that Python's. Maybe the . DSL could implicitly add a tuple if the lists have multiple elements, e.g. (. Generator [User None None]).

provided a mechanism for defining new type aliases and types

I thought Python could already do type aliases via the Typing module. So ints = List[int] is (setv ints (. List [int])) in Hy. I'm not sure what else you want here.

only support string annotations

It seems a little weird to me to write annotations in Python when we're writing everything else in Hy. We want our code to be made of data structures so we can manipulate it with macros. If we have to manipulate strings to do our metaprogramming, we might as well just use Python.

[vodik]

Does (. Generator [(, User None None)]) look any better? It's equivalent in Hy. Tuples inside [] come up a lot in Python, especially with numpy slices. Maybe Hy just needs better syntax for this. But how would you want it to look in general?

It does actually. Didn't know that.

I thought Python could already do type aliases via the Typing module. So ints = List[int] is (setv ints (. List [int])) in Hy. I'm not sure what else you want here.

My concern was that ints would show up as the type hint. But I tested it and it works as expected:

In [1]: import typing

In [2]: ints = typing.List[int]

In [3]: def foo(bar: ints) -> None:
   ...:     pass
   ...: 

In [4]: foo?
Signature: foo(bar:List[int]) -> None
Docstring: <no docstring>
File:      ~/<ipython-input-3-a26253ce8ac9>
Type:      function

So I'm completely happy with that. No need for anything special then. Should have tried that first.

It seems a little weird to me to write annotations in Python when we're writing everything in Hy.

Just musing out loud. I was worried about potential complication of PEP 563, but I somehow missed this line my first few glances through it:

The string form is obtained from the AST during the compilation step, which means that the string form might not preserve the exact formatting of the source. Note: if an annotation was a string literal already, it will still be wrapped in a string.

Which means its really a non-problem as well and this is a lot more straightforward than I was worried it be.

[Kodiologist]

def is gone as of #1483.