Recompiling .hy files when macro dependencies have changed

[scauligi]

So currently I'm working on a project where I've got a "macro library" in one Hy source file, as well as several other files that use the higher level API that the macros provide. The API itself stays pretty much unchanged, so most of my development work is in the macros themselves.

The annoyance is that every time I change/update/bugfix a macro, I have to touch/resave all the other files before I re-run to force them to recompile. Not a huge deal, but an annoyance nonetheless.

@Kodiologist I wanted to pick your brain if you had any tips or a better workflow for this? And otherwise discuss potential solutions -- in #1324 and #2308 I proposed that Hy could save macro dependencies alongside .pyc files to automatically check when macros may have changed. Do you still recommend against this? (And why? I'm still not totally clear)

Alternatively (or additionally, they're not mutually exclusive), we could add a new flag to hy to force it to recompile any .hy files it executes/imports/requires, kind of like a counterpart to -B?

[Kodiologist]

What I tend to do in many cases is git clean -qdfx {.,**}/__pycache__ && pytest (in Zsh), so I know all the bytecode is regenerated before testing. It's overkill, but it only takes another 1 to 3 seconds for a medium-to-large project like Infinitesimal Quest 2 + ε, and it saves me from having to think about whether the bytecode is actually out of date. If you're working largely on the macros that are required by other files, and so you expect the bytecode to be out of date pretty much every time, perhaps you should use PYTHONDONTWRITEBYTECODE.

I've never totally thought through the implications of your proposal to save macro dependencies. The reason I've been inclined against it is that I expect you could protect against common cases of out-of-date bytecode, but not all cases. The most pathological cases I've thought of so far are the ones I mentioned in #1324, where a macro returns a nondeterministic result, or the file that macros are drawn from is chosen nondeterministically. I'd rather have the neophyte Hy programmer be able to see and understand this problem in the simple case first rather than be protected against it for a while until he begins to do something more complicated. As a bonus, the programmer has the option of not regenerating bytecode if he knows that it's not actually necessary even though a macro-containing file has changed (because e.g. the change was to a macro he's not using).

Admittedly, we already have the issue that (print (do-mac (hy.M.random.random))) won't rerandomize the number when you reexecute the file. And maybe those really pathological cases are so pathological that we can be confident the typical programmer will never encounter them. And I've been wrong before about what's too hard to do. I thought let could never quite work and you proved me wrong. I guess the proof is in the pudding.

… we could add a new flag to hy to force it to recompile any .hy files it executes/imports/requires, kind of like a counterpart to -B?

This seems more likely to work in general, but ultimately may not be much of an advantage over git clean -qdfx {.,**}/__pycache__ or touch: you're still forcing bytecode regeneration, without consideration of whether it's truly necessary. I'm not sure. You'll have to deal with permissions issues when the library is installed system-wide and so the user can't change the bytecode.