-
Notifications
You must be signed in to change notification settings - Fork 371
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
.pyc files don't get recreated when they use a macro from a file that was changed #1324
Comments
I can't reproduce this with Hy 0.13.0 and Python 3.6:
|
I'd start looking here https://github.com/hylang/hy/blob/master/hy/importer.py#L57 @paultag, a reliable reproduction guide would be helpful. There was a recent automation of hyc #1094 #1269. |
Weird. The above works for me too, on I can't get a minimal reproduction at the moment - I'm seeing weird stuff happening, but I did see this happen in the code, and removing the exact .pyc from the cache caused it to recompile the code and start working. I lost a few minutes on that. Let me try to figure out why it's doing this |
Oh crap I know why. I have a macro in a file and I was only changing the macro. I bet that when you have a macro in one file, require it in another, and generate the file off the macro, it will cache the pyc, and modifying the original file won't cause anything using that macro to update. |
Writing a reproduction now |
Is this related to #1268? |
Sorta, this would still be an issue if that was fixed |
|
The issue is things we require may change the bytecode, and we have no way of knowing without going through requires and invalidating if the required file(s) have been updated after our bytecode. I remain skeptical hacking that on top would work. I don't know that we should be writing .pyc files out by default -- maybe we need some sort of production-mode or compile-package helper and let admins do this explicitly? |
I actually made this example evil by not using require - so that it proves the point about how hard this would be to detect |
The fact that macros are only run when a file is byte-compiled is a feature, not a bug. As you note, there's no way to check that a macro would expand to something different from last time without actually running it.
So I'm inclined to regard this as a gotcha that the programmer needs to be aware of, not a bug in Hy itself. |
I didn't know .pyc files were being generated - this is a huge gotcha, and I'm going to claim that we ought to not magically write out pyc without a human saying they should |
It should absolutely be documented. This was bigger change than we thought. @Kodiologist is calling this a feature, not a bug. I think macros are pointless without compilation. If you're interpreting only, you might as well use fexprs, right? But how do Clojure and Common Lisp handle this kind of situation? We should compare those before deciding. As I recall, some CL implementations have both a compiler and interpreter, but |
As of f2278cf, Hy respects Python's environment variable
That's a good question. I was always pretty mystified about how Common Lisp handles these issues. |
Can't Hy just track the hashes of both the main file and its requirements? |
IMO this is an incredibly bad bug that violates one of the core features of Hy... |
Also @paultag is alive :D |
No; e.g., #1324 (comment). Similarly, a macro can choose a file to import randomly. |
We made compilation automatic because Python does it that way. The speedup is considerable and compiling each file by hand seems like a pain. But there might be more nuanced options than global opt in or out. In any case, auto-compiling Hy itself (the libraries) seems like a good idea.
Common Lisp compiles forms rather than files. You can compile the contents of a file, but most implementations have no option do this at the command line. The standard way to do it is from the repl with the Python also has a This way we could turn off auto-compilation on a per-import basis while developing, without turning it off for everything else, and then turn it back on for production. Perhaps a context manager or modified import macro could make this easy to do. (This kind of flag would be a dynamic variable in Common Lisp.) Clojure usually compiles at load time. But it also has an ahead-of-time (AOT) compiler for namespaces. Certain Java interop features require a namespace to be compiled AOT. This is also usually done at the repl using the |
Correctness should always trump speed |
No, it did not. The environment variable is what's checked. |
I'm going to relabel this issue, since the compiler is working as designed, although the design is questionable. |
This. This is Python, not C++. The developer expects full correctness first.
But if the required file changes, it's decently likely that the macro will have changed in some way. So just track the hashes of required files, as well as the hashes of the files that are required by the required files and so forth. This is what C build systems do with headers anyway. IMO this is a pretty horrible gotcha... |
Would it be appropriate if I added this gotcha to the documentation somewhere? I'm a newbie to Hy and was bit by this one too. I thought the behavior was a bug until I found this issue. |
Yes, the tutorial now mentions bytecode, but beyond that, I don't think automatic byte-compilation or its consequences are documented. |
Update: they are now. |
I steped on this issue too but I found the answer in the log.
But I wrote and I ran:
I am really confused now. Why is it created?
FYI: In Windows 10, Hy works fine too.
The description below is what I tried but is not supported in Hy. I wrote a python code to test to avoid creating __pycache__ and I ran:
But in Hy:
|
Sorry if I'm missing something, but I don't see any reference to this issue in the docs. Could you point it out? I agree with @Kodiologist - I don't think that this behavior is a bug. It's just a consequence of that fact that macros are fully expanded out at compile time. Maybe we should include an additional flag for hy to regenerate all |
The section "Macros" in tutorial.rst begins with an example of automatic byte-compilation and what that means for macro expansion. I didn't mean to imply that there's an example of the kind of gotcha this issue is about. |
I agree that nominally this isn't a bug as long as you understand the compile-time vs run-time modes, but it is fairly unintuitive for new users (and even more experienced users -- I've already tripped over this on at least a couple occasions). One possible solution (expanding on #1324 (comment)) is to have Hy write additional files to the |
It seems like Hy is using the .pyc cache even if the file we're loading has a newer filesystem mtime than the .pyc file -- Python will default to overwriting the pyc.
This winds up biting people who update a script and wind up very confused about why something isn't being invoked :)
(Just bit me on the version on pypi - anyone know if this is being tracked with another issue?)
The text was updated successfully, but these errors were encountered: