Very slow precompile (>50min) on julia 1.6.0 on Windows #1554
Comments
|
What happens if you try to precompile Flux on windows again? Is this behavior reproducible? I should reinstate windows CI too |
|
I tried deleting everything under |
|
Weird, it would be difficult to address if we can't reliably reproduce this, and CI hasn't shown that behaviour either. |
|
Now saw a similar thing on a different machine (the top part of this was no longer available in the terminal by scrolling up) |
|
Are you possibly running a slow disk? I've found that precompiling (not Flux, but in general) on a cold disk cache on an HDD can be very, very slow. Other avenues to explore might be whether the AV on Windows is interfering with this somehow. |
|
I'm having the same issue (~20 mins on my machine). I'm using an SSD and disabled my AV. |
|
These issues are hard to fix without reproducers. Do you observe the same behavior consistently if you nuke the compiled versions of all the dependencies? |
|
No, I've just deleted |
|
I'm having the same issue on an AWS box with Ubuntu 18.04 -- precompilation seems to take upwards of 20+ minutes. Honestly, I haven't finished it yet because I keep getting frustrated. I've repeatedly nuked Box info: |
|
Ah -- I think I may realize the issue. It's possible that it's pulling the CUDA drivers in > 1.6 but not showing that it is doing it. |
|
I think I have the same problem of pulling CUDA drivers. But no message is shown, and this process takes a lot of time (even if my connection is not bad). But I did left it precompiling for two hours, with no success (linux 64bits, Julia 1.6.1, SSD disk): Now when I try to use Flux, this occurs: (this has been running for more than an hour now) |
|
Cc @maleadt |
|
Do you have slow internet? If not, the package servers may be experiencing an issue again (cc @staticfloat). That said, Flux.jl should not initialize CUDA.jl when loading the package, but only when the user calls the |
|
No, my internet is not particularly slow, it is a broadband, I can download gbs in few minutes. Now it is like this: |
|
Just for the record, it finally finished and apparently worked. |
|
You can always try with |
|
If I understand correctly, I did I am now at this:
It is unclear if that is downloading anything behind the scenes or not. If the problem is the download speed, probably this is an issue that should be reported somewhere else, to improve the user experience at least such that the user knows that something is going on, and not just a stall. edit: finished in ~5 minutes. |
|
It never downloads CUDA artifacts during an |
|
Uhm... so maybe there are two separate issues here. Yesterday night I had to Control-C the precompilation (which was on the above state) because it seemed to never end (>2 hours). Today I tried After removing |
|
Can you also show me the output of |
|
This actually does not seem to be true anymore. With Julia v1.6.1 and a fresh depot on cyclops, I see @lmiq can you test how long it takes to download this artifact on your machine? You can test by doing something like: On my machine, I get an average download speed of ~8MB/s, which means it takes a little under two minutes to download the full artifact. |
Ah yes, Flux does bad things. I've created #1597 to track this. |
|
My connection is not as good but, for example, this download occurred with 0.8Mb/s: while the artifact is still downloading: at, for the moment, at ~0.25Mb/s in average if I computed it correctly. Both are not very good, but we suffer most fetching the file from the Julia server. |
|
Hmmm, I can download from the same server at much higher speeds, so I'm afraid this is most likely an issue due to your internet connection. Any way you cut it, downloading an 800MB file at <1MB/s is going to take a long time, and many things can go wrong during a download that takes >10 minutes. Not sure there's much we can do to improve this right now. |
|
Thanks, I completely understand. I will check if there is a Julia server available nearby (at one of the Brazilian universities) from which I can fetch things at higher speed. If not, perhaps I will think about building one. So even the precompilation time was associated to that, as pointed by @maleadt ? Should I/we/someone report an issue in general for download be dissociated from precompilation? Or at least show a progress bar? |
|
We tend to use amazon lightsail for hosting, but we are open to working
with you or other partners to provide package servers that are
geographically closer to you! Lightsail unfortunately doesn’t have a South
American data center, and the EC2 billing structure is much more expensive
since our main cost is bandwidth.
If you have a university that wants to sponsor a Pkg server, I would be
happy to help you set that up. It’s actually quite easy these days. We use
the 1 core, 2GB RAM, 60 GB storage tier for most of our pkg servers. I
would even be willing to help you set it up, if you can get me shell
access, all I really need is the ability to launch docker containers and
login to administer it.
-E
…On Wed, May 12, 2021 at 10:54 Leandro Martínez ***@***.***> wrote:
Thanks, I completely understand. I will check if there is a Julia server
available nearby (at one of the Brazilian universities) from which I can
fetch things at higher speed. If not, perhaps I will think about building
one.
So even the precompilation time was associated to that, as pointed by
@maleadt <https://github.com/maleadt> ?
Should I/we/someone report an issue in general for download be dissociated
from precompilation? Or at least show a progress bar?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1554 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAA762F3POBNLA3S3MLJYZ3TNK6FDANCNFSM42DBRGTQ>
.
|
|
I could easily give you access to a machine with those characteristics, and that would be fine if the traffic is not too high to/from it. But for a more persistent and supported solution I need to talk to the university staff. But it may be possible. Another university here that traditionally hosts repositories is UFPR. I will try to link one colleague from there: @adolfont Adolfo, how are you? The UFPR has a long tradition in hosting repositories for linux distributions. Do you think it would be easy there to host a repository for Juila? |
|
@lmiq I actually ended up being able to set up an EC2 instance in Sao Paulo today; so in the short term, we won't need to partner with a university (although they are of course welcome to set up a pkg server for their own internal use; it's quite easy. Can you try some Pkg operations again now and see if things are any faster? You can check that the Pkg server you're connecting to is the new south american one by looking at the |
|
That download is definitely (much) faster: Package operations appear to be faster now (although is night now, I will try again tomorrow at working hours). edit: Day-time, and still very, very fast! That changes significantly the user experience here. Thank you! @staticfloat |
Hi, @lmiq. I actually work at UTFPR Curitiba but I have friends at UFPR Curitiba. I understand part of the problem was already solved ("in the short term, we won't need to partner with a university" @staticfloat) but that we/they could set up a pkg server for our/their own internal use). I am not sure there is enough people using Julia here to justify that. In case there is, I will point them here. |
|
@adolfont Thank you Adolfo! Apparently that is solved indeed. We'll keep one eye on those other possibilities if they become useful. Grande abraço! |
|
@abelsiqueira works at UFPR and has several Julia repos https://github.com/abelsiqueira?tab=repositories |
|
Unfortunately, I'm not involved with the mirrors, they are maintained by a research group: https://www.c3sl.ufpr.br/espelhos/ |
|
Thank you @abelsiqueira . I think that for the moment a good solution was found by establishing a server in São Paulo. If Julia becomes (and hopefully it will :-) ) widely used here, it may be interesting to have a server within the university-network. |
Most of that time the progress bar was on 52/53, with only Flux still precompiling.
On WSL on the same machine precompilation finished in 59 seconds (but the environment was not identical)
The text was updated successfully, but these errors were encountered: