Optimize speed #13
Comments
|
(cc @exercism/scala) |
|
If you run |
|
@MeerKatDev Would you be able to PR that? Erik is off for the next 10 days and it would be nice to get Scala in the beta with a working test runner! :) |
|
@MeerKatDev It should be a question of using If you have Docker installed, you can verify that everything still works by running |
|
Even with #14 merged, the testing still times out :( |
|
@ErikSchierboom I left a comment in the PR about it |
|
Commented |
|
I don't understand, run-tests takes few seconds in the first cases. What's running so slow? |
|
That's what we're trying to find out :) |
|
I mean, which command do you run? What exact operation? Until I cannot see on my local what's taking so long I cannot help :) |
|
There is of course a difference between running tests locally and on a remote machine that probably has less resources. Also, run-tests is running things locally, not in the Docker container so that is a huge difference. Do you have Docker installed? |
|
I have, and docker is taking quite some time, but that's due to the building process, only that takes 50+ seconds |
|
That should only be the first time. If you want to time a individual test run, you can run: |
|
but docker is running EDIT: Also, I get this: and I assemblied it just before |
Yeah, it runs sbt assembly once when creating the Docker image. Afterwards, each time a Docker container is created from the image, that image already contains the compiled test runner.
That means there must be local changes to the |
|
I modified it because I had already the jar compiled, so I changed the casing for it to be found. I think there is some issue with the searched directory, at least on my local. which apparently is a version-related issue, so I'm using scala-2.13 in local. More in general, I think the execution velocity is dependent on one of these factors:
The jar takes max 2 seconds on my local to execute, it cannot take much more on any other machine. So I think the culprit is either one of the above or a mix of the two. |
|
Just for the sake of clarity: The issue at hand here is the time here, after running the command you gave me above, that: the time it takes between |
|
That is correct. I'm fairly sure that the line that is being slow is running |
|
can you see the runtime of the tests in the logs? In the part where it says this: above, per each test we can see the time it takes to run adding this to the
|
|
@MeerKatDev I would have to add that to the exercise's |
that was what I was aiming at, yes. Add logging and timing wherever you can and run it in the production environment, to see what's taking so long. |
|
Do you know how to output more verbose logging with compile steps etc? |
|
About general information, as they wrote here, this is the maximum we can achieve. For timing, that has to be done manually from the scala code itself, or with the arguments I wrote above for the tests |
|
I've run the partial fail test with an additional If I'm reading this correctly, just compiling the source and the tests ends up taking 10 seconds! The actual running of the tests is fast enough: |
|
I see. Good effort! Have you tried to tune jvm parameters to give, for example, the maximum amount of memory the machine can offer? here a good example. I'd say maybe look in the Java test runner for this, but maybe they don't use sbt. What can also speed up, it's first entering in the |
I haven't tried that, as I know nothing about the JVM. But I'll look at the example.
sbt is Scala-specific unfortunately
Yeah I read about that, but I don't think it's possible for us to do that as we only have a single sbt command that is being executed. |
|
@MeerKatDev I've tried the settings suggested in the example, but to no avail. The compile times are virtually identical. We'll have to try something else. |
|
the trouble is that in some systems the jvm doesn't respond to those parameters for some reason. Also, the settings in that example might be too high, so it won't take them. Try with 512M, 1G, and so on. It also depends on the JVM used.
Sbt can be used to run Java too :) I found this article, if you're in the mood for some profiling |
Really? I've never seen that being used, but that's fascinating! I'll look at the other article. |
|
The profiling is too much for me. I also don't really want to dive into the JVM tweaking, so I'd rather someone experienced in tuning Scala/SBT/JVM looks into that. It will take too much time for me. |
|
Indeed. While someone like that will appear, I could try and look into it in the next days, which machine are you running this in? Can you give me some specs like OS, memory, cpu, and so on? |
|
Good question. I'll defer to @iHiD for the exact specs. One thing to consider is that by default, the Docker container runs with 3GB of memory (which you can simulate by passing |
|
Erm, the specs will vary, so I wouldn't want to commit to anything. I think Erik's benchmarks are all him just running locally. In production it's a pretty heavy machine - it should be as good as any laptop/PC we have in production, but as Erik says it's resource limited to 3GB currently. |
ErikSchierboom
added
x:action/improve
|
@ErikSchierboom can you test something?
What could be happening here is that the price you are paying to load I experimented a bit locally and that seems to be the case. Wdyt? |
|
@jcranky I've retested things a bit, and it looks like the test runner is actually a bit faster now. No idea why. I'm tempted to at least temporarily enable the test runner and see how it fares on our production infrastructure. That said, it is still quite slow. Here are the timings: Running sbt versionreal 0m3.335s
user 0m15.682s
sys 0m1.262sRunning sbt testreal 0m6.692s
user 0m29.615s
sys 0m2.230sRunning the jarreal 0m0.245s
user 0m0.259s
sys 0m0.157sWe can see that sbt is taking about 3 seconds to run just |
|
running commands like Also, there is this: https://github.com/scalacenter/bloop but I never used it. It could be tested in place of sbt. |
|
@MeerKatDev That works great locally, but I don't see how that would work in Docker. We're already running |
Unfortunately I don't think sbt does that. That was the point I was trying to make: you are paying the sbt loading time price, all the time, and that is painful - quite a waste, actually 😄. sbt was not meant to be used like that. The real fix here is probably to completely take sbt out of the picture. I have some ideas and will try to find some time to experiment a bit 😸 |
|
Ah okay, I didn't know! Looking forward to any different options to use. |
|
@ErikSchierboom a question: the tests for the exercises are generated with the |
|
They're part of each exercise: https://github.com/exercism/scala/tree/main/exercises/practice/allergies |
The Scala test runner is functional right now, but it is quite slow. In fact, it is so slow that the test runner times out before being able to return the results. We should speedup the test runner to not have it timeout.
Some possibly useful references:
The text was updated successfully, but these errors were encountered: