Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow avoiding recompilation #37

Merged
merged 3 commits into from
Jan 13, 2022
Merged

Allow avoiding recompilation #37

merged 3 commits into from
Jan 13, 2022

Conversation

rikhuijzer
Copy link
Owner

@rikhuijzer rikhuijzer commented Jan 13, 2022

This PR adds a keyword argument use_distributed which can be disabled to disable parallel processing. Instead, everything will be executed in the same process which avoids recompiling for each process. This can be more quick in systems with few threads such as GitHub Runners depending on the workload.

Also, rename run_notebook to run_notebook! because it mutates the first argument.

@rikhuijzer
Copy link
Owner Author

rikhuijzer commented Jan 13, 2022

0a21852 manages to avoid recompilation completely. All tutorials are evaluated in the same process:

Sampling 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| Time: 0:00:20
106.714471 seconds (237.59 M allocations: 22.918 GiB, 4.17% gc time, 1.20% compilation time)
[ Info: for linear_regression.jl
[ Info: Starting evaluation of Pluto notebook logistic_regression.jl
  Activating environment at `~/git/TuringGLM.jl/docs/Project.toml`
   Resolving package versions...
  No Changes to `~/git/TuringGLM.jl/docs/Project.toml`
  No Changes to `~/git/TuringGLM.jl/docs/Manifest.toml`
┌ Info: Found initial step size
└   ϵ = 0.003125
Sampling 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| Time: 0:00:27
 41.427996 seconds (42.97 M allocations: 3.075 GiB, 1.58% gc time, 10.62% compilation time)
[ Info: for logistic_regression.jl

notice only 3 GiB allocations on the second tutorial

@rikhuijzer rikhuijzer changed the title Avoid recompilation Allow avoiding recompilation Jan 13, 2022
@rikhuijzer
Copy link
Owner Author

I've tested this on TuringGLM's tutorial building. Everything appears to be fine

@rikhuijzer rikhuijzer enabled auto-merge (squash) January 13, 2022 13:23
@rikhuijzer rikhuijzer merged commit 291f1e9 into main Jan 13, 2022
@rikhuijzer rikhuijzer deleted the avoid-recompilation branch January 13, 2022 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant