approach to default packages #23876

JeffBezanson · 2017-09-25T21:41:11Z

Here's a strawman to get started on #18795. It's simple but basically does what I'd want. Here's what I did:

Make a new top-level directory called stdlib.
As a demo, move DataFmt (readdlm and writedlm) to stdlib/DelimitedFiles.
During build, link to stdlib from usr/share/julia/site/VERSION so it's available in LOAD_PATH. The files will be copied by make install as well.
(optional) Put using DelimitedFiles in the default juliarc, so the default repl experience is identical to before.

It should also be possible to put the using Whatever in sysimg.jl for faster startup. In that case I think there's still a clear code organization improvement; increased coupling can only take the form of adding a single using line to sysimg.jl or juliarc.jl.

Initially, the files can remain in this repo. Eventually, we might want to make some into external packages, and clone those repos into stdlib/ as a build step. By path/depot priority, it will be possible to install different versions of these packages and load them instead of the defaults.

If we move lots of Base submodules right away using this method, we can gain lots of flexibility, e.g. being able to update Distributed using the package manager during 1.x.

vtjnash · 2017-09-27T00:05:07Z

If we add these back to the bottom of sysimg.jl (for now, e.g. outside of Base), I think we can get on with merging this, and then work on making it lazy later in this development cycle. Otherwise, lgtm! Glad to see progress getting made on this.

JeffBezanson · 2017-09-27T03:28:26Z

Done.

quinnj

Exciting! It'd be nice to maybe outline the process somewhere for moving something to /stdlib. Something along the lines of:

Create directories in /stdlib/Package/src and /stdlib/Package/test`
Remove code/references from Base, exports.jl, docs, etc. to /stdlib/Package/src
Move code from /test to /stdlib/Package/test
Add using Package to sysimg.jl

Though what's the general thought on doing sysimg.jl vs. .juliarc.jl?

vtjnash · 2017-09-27T04:36:26Z

base/deprecated.jl

+          :detect_ambiguities, :detect_unbound_args]
+    @eval Base.@deprecate_moved $f "Test" true true
+end
+end


Can we do Base.@deprecate_moved Test "Test" true true, or would that do something different?

@deprecate_moved only seems to support the format @deprecate_moved function package.

JeffBezanson · 2017-09-27T05:12:42Z

@quinnj Adding to sysimg.jl just improves startup time. Adding to juliarc.jl makes the names still available by default in the repl. For each package you can pick whether to do one or both of those.

I think I'm going to need help with the doc build for this. How can we set things up so docs live in package directories, but each one becomes a section of the manual?

KristofferC · 2017-09-27T12:09:07Z

I did the following hack to seems to make the docs generate like before:

Move the doc/src/stdlib/test.md file to e.g stdlib/Test/docs/src/index.md.
Change all the Base.Test in that file to just be Test.
Make a symlink from the current doc/src/stdlib/test.md to the moved stdlib/Test/docs/src/index.md.
Add using Test to doc/make.jl above the makedocs call.

Diff: https://gist.github.com/KristofferC/d0dd3009fd42d13bef91c919f186c3b7

Not pretty but perhaps it allows for moving forward on this and move out more packages, and a better workflow for the documentation could be worked out later.

This Documenter issue and PR are relevant: JuliaDocs/Documenter.jl#551, JuliaDocs/Documenter.jl#552

KristofferC · 2017-09-27T13:12:31Z

Any thought of what should be done with packages that are somehow used in precompile.jl?

yuyichao · 2017-09-27T13:29:33Z

Just have their own precompile?

KristofferC · 2017-09-27T13:33:40Z

Yeah, of course :). I forgot that since they are included in the sysimg.jl the precompile.jl in the stdlib packages will work just like if it was inside the main precompile.jl.

JeffBezanson · 2017-09-27T16:17:20Z

Thanks, @KristofferC ! I think this should be ready to go now.

vchuravy · 2017-09-27T16:36:32Z

base/deprecated.jl

-           end),
+         default_package ? :(function $eold(args...; kwargs...)
+                                 error($eold, " has been moved to the standard library package ", $new, ".\n",
+                                       "Restart Julia and then run `using ", $new, "` to load it.")


There should be no need to restart Julia in this case.

Yeah, maybe, but it's fairly easy to end up with conflicting identifiers and such.

JeffBezanson · 2017-09-27T16:43:26Z

Is there any way to see the doc build log on AV?

EDIT: nvm, I suspect we can't do symlinks there.

KristofferC · 2017-09-27T21:03:46Z

doc/make.jl

@@ -16,6 +16,15 @@ end

 # Documenter Setup.

+# make links for stdlib package docs
+if Sys.iswindows()


Will this dirty the repo if one builds the docs? Locally, I just committed the symlinked file that is being created here.

Will that work everywhere? Do we have any symlinks in the repo currently? I didn't want to be the first :)

I've seen people do it for linking in the README.md into the documentation. Whether it will work everywhere... your guess is much better than mine.

JeffBezanson · 2017-09-28T20:02:25Z

Ok, done. I'm just making symlinks (unix) or copies (windows) during the doc build, with an appropriate .gitignore file. Can be improved later. I believe this is now ready to merge.

as a demo, moves DataFmt out of Base. adds a line to default juliarc so the functions are still available by default.

StefanKarpinski · 2017-09-29T15:48:31Z

Do we want to call this stdlib? The set of functionality that ships with a language is often called the "standard library", which would make that name kind of confusing. If we call this directory stdlib then we should be ok with referring to package in it as part of Julia's standard library.

JeffBezanson · 2017-09-29T16:00:00Z

Yes, I'm ok with considering everything in there the standard library. I like what wikipedia says in the very first paragraph of that entry though: "contents of a language's associated library may also be determined (in part or whole) by more informal practices of a language's community". So what happens to be in stdlib is not necessarily the end of the story.

Another good name might be stdpkg, for standard packages. Any other ideas?

StefanKarpinski · 2017-09-29T18:08:48Z

The name stdpkg was also bouncing around in my head.

fredrikekre · 2017-10-01T10:22:34Z

julia> using Base.Test
WARNING: Base.Test is deprecated, use Base.Test instead.
  likely near no file:0

rfourquet · 2017-10-03T14:34:39Z

Running make after modifying "stdlib/Test/src" doesn't re-build Julia, but using Test loads the old version... what is the way to rebuild a package in "stdlib"?

JeffBezanson force-pushed the jb/stdlib branch from 2581508 to 4d326b0 Compare September 25, 2017 21:43

ararslan added the packages Package management and loading label Sep 25, 2017

JeffBezanson force-pushed the jb/stdlib branch 2 times, most recently from 5cb8252 to 1e65047 Compare September 26, 2017 04:08

JeffBezanson added the excision Removal of code from Base or the repository label Sep 26, 2017

JeffBezanson force-pushed the jb/stdlib branch from 34e4425 to bb244c3 Compare September 26, 2017 22:33

JeffBezanson force-pushed the jb/stdlib branch from bb244c3 to 568fb33 Compare September 27, 2017 03:22

quinnj approved these changes Sep 27, 2017

View reviewed changes

vtjnash approved these changes Sep 27, 2017

View reviewed changes

JeffBezanson force-pushed the jb/stdlib branch from 568fb33 to f7cf4f9 Compare September 27, 2017 16:16

vchuravy reviewed Sep 27, 2017

View reviewed changes

JeffBezanson force-pushed the jb/stdlib branch 2 times, most recently from badbe0c to ab1878f Compare September 27, 2017 18:54

KristofferC reviewed Sep 27, 2017

View reviewed changes

JeffBezanson force-pushed the jb/stdlib branch from ab1878f to cbda9d0 Compare September 28, 2017 01:36

JeffBezanson changed the title ~~RFC: prototype approach to default packages~~ approach to default packages Sep 28, 2017

JeffBezanson force-pushed the jb/stdlib branch from cbda9d0 to e562890 Compare September 29, 2017 02:03

JeffBezanson added 2 commits September 28, 2017 23:09

reorganize Base with default packages, installed in site/

4a6e64d

as a demo, moves DataFmt out of Base. adds a line to default juliarc so the functions are still available by default.

move Test from Base to stdlib

d759779

JeffBezanson force-pushed the jb/stdlib branch from e562890 to d759779 Compare September 29, 2017 03:10

JeffBezanson merged commit ae8bbf6 into master Sep 29, 2017

JeffBezanson deleted the jb/stdlib branch September 29, 2017 19:16

PerezHz mentioned this pull request Oct 3, 2017

norm methods for AbstractSeries JuliaDiff/TaylorSeries.jl#128

Merged

rdeits mentioned this pull request Oct 3, 2017

Add isgood function JuliaRobotics/LCMCore.jl#15

Merged

PerezHz added a commit to PerezHz/HORIZONS.jl that referenced this pull request Oct 4, 2017

Fix julia version associated to JuliaLang/julia#23876

7d054d2

PerezHz mentioned this pull request Oct 4, 2017

Deal with recent breaking changes on julia nightlies PerezHz/TaylorIntegration.jl#39

Closed

Sacha0 mentioned this pull request Nov 4, 2017

Split the docs for "Standard Library" into "Base" and "Standard Library" #24461

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

approach to default packages #23876

approach to default packages #23876

JeffBezanson commented Sep 25, 2017

vtjnash commented Sep 27, 2017 •

edited

Loading

JeffBezanson commented Sep 27, 2017

quinnj left a comment

vtjnash Sep 27, 2017

JeffBezanson Sep 27, 2017

JeffBezanson commented Sep 27, 2017

KristofferC commented Sep 27, 2017 •

edited

Loading

KristofferC commented Sep 27, 2017

yuyichao commented Sep 27, 2017

KristofferC commented Sep 27, 2017

JeffBezanson commented Sep 27, 2017

vchuravy Sep 27, 2017

JeffBezanson Sep 27, 2017

JeffBezanson commented Sep 27, 2017 •

edited

Loading

KristofferC Sep 27, 2017

JeffBezanson Sep 27, 2017

KristofferC Sep 27, 2017

JeffBezanson commented Sep 28, 2017

StefanKarpinski commented Sep 29, 2017

JeffBezanson commented Sep 29, 2017

StefanKarpinski commented Sep 29, 2017

fredrikekre commented Oct 1, 2017

rfourquet commented Oct 3, 2017

approach to default packages #23876

approach to default packages #23876

Conversation

JeffBezanson commented Sep 25, 2017

vtjnash commented Sep 27, 2017 • edited Loading

JeffBezanson commented Sep 27, 2017

quinnj left a comment

Choose a reason for hiding this comment

vtjnash Sep 27, 2017

Choose a reason for hiding this comment

JeffBezanson Sep 27, 2017

Choose a reason for hiding this comment

JeffBezanson commented Sep 27, 2017

KristofferC commented Sep 27, 2017 • edited Loading

KristofferC commented Sep 27, 2017

yuyichao commented Sep 27, 2017

KristofferC commented Sep 27, 2017

JeffBezanson commented Sep 27, 2017

vchuravy Sep 27, 2017

Choose a reason for hiding this comment

JeffBezanson Sep 27, 2017

Choose a reason for hiding this comment

JeffBezanson commented Sep 27, 2017 • edited Loading

KristofferC Sep 27, 2017

Choose a reason for hiding this comment

JeffBezanson Sep 27, 2017

Choose a reason for hiding this comment

KristofferC Sep 27, 2017

Choose a reason for hiding this comment

JeffBezanson commented Sep 28, 2017

StefanKarpinski commented Sep 29, 2017

JeffBezanson commented Sep 29, 2017

StefanKarpinski commented Sep 29, 2017

fredrikekre commented Oct 1, 2017

rfourquet commented Oct 3, 2017

vtjnash commented Sep 27, 2017 •

edited

Loading

KristofferC commented Sep 27, 2017 •

edited

Loading

JeffBezanson commented Sep 27, 2017 •

edited

Loading