Tracking issue for libcore + no_std stabilization

[alexcrichton]

This issue is intended to represent the outstanding issues for stabilizing libcore and allowing its usage on stable Rust. There are a number of features currently associated with libcore:

• core
• core_char_ext
• core_prelude
• core_slice_ext
• core_str_ext

(note that core_float will be handled in a separate issue)

The design of libcore largely mirrors that of the standard library (good) but there are a few deviations:

• core::atomic differs from std::sync::atomic
• Modules like nonzero, panicking, and array are public

Overall there are a number of tasks that probably need to be done before stabilizing these items:

• A full audit should be done to ensure that the structure of libcore is the same as the structure of libstd
• The name core needs to be agreed upon as the stable name for the library
• The set of extension traits for primitives needs to be decided upon. This strategy of trait-in-core and inherent-above-core should be agreed upon as the best path forward.
• The set of items in the prelude should be audited to ensure it's a subset of the standard library's

[alexcrichton]

I think that all the major issues here have been resolved, so I'm nominating this for stabilization in 1.5

[bluss]

Is it possible to support multiple impl primitive blocks, so that core doesn't have to use as many extension traits?

[alexcrichton]

Not currently that I'm aware of at least, and that would definitely be part of the stabilization aspect of this issue.

[aturon]

Idea: mark the methods on the extension traits as stable, but leave the traits themselves unstable. Works because they're in the prelude. (We did this for SliceConcatExt.)

Not a great permanent story, but it's a place to start.

[brson]

I want to understand the story for undefined symbols before making this stable. There's a real possibility of people defining these themselves in ways that aren't forward compatible.

[alexcrichton]

This issue/feature is now entering its cycle-long FCP for stabilization in 1.5

---

@brson, we currently have very few undefined symbols actually:

$ nm -u libcore*

core-10cbabc2.0.o:
                 U fmod
                 U fmodf
                 U memcmp
                 U memcpy
                 U memset
                 U __powidf2
                 U __powisf2
                 U rust_begin_unwind
                 U rust_eh_personality

Of these, rust_begin_unwind and rust_eh_personality are verified by the compiler to exist in the form of a lang item (e.g. it's unstable to define them) whenever you're producing a staticlib, executable, or dylib, so we're covered on that front. The memcmp, memcpy, and memset functions are lowered to by LLVM and I think it's safe to say that our reliance on their meaning will never cause breakage. The __powi{d,s}f2 function dependencies may disappear but otherwise they're provided by compiler-rt so I don't think we have to worry about them.

Otherwise, the only symbols we rely on are fmod and fmodf and that in turn is only because Div and Rem must be implemented in libcore instead of being able to define them in libstd for floats. I'm personally comfortable with this because --gc-sections makes the dependency go away and the symbols also have very well known meanings.

Do you have further concerns in light of this information?

[alexcrichton]

Oh and as a further clarification that I forgot to point out, this would also involve stabilizing #![no_std]. As a result I'm tagging this T-lang and renominating to make sure it comes up in triage over there. Triage in all the teams!

[alexcrichton]

And to further refine my comment, I remember now that we have more undefined symbols on 32-bit because some 64-bit integer operations frequently require support from compiler-rt. Taking a look at the undefined symbols from i686-unknown-linux-gnu:

         U __divdi3
         U fmod
         U fmodf
         U _GLOBAL_OFFSET_TABLE_
         U memcmp
         U memcpy
         U memset
         U __moddi3
         U __mulodi4
         U __powidf2
         U __powisf2
         U rust_begin_unwind
         U rust_eh_personality
         U __udivdi3
         U __umoddi3

It's basically the same story here, though, nothing unexpected and I don't think we should worry about people relying on their own incorrect versions of compiler-rt intrinsics.

[briansmith]

Some things I noticed when I worked on adapting ring to work with #![no_std] and libcore:

I don't get why I have to use core::foo instead of std::foo. I ended up having to write things like this:

// Re-export `{mem, ptr}` from either `std` or `core` as appropriate. This
// keeps the conditional logic on one place.
#[cfg(feature = "lib_no_std")]
pub use core::{mem, ptr};
#[cfg(not(feature = "lib_no_std"))]
pub use std::{mem, ptr};

Then I refer to, e.g. ring::ffi::mem::uninitialized() and ring::ffi::ptr::null() from all my other modules. But, it doesn't really make sense. I feel like it would be much better to just be able to use std::ffi::ptr::null() in #![no_std].

Also, it is confusing to me about whether #![no_std] applies to the entire crate or only to individual modules. Especially when building using Cargo, it isn't clear to me how it is useful to have a mix of #![no_std] modules within a single crate.

Finally, the documentation should be clearer about how to write a library that automatically works in both #![no_std]mode and non-#![no_std] mode depending on whether the executable is built with or without #![no_std]. Right now I have my library crate define a feature that controls whether or not I enable #![no_std] in the library, but it would be nice if it were automatic.

[SimonSapin]

@briansmith Why not only use no_std rather than have two modes? If you have optional features that require std, can’t you still unconditionally use use core::… for most of your imports?

[alexcrichton]

@briansmith I totally agree that we can bolster our docs in this respect! As @SimonSapin mentioned, there's actually no need to have a "dual mode" where sometimes you use std and the rest of the time you use core, the intention is for the crate to permanently mention #![no_std] and then everything "just works". In that sense, it sounds like a number of concerns you have would be addressed?

Part of the pain of using #![no_std] today is that if you want to work on both stable and nightly Rust you've got this whole import and duality problem, but that's why I'd like to stabilize #![no_std] :).

Additionally, #![no_std] is only a crate level attribute (which the docs should explain), so there's actually no notion of a "no_std module" vs another module as a whole crate should behave the same way.

Does that help explain things? I can try to bolster up our docs on this subject before we push this over the finish line!

[briansmith]

@briansmith I totally agree that we can bolster our docs in this respect! As @SimonSapin mentioned, there's actually no need to have a "dual mode" where sometimes you use std and the rest of the time you use core, the intention is for the crate to permanently mention #![no_std] and then everything "just works". In that sense, it sounds like a number of concerns you have would be addressed?

When I add this to my lib.rs:

#![feature(no_std)]
#![no_std]

I get:

src\digest.rs:28:5: 28:8 error: unresolved import `std::mem`. Maybe a missing `e
xtern crate std`? [E0432]
src\digest.rs:28 use std::mem;

src\aead.rs:27:5: 27:8 error: unresolved import `std`. There is no `std` in `???
` [E0432]
src\aead.rs:27 use std;

What am I doing wrong? The repo is at https://github.com/briansmith/ring, branch "no_std".

[alexcrichton]

Wow we really do need some documentation on this badly... The detailed design of #![no_std] can currently be found in an RFC, but the gist of it is that #![no_std] doesn't import std at the crate root or the std prelude in every module, but rather core at the crate root and the core prelude in every module.

If you're core-compatible you can basically rewrite all imports to be from core instead of std and you should be good to go!

[briansmith]

OK, if I use core:: instead of std:: it works. But if I remove #![no_std] then it stops working:

src\aead.rs:27:5: 27:9 error: unresolved import `core`. There is no `core` in `?
??` [E0432]
src\aead.rs:27 use core;
                   ^~~~
src\aead.rs:27:5: 27:9 help: run `rustc --explain E0432` to see a detailed expla
nation
src\digest.rs:28:5: 28:9 error: unresolved import `core::mem`. Maybe a missing `
extern crate core`? [E0432]/
src\digest.rs:28 use core::mem;

Why is core only implicitly imported when we use #![no_std]? Why not always implicitly import core like std is implicitly imported?

Also, all the documentation uses std::foo instead of core::foo. Are you planning to update the documentation to use core:: ubiquitously? It is confusing to have two sets of names for the same things.

[briansmith]

Also, let's say my module uses core:: everywhere, and it is imported into a program that uses std:: everywhere. Do I have to worry that there will be code duplication between the core:: and :std:: variants of features?

[alexcrichton]

Why is core only implicitly imported when we use #![no_std]?

At 1.0 we didn't inject the core library, and it would be a breaking change to do so now. Additionally I would personally not want it injected by default as std is "The Standard Library" which should be ubiquitously used instead of a mixture of core/std in my opinion. I would see it as a failure mode if a style guideline existed where in normal Rust code one should use core::foo for imports from core and std::foo for std-only imports.

Are you planning to update the documentation to use core:: ubiquitously?

Not currently, the documentation for #![no_std] would indicate that core is simply a subset of the standard library, so if a code example will work with only libcore it'll work by just rewriting std to core imports.

It is confusing to have two sets of names for the same things.

I think this confusion primarily stems from a lack of documentation. I think it's crucial to maintain the distinction of whether code is in a "core only" or "std only" context. It's important to know what abilities you have access to (e.g. can you do I/O?). Beyond that the structures are exactly the same, so once this piece of information is known I don't think it'll be too confusing (but it definitely needs to be explained!)

Do I have to worry that there will be code duplication between the core:: and :std:: variants of features?

That's actually the great part! Because the standard library simply reexports everything from core, a #![no_std] crate can still be used with any other crate seamlessly, there won't be any duplication of anything.

[briansmith]

I am surprised that it is so difficult to convey how confusing this is.

• Why should the Rust programmer have to wonder whether core::foo is the same as std::foo or not?
• Why should the default be to write code that is incompatible with #![no_std]? In particular, let's say a library only uses features of std that are also in core. Why should that library be fail to work in a #![no_std] executable?
• Why does the documentation recommend coding patterns (using std:: instead of core::) that don't work ubiquitously?
• Why does Rust have an elaborate style guide about how to format and organize code, but not have canonical names for the most core library features, thus making it impossible to write code in a canonical form that works ubiquitously?

IMO, it would make a lot more sense to NOT have #![no_std] and instead have #![feature(no_std)] that disables everything in std:: that requires runtime support, and then deprecate core::. This would be more consistent with how other sets of features within std:: are enabled/disabled and would allow every library feature to have one, canonical, ubiquitous, name.

[briansmith]

Here is the development mode that I see for libcore compatible libraries on crates.io:

1. Somebody makes a library that isn't libcore-compatible.
2. Somebody else that needs the library to be libcore-compatible submits pull requests to make it libcore-compatible so they can use it in their no_std projects.

Step #2 will likely involve making some features of the third-party library conditional on whether std is being used or whether core is being used. Note that it isn't clear how how this conditional enablement of features should work. In particular, which #![cfg(feature = "???")] should be used? In my suggestion above, this would be #![cfg(feature = "no_std")].

Step #2 will also likely involve changing some std:: references to core:: references. According to the Drawbacks section of the RFC, this is likely to be an source-copmatible-breaking change in the library. For example, if the library exports macros that reference std:: and those changes are changed to core::. This would put the third-party library developers in a Catch-22: they either has to choose between backward-compatibility or libcore compatibility. In my suggestion above, this would not be an issue because there would be no separate core::.

[SimonSapin]

core and std are crates that can be used from other crates with extern crate core; or extern crate std;, just like ring is a crate that can be used with extern crate ring. The only thing that makes them special is that, by default, the compiler implicitly injects extern crate std; at the top of every crate and use std::prelude::v1::*; at the top of every module and every crate. #![no_std] inhibits this injection. The plan is that extern crate core; and use core::prelude::v1::*; are injected instead, but I don’t know if that part is implemented yet.

So #![no_std] only influences which crates are used. It can not influence what’s in these crates, that would require re-compiling these crates. We could have two different crates that are both called std, one of which only includes the core functionality, and use rustc --extern std=some/thing.rlib to disambiguate… I don’t have a opinion on whether this would be a good idea.

There is no such thing as a #![no_std] executable, only #![no_std] crates. Each crate has a number of dependencies, which may or may not (implicitly) include std. An executable links the union of all recursive dependencies. It can mix crates that use #![no_std] and crates that don’t. In that case std will be linked.

std has a number of pub use statements that “reexport” items defined in its own dependencies. These items are not duplicated, just given an (additional) other name. An executable that links std also links std’s own dependencies.

[Ericson2314]

To be clear, getting rid of std is most elegant solution to both @dschatzberg points. I hope that its elegance (as opposed to practicality) is uncontroversial.

I think 1 can be refined--it's less that the silent transition from freestanding-capable to not freestanding-capable is the goal, but rather that there be only one style (a "normal form") for writing code that in principle should work free-standing. Right now there is two, with core and with std, and that's just awkward. Of course in a world without std its still possible to extern crate more than needed, but that's a very simple fix, and I consider cleaning imports part of the "stylistic normalization" process.

Now the elastic/subset std also means there is two ways to write such code, but at least they are equivalent in terms of the burdens they place downstream. Right now that the std way is more conventional but core strictly nicer to downstream. That means not only is there two different styles, but meaningful trade-offs between them.

@SimonSapin I'd hope we'd just be stabilizing the things in core that are already stable in std.

[dschatzberg]

@SimonSapin
It seems perfectly reasonable to stabilize #![no_std] and core without the rest. This allows stable #[no_std] library crates. I think the further discussion about stabilizing the necessary support for #[no_std] binaries can be done separately.

• fmod is only used for floating point modulus not modulus in general
• _Unwind_Resume is usually defined in some runtime support library like libgcc

[SimonSapin]

std is not going anywhere. It’s not just a facade, a lot of things are defined there. And there’s plenty of code out there with use std::something; that we promised not to break.

[dschatzberg]

@Ericson2314
I'm not quite sure what you mean about there being two styles for writing freestanding code. Could you elaborate?

@alexcrichton
I believe that making freestanding implicit and allowing users to unknowingly make their library non-freestanding is strictly better than making freestanding explicit. In the explicit case, developers who want to construct freestanding libraries must, well, be explicit about this, otherwise their library cannot be used in a freestanding environment. In the implicit case, those same developers who want to construct freestanding libraries can still use a lint to enforce their desire. The only difference is that freestanding users can still use those libraries whose developers haven't considered the freestanding environment. As a freestanding user this is a pure win - I get more libraries available to me.

[Ericson2314]

@dschatzberg are you on IRC by any chance? might take a few iterations for me to express that clearly :)

[Ericson2314]

@SimonSapin Sure no cold-turkey removal, but we can deprecate it, and since it is so easy to define as a facade, it's easy to keep around without removing. The downside is more appearing fickle making such a visible change after 1.0, but it's more eye-cactching than actually heavy-weight.

[dschatzberg]

@Ericson2314 yes - same user name

[Ericson2314]

Ok, edited #27701 (comment) to hopefully clear up any confusion.

[SimonSapin]

-1 on deprecating std. It’s just not worth the cost, which is huge: every existing program that is not #![no_std] and not trivial imports something from std.

[bstrie]

"Deprecating std" isn't sufficiently well-defined at the moment, we'd need to see an RFC for it before it could be discussed seriously.

[bstrie]

Regardless of whether or not this RFC is accepted, I'd like to propose that the actual stabilization PR be blocked on resolving #23355 , because otherwise we'll have the compiler guiding people into wholly silly situations.

[phil-opp]

I really like the explicit #![no_std]. If you accidentally add some std dependent feature, it fails to compile. If no_std were implicit, it would silently include std and break all downstream crates. Travis would not catch this either, because it compiles without problems.

[bstrie]

Actually never mind about #23355 , I just determined that it's already possible to follow rustc's advice on the stable branch to produce the silly situations that I had in mind.

[dschatzberg]

@phil-opp
Implicit '#![no_std]` + a lint would resolve this issue.

[bluss]

Having no_std explicit or not is a red herring. I think it's very reasonable that if you have a crate using only core features, making the core-only compatible should only be an explicit attribute away. This requires (? may be an assumption too far!) a design where we use the name std for the library in any mode, we just don't fill it with as much features if you compile core-only.

[dschatzberg]

@bluss I don't completely agree. I still think it's beneficial for freestanding users to use libraries for which their use case wasn't intended. While an explicit attribute is a relatively minor change, it's still a burden on the library developer to think to do it.

[Ericson2314]

@bstrie "deprecating std" in my mind would be encouraging imports from crates behind the facade rather than std were the former is stable. Once everything is available from a crate behind the facade, then std cannot be used at all without tripping a "try use core/alloc/instead for foobar" warning.

I would write an RFC for that, but I rather iron out the kinks in using the crates behind the facade exclusively first, so it's clear what we would be migrating to in the deprecate std RFC.

[bluss]

Just my opinion, but deprecating or linting std is a non-starter, no use in exploring that idea.

[phil-opp]

AFAIK the std library is more than a facade. It implements everything operating specific (like threads, files, network, etc.).

So to deprecate std, we would need to move those parts to a new crate. And then the "normal" user would have to think about implementation details. For example, Hashmap is only in std (because it uses a thread local random number generator) but BTreeMap is also in collections.

[Ericson2314]

@phil-opp I'd consider finishing making std a facade a good thing whether or not we keep std. For example, embedded Systems ("internet of things") may have networking but not filesystems. As for your example, only the default hasher param uses the OS like that. I think everybody rather put HashMap in collections, and just have a wrapper type alias that adds the default hasher param in std, but there was some language restriction that prevented that.

[bstrie]

There are too many details to the std-as-facade/deprecating-std to propose and discuss them here in this comment section. Again, I implore those interested to write up an RFC if you wish to pursue it.

[alexcrichton]

The libs team discussed this during triage yesterday, as well as the core team two days ago, as well as the lang team a few weeks ago, and the conclusion is to stabilize.

The extension traits for primitive types in libcore will be left unstable for now with stable methods (e.g. the SliceConcatExt trick and otherwise all method stabilities will match that of the standard library.

[alexcrichton]

Closing as this was stabilized in #30187