RFC: Packages as (optional) namespaces

[Manishearth]

🖥️ Rendered 🖥️

crates.io FCP
cargo FCP
lang sign off
compiler sign off

Rust Issue: rust-lang/rust#122349

• crates.io tracking issue: Tracking Issue for Packages as (optional) namespaces crates.io#8292
• cargo tracking issue: Tracking Issue for Packages as (optional) namespaces cargo#13576

---

This was previously discussed as a pre-RFC here, and later iterated upon in this repo.

There is a prototype of this RFC available on https://cratespaces.integer32.com/ with instructions here

This RFC brings forth a long-discussed and desired feature: some kind of namespacing for crates.io, achieved via treating "toplevel" packages themselves as namespaces, with ownership being inherited.

Please read before participating

I'm going to include some of the same disclaimers I included in the pre-rfc:

The general topic of namespacing is one that has been discussed many times. In my understanding, a lot of the tension here comes from people having different problems they wish to solve, and these discussions not always acknowledging that, leading to people feeling unheard.

This proposal comes out of many years of one on one discussion, reading threads, and incubation. I do think this is a viable and good path forward, but for this to work out we need to be our best selves here.

I would like to request everyone to keep this discussion constructive and respectful. If a discussion is getting super involved, it may be worth opening a discussion issue on https://github.com/Manishearth/namespacing-rfc/ and linking to it instead.

This RFC is attempting to solve the problem of making crate ownership clear in terms of other crates; it is not attempting to solve the general problem around the squatting of root crate names. As such, if a crate can be published today under a particular name by anyone, it would also also be publishable under the same name by anyone in the world of this RFC proposal. I would prefer if we did not spend much time discussing what I see as the radically different problem of squatting in general.

Thank you all in advance for what I hope will be a constructive and fruitful RFC!

Update: Discussions about separator choice are being conducted here.

[Manishearth]

I am rather partial to this approach; it has a bunch of benefits:

• It has a clear mapping to crates.io/docs.rs URLs
• There's no ambiguity in feature specs
• We could use the same syntax directly in source (@ <ident> / triggers parsing as a namespaced path) though this might be too large a syntax change

[pitaj]

I do think it would be ideal for the same syntax to be used both in Cargo.toml and in source code. I'd like to suggest the usage of ::: but I think it has the same issue that led to ... becoming ..=.

Can you expand on why @foo::bar might be too large a syntax change?

[Manishearth]

Because it changes what counts as an identifier in Rust: it's a change at the token level, which in turn affects proc macros and such.

[Manishearth]

(it doesn't have to be at the token level tbh, but that does still complicate how paths work)

[Manishearth]

Would be nice to have @rust-lang/lang input on this as a syntax change

(if this RFC starts going down this route I'll of course add them as an approving team)

[timleg002]

what does @ mean in npm's context?

[fangmarks]

what does @ mean in npm's context?

The way npm does package names in a package.json is like @scope/package@version.
Where scope is a existing organization on npm and @version a published version.

Though they also allow you to use Github repos as shown in the image

[nbittich]

please don't introduce new keywords. I like the idea of namespace, but please take a simple path, or don't implement it at all until you find one. rust is already complex enough, I'm affraid we have to write things like this in the future:

from crate<'a,T> import T::foo::<'a> where T: Pin<Box<dyn Crate>>> + 'a Namespace + Copy + Debug as @foo

[heaths]

Using a character that is currently not possible within a crate name does avoid issues like a bunch of crates already being published with a name that want as a root. For example, if your org uses something like foo_bar and foo_baz but someone else already published foo, you can't use those names. But you could for something like @foo/bar.

That said, I see @ehuss's comment below that cargo has already closed on support for ::. Does that effectively mean this RFC is decided?

[epage]

Its been approved by 2 of the 3 related teams and is just waiting on enough votes from the last team.

[steffahn]

🤔... semi-serious idea: perhaps backslash could be used in code: foo\bar. That doesn't even tokenize at the moment, so it's fully backwards compatible. Too bad neither URLs not toml supports those.

[jklamer]

Thank you for preparing this! I'd like to second the support for ::: as both cargo and path syntax. And maybe even go as far to push back on the idea that it is confusing (or more confusing than other options). "This same thing looks different in cargo.toml than in use statements" is a larger source of confusion for me and my lizard brain than most other things. I may be missing some expertise for the true confusion of :::

[jhpratt]

I don't have time to look through the RFC this moment, but I'm hoping this will be forward compatible with intra-package libraries? I described my thoughts a bit in this issue.

[Manishearth]

@jhpratt i think so

[yigal100]

Cargo and rustc have different roles (bounded contexts) and I find it incredible we keep trying to shoehorn the same design into both.

Cargo must use full paths to namespaced crates for obvious reasons. However, this isn't the right design for use within rust code:

• A single project only has a tiny subset of the entire crates eco-system. The chances of a name conflict are slim to none.
• It's highly unlikely a user would need two competing implementations from two orgs within the same project.
• Adding some sort of separator mapping to code adds redundant complexity (magic) to an already complicated language and would likely introduce edge cases - what if a crate name itself contains an underscore?
• Duplicating the full crate namespace for each use statement is redundant verbiage. If the user wants to move a crate to another namespace, they'll need to touch many places in the code instead of just a single line in the cargo.toml configuration.

To summarise, cargo should be the sole manager of this concern and this should not bleed into code. I should be able to add a dependency of "foo/bar/baz" to cargo, and expect to use "baz" as the crate name in rust code. In the unlikely event There is a name conflict, I should be able to rename the crate in cargo config - that should be an already existing feature of cargo.

For example, a company might choose to have a naming scheme like:
"Company/product/libfoo"
"Company/product/bar"
And say bar depends on libfoo.

What happens if the company decides to rebrand "product"? Seems to me this is redundant busy-work to go and update all references to libfoo when this could have been a simple config file change.

Final edit:
To preempt possible "OSS" objections to this "comercial" use case as invalid:

• Enterprise adoption is already happening and is crucial to Rust's success
• this is as much about the implementation of cargo as it is about crates.io and cargo already supports 3rd party registries in stable.

[Manishearth]

@yigal100 I think you have misunderstood the problem people are attempting to resolve by making the syntax the same in Cargo and rustc.

I'm also going to somewhat preemptively ask you to tone it down a bit. Using language like "I find it incredible that..." is feigning surprise and is in general not a constructive way to engage. Consider that you may have misunderstood, or that the people you are discussing with hold different values. If you find this pushback to be harsh, bear in mind that this topic has an extremely hard time being discussed, and even minor unconstructive comments can and will snowball. Either people are on their best behavior or this discussion goes nowhere for the millionth time, and I truly wish for the former.

I am super sympathetic to the goal of separating rustc and Cargo here (indeed, the current RFC draft does not propose any rustc changes, just talks about them in the alternatives: I share this goal to some degree!), however I urge folks to be constructive when talking about it.

To address some points you brought up specifically

What happens if the company decides to rebrand "product"? Seems to me this is redundant busy-work to go and update all references to libfoo when this could have been a simple config file change.

This already exists. You can set the name Cargo passes down to rustc.

• A single project only has a tiny subset of the entire crates eco-system. The chances of a name conflict are slim to none.
  It's highly unlikely a user would need two competing implementations from two orgs within the same project.

The argument is more about the different syntaxes being confusing (people already have trouble with - and _). At no point is it said that conflicts are the problem (conflicts are resolved with the fact that you can set the name rustc sees in Cargo.toml), you are addressing a straw man here. Conflicts are talked of as a problem only inasmuch as it enables dash typosquatting, where malicious people will take foo-bar and hope that people looking for foo/bar will sometimes make a mistake.

Duplicating the full crate namespace for each use statement is redundant verbiage. If the user wants to move a crate to another namespace, they'll need to touch many places in the code instead of just a single line in the cargo.toml configuration.

This argument already applies to renaming crates in general; and is not a new problem. Furthermore, this RFC doesn't quite propose any form of "moving" crates between namespaces in the first place. Finally, as already mentioned, there are ways to handle crate renames in just Cargo.toml.

[yigal100]

@Manishearth:

What happens if the company decides to rebrand "product"? Seems to me this is redundant busy-work to go and update all references to libfoo when this could have been a simple config file change.

This already exists. You can set the name Cargo passes down to rustc.

We're in agreement here as I've mentioned the same point myself. My comment is really about the default behaviour. Defaults matter a lot for ergonomic reasons and the best default here is to map to the leaf crate name and not pass along the full path under some transformation.

As you noted yourself, the current scheme is already confusing people with '-' vs. '_' and imho we shouldn't add to that an additional transformation.

[Manishearth]

I consider removing the root crate name to also be a transformation that can potentially be confusing.

A thing that I might not have mentioned in the RFC (but I'll edit it in now, it'l llargely have the text I'm putting below) is that there are, broadly speaking, two ways this feature may be used.

One of the ways is for one organization (say, unicode) to release a lot of crates under the namespace as asserting organizational ownership. Think unicode/segmentation, unicode/line-breaking, unicode/script, etc. In such a case, using just the leaf crate name makes a lot of sense.

However, there's another case: where a project wishes to use namespaces to talk about a related set of crates. For example, a very common use case I see is things like serde/derive and icu/provider, where "derive" and "provider" on their own are rather generic and would be prone to clashes (what derive? provider of what?)

It's still a valid design to just pick leaf nodes and ask people to rename, however. I'll mention that in my edit, I feel like I had talked about this at some stage of the discussions but it may not have made it into the RFC.

[Manishearth]

@yigal100 I've added a section. I do somewhat like the leaf approach and it was closer to my original idea but I found in feedback that people who wanted to use it for the "project" use case did not quite enjoy it. But it's now an explicit section where this topic can be discussed further.

[rpjohnst]

A minor note on the question of the separator:

Using :: there not only aligns well with existing Rust syntax generally, but I think also aligns well with with 2018 edition use path changes. While in the 2015 edition the crate root is the top-level :: namespace, the 2018 edition pushes the current crate down a level into crate:: to make room for other crates to live next to it. Using :: as the separator would take this one step further, and allow that new top level namespace to contain package groups as well as individual crates.

Long term, migrating to a separator that is the same in Rust source code might even reduce some of the confusion around hyphens- crates that use them for namespacing would no longer need any crate name translation from Cargo.

[Absolucy]

Agreeing with @rpjohnst, my opinion on how to handle use without syntax changes would be something along the lines of this:

• Check for a namespace first and prioritize it, ie the namespaced serde::derive takes priority over the root namespace serde::derive in the serde crate
• Reverse this logic and prioritize root namespace first if the use is prefixed by a ::

So,

use serde::derive;   // resolves to namespace "serde", crate "derive"
use ::serde::derive; // resolves to root namespace, crate "serde", module "derive"

Ultimately I see no better way to resolve this without parser/syntax changes, which could require another Rust edition if I'm not mistaken?

[pksunkara]

use serde::derive;   // resolves to namespace "serde", crate "derive"
use ::serde::derive; // resolves to root namespace, crate "serde", module "derive"

There are scenarios where people would want to refer to module "derive" of crate "serde" without the leading ::. Especially in marcos. If you look at clap, we do this exact thing in order to let clap be dependency of another library.

[Manishearth]

Would folks proposing :: be willing to come up with a proposal for :: that includes all the different aspects that need to be addressed? I'd also love to handle it with :: but I'm worried about confusion and ambiguity; and in the past I recall us having a hard time coming up with something that worked with ::.

[Manishearth]

If separators are going to be a major discussion I'm also happy to direct people to Manishearth/namespacing-rfc#1 and/or Manishearth/namespacing-rfc#2 and request in-depth discussion occur there.

[joshtriplett]

@Manishearth I posted a sketch of a proposal for :: at Manishearth/namespacing-rfc#1 (comment) . Happy to continue there.

[Manishearth]

Yeah, let's consolidate discussion on separator choice there. I've updated the RFC with links to the issue, further comments about separator choice may be hidden and redirected there instead (until we reach consensus there).

[Nemo157]

Another option for disambiguation here could be namespaced features. If namespaced crates did not automatically create an unnamespaced feature like current optional dependencies do, then you would have to use dep:foo/bar to refer to it and foo/bar would unambiguously refer to the feature bar on crate foo.

[tarcieri]

@jtgeibel there was previous discussion regarding Windows paths here: https://github.com/rust-lang/rfcs/pull/3243/files#r983881223

[epage]

@jtgeibel the index / filename issues are marked for resolving during stabilization by being in the "Unresolved questions" section. Is that sufficient?

As for dep:foo vs dep::foo, that seems like more of a concern for the cargo team than crates.io for deciding whether that is a blocking concern or not. We ran three separate FCPs for this RFC so we could make sure that all respective teams signed off.

[jtgeibel]

in the "Unresolved questions" section. Is that sufficient?

Somehow I missed that. Yeah, that is fine with me.

For dep:foo I wasn't intending to raise that as a blocking concern (at least from my perspective). Looks good to me then, so checking my box now.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[Manishearth]

Somehow I missed that. Yeah, that is fine with me.

Thanks! Yeah I feel like the nitty gritties of the exact implementation strategy especially for Windows doesn't really need to be RFCd, y'all can pick something when the time comes. I was trying not to overconstrain the teams with this RFC.

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

This will be merged soon.

[simonsan]

🚀 Wow! Thanks for all your work! <3

[sunmy2019]

Did you forget to change this?

[sunmy2019]

fixed via #3586

[mattheww]

I noticed that #3243 (comment)

wanted a note to be added on the tracking issue:

There was some discussion about the exact name conflict rules, and we'd like to see a note in the eventual tracking issue that it's important that those get decided explicitly and written down

which hasn't appeared.

(I would mention it on that issue, but it's locked.)

[Manishearth]

Good catch, done!

[cbiffle]

I'm not sure if comments like this are helpful, but: I've read through the RFC as merged, and the proposal would almost certainly cover the needs of both operating systems I work on. On Hubris in particular, this is a missing piece we've been trying to figure out before publishing any of the operating system to crates.io (currently it's all git deps or path deps to keep interlopers from sneaking crates in).

So, thank you for threading the needle on this, this solves a real problem in a thoughtful way, and I'm excited to see where this goes.