Tracking issue for RFC 1566: Procedural macros

[aturon]

Current Status

This issue has been closed in favor of more fine-grained tracking issues

~~~Updated Description~~~

• FCP request
• FCP check off
• All macros-1.2 related issue

Next steps:

• Stabilize use_extern_macros
  • waiting on crater
• Stabilize the proc_macro feature

Possible Stabilization Showstoppers

• Hygiene is weird
  • proposed solution
• Hygiene is weirder
  • disallow modules/macro definitions in output
• Supporting module system macros with macro_rules! and also having pre-1.29 compat is hard
  • current possible routes

Original Description

RFC.

This RFC proposes an evolution of Rust's procedural macro system (aka syntax
extensions, aka compiler plugins). This RFC specifies syntax for the definition
of procedural macros, a high-level view of their implementation in the compiler,
and outlines how they interact with the compilation process.

At the highest level, macros are defined by implementing functions marked with
a #[macro] attribute. Macros operate on a list of tokens provided by the
compiler and return a list of tokens that the macro use is replaced by. We
provide low-level facilities for operating on these tokens. Higher level
facilities (e.g., for parsing tokens to an AST) should exist as library crates.

Roadmap: #38356 (comment).

---

Tasks

• Implement #[proc_macro_attribute] (PR Implement #[proc_macro_attribute] #38842).
  • Fix feature(proc_macro) breaks attributes on custom derive #39347 (PR macros: fix inert attributes from proc_macro_derives with #![feature(proc_macro)] #39572).
  • Fix macros: don't cfg-process items before applying attribute proc macros #39336 (PR Apply attr proc macros before cfg processing #44528).
• Implement #[proc_macro] (PR Implement function-like procedural macros ( #[proc_macro]) #40129).
• Identify and collect uses of proc_macro_derives in the InvocationCollector (PR [WIP] Expand #[derive(..)]s in the InvocationCollector #39391).
• Support macro-expanded proc_macro_derive imports.
  • For example:

#[derive(Trait, OtherTrait)] struct S; // Both these derives should resolve
macro_rules! m { () => {
    #[macro_use(Trait)] extern crate derives;
    use derives::OtherTrait; // this kind of import is gated behind `#![feature(proc_macro)]`
} }
m!();

• Expand items before expanding applied proc_macro_derives (PR Expand items before their derives #48465).
• Implement warnings for unused #[macro_use] imports (PR Improve unused extern crate and unused #[macro_use] warnings #39060).
• Refactor the parser to consume token trees (PR Refactor the parser to consume token trees #39118).
• Clean up TokenStream in preparation for further refactoring (PR Refactor TokenStream #39173).
• Remove TokenTree::Sequence (PR Simplify TokenTree and fix macro_rules! bugs #39419).
• Use TokenStreams instead of Vec<TokenTree> in tokenstream::TokenTree's Delimited variant (PR syntax: integrate TokenStream #40202).
• Use Paths and TokenStreams in ast::Attributes (PR TokenStream-based attributes, paths in attribute and derive macro invocations #40346).
  • Support nontrivial paths in attribute/derive macros (e.g. #[foo::bar], #[derive(foo::Bar)]).
• Include hygiene information with all tokens, not just identifiers (PR macros: improve Span's expansion information #40597).
• Implement a minimal API for proc_macro::TokenStream as outlined in the RFC (PR proc_macro: implement TokenTree, TokenKind, hygienic quote!, and other API #40939).
  • Include source TokenStreams for interpolated AST fragments in Token::Interpolated tokens.
  • Include a TokenStream quoter proc_macro::quote! behind the proc_macro feature gate.
• Provide a way for proc_macro authors to create expansions that use items in a predetermined crate foo without requiring the macro user to include extern crate foo; at the crate root (PR proc_macro: implement TokenTree, TokenKind, hygienic quote!, and other API #40939).
  • Improve ergonomics.
• Include source TokenStreams for items in the AST.
• Stability check (proc-)macros (issue able to invoke unstable macro 2.0 #34079).
• Allow proc macro to initialize a private field with a def_site value (issue Allow proc macro to initialize a private field with a def_site value #47311). (PR macros: improve struct constructor field hygiene, fix span bug #48082)
• Inconsistency between accessing field of braced struct vs tuple struct in proc macro (issue Inconsistency between accessing field of braced struct vs tuple struct in proc macro #47312). (PR Improve tuple struct field access hygiene #48083)
• Make std available to proc macro root in phase 1 (issue Make std available to proc macro root in phase 1 #47314).
• Improve error from invalid syntax inside proc_macro::quote! (issue Improve error from invalid syntax inside proc_macro::quote! #47315).
• Inconsistency between Display and IntoIterator for a TokenStream containing a module (issue Inconsistency between Display and IntoIterator for a TokenStream containing a module #47627).
• #[cfg_attr] makes .to_string() and TokenStream disagree (issue Macros 2.0: #[cfg_attr] makes .to_string() and TokenStream disagree #48644).
• Wishlist for libproc_macro (checklist in Wishlist for libproc_macro #47786).

[aturon]

cc @nrc @jseyfried

[abonander]

I'd love for #[proc_macro_attribute] to be implemented soon. I already have a prototype and test usage that I banged out before realizing there's no compiler support yet 😒 :

Prototype: https://github.com/abonander/anterofit/blob/proc_macro/macros/src/lib.rs
Example/Test: https://github.com/abonander/anterofit/blob/proc_macro/examples/post_service_proc_macro.rs

[jseyfried]

Tasks

(dtolnay edit: moved the checklist up to the OP)

cc @nrc @petrochenkov @durka @Ralith

[abonander]

@jseyfried I ran into an issue where if a legacy macro and an attribute with the same name are imported into the same scope, trying to use the attribute throws an error that macros cannot be used as attributes. Can we make this work so that both can be in the same scope and can be used as intended?

[jseyfried]

@abonander All macros (bang, attribute, and derive) share the same namespace, so we can't use two different macros with the same name in the same scope. However, we could improve that error message -- could you open an issue?

[SimonSapin]

Sorry I’m late to the party. I’m happy with the direction to expose tokens rather than an AST, but I have some concerns about the specific TokenStream API proposed in the RFC:

pub enum TokenKind {
    Sequence(Delimiter, TokenStream),

    // The content of the comment can be found from the span.
    Comment(CommentKind),

    // `text` is the string contents, not including delimiters. It would be nice
    // to avoid an allocation in the common case that the string is in the
    // source code. We might be able to use `&'codemap str` or something.
    // `raw_markers` is for the count of `#`s if the string is a raw string. If
    // the string is not raw, then it will be `None`.
    String { text: Symbol, raw_markers: Option<usize>, kind: StringKind },

    // char literal, span includes the `'` delimiters.
    Char(char),

    // These tokens are treated specially since they are used for macro
    // expansion or delimiting items.
    Exclamation,  // `!`
    Dollar,       // `$`
    // Not actually sure if we need this or if semicolons can be treated like
    // other punctuation.
    Semicolon,    // `;`
    Eof,          // Do we need this?

    // Word is defined by Unicode Standard Annex 31 -
    // [Unicode Identifier and Pattern Syntax](http://unicode.org/reports/tr31/)
    Word(Symbol),
    Punctuation(char),
}

pub enum StringKind {
    Regular,
    Byte,
}

It’s not clear if this API was intended as a complete plan that was accepted when the RFC was merged, or just an example to be worked out later.

Generally, this seems far from the "normal" Rust syntax accepted by the compiler outside of macros. While some macros will want to parse some ad-hoc domain-specific language, others will want to parse "actual Rust" syntax and make sense of it.

1. (Minor) I don’t think Eof is necessary. An Iterator will presumably be used to, well, iterate over a TokenStream and Iterator::next already returns None to signal the end of iteration.

2. (Minor) I don’t think Exclamation, Dollar, or Semicolon are necessary. Matching on Punctuation('!') for example is not more difficult.

3. (Minor) As others have mentioned in the RFC PR, we might want to omit comments that are not doc-comments. (Any use case that wants to preserve comment likely wants to preserve whitespace too.)

4. As far as I can tell, what to do with multi-character operators (that probably should be a single token each) is still an open question. A possible solution is discussed in PR comments, but it looks like that didn’t make it into RFC text.

5. Number literals are missing. Are macros supposed to parse [Punct('1'), Punct('_'), Punct('2'), Punct('3'), Punct('4'), Punct('.'), Punct('5'), Punct('e'), Punct('6')] by themselves to evaluate a literal? They can’t even use str::parse::<f32> to do that, since the syntax it accepts is not the same as Rust literal syntax (which can have _ in the middle, for example).

   I imagine that there’s a stability concern here. Can we introduce new numeric types like u128 / i128 (and possibly in the future f128, u256, …) and their literals, without breaking changes to the tokens API? One way to make this possible might be:

   struct IntegerLiteral { negative: bool, decimal_digits: String, type_suffix: Option<String> }
   impl TryInto<u32> IntegerLiteral { type Err = OutOfRange; /* … */ }
   // Other impls for integer types supported in this compiler version
   
   // Something similarly for floats

   Or maybe something else. But I don’t think "pretend numbers don’t exist" is a good way to do it.

6. // Word is defined by Unicode Standard Annex 31 -

   This definition needs to be more precise than that. UAX 31 specifies a couple different variations of identifier syntax, and none of them is called "word". But choosing which exact variation we want is why non-ASCII identifiers are feature-gated at the moment.

   Instead, I think this should be defined as "whatever the current compiler accepts as an identifier or keyword" (which can change per Tracking issue for non-ASCII identifiers (feature "non_ascii_idents") #28979). Maybe with a pub fn is_identifier(&str) -> bool public API in libmacro.

7. Unicode strings and byte string literals share a single token variant, which I think is wrong as the memory representations of their values have different types (str vs [u8]). It’s also not clear if the text: Symbol component is intended to be a literal slice of the source code or the value after resolving backslash escapes. I think it should definitely be the latter. (For comparison, Char(char) has to be the latter since \u{A0} takes more than one char to represent literally.)

[porky11]

another way to write high level macros would be using lisp like macros, but this would need a s-expression representation for the whole rust ast.

[petrochenkov]

With #![feature(proc_macro)] stabilized, what's left of this issue?

Ideally, fresh issues need be filed for all individual remaining issues and not stabilized features, and then this issue can be closed (in the same way like it was done for #44660).

[mark-i-m]

Oh, I thought Spans were only for error reporting, as early blog posts mentioned a Hygiene (or similarly named) struct that played this role. I had assumed that these had vanished, and was apparently wrong. Thanks! :)

IIUC, Spans are the primary way of tracking hygiene context.

[alexreg]

@mark-i-m Kind of. They include both source code location information (for user-facing messages/diagnostics) and syntax context (i.e. hygiene information).

[sgrif]

Given the amount of discussion/traffic this issue gets, does it make sense to move proc_macro_diagnostic to its own tracking issue? I'd also like to figure out what the blockers are for the stabilization of that feature, and see if we can push it through. Diesel uses it, and it's been great so far. The lack of this feature on stable is leading to the community creating funky workarounds, like syn's newest version using compile_error! in its place.

[SergioBenitez]

@sgrif I've opened up such an issue: #54140.

[xd009642]

So I'm interesting in helping to stabilise the methods in Span and the LineColumn struct. I'll look at the open issues in the first comment, but if anyone wants to point a newbie to the compiler in a particular direction I'd appreciate it 👍

[jjpe]

The proc_macro_gen feature gate pointed me here, but in the check list at the top I see nothing that obviously refers to a (proc_)macro that generates other macro definitions. Has this been accounted for (other than in rustc's feature gate)?

[alexcrichton]

@jjpe it's probably best at this time that we spin off dedicated tracking issues for those feature gates, this issue was largely dedicated to the initial wave of stabilizations

[jjpe]

@alexcrichton I'm perfectly fine with that, it's more that in the process of looking at the proc_macro_gen feature, it referred me here only to find as good as no mention of it. That's a bit odd to me, so I decided to mention it, at least.

@xieyuheng What would CodeString/Code even look like i.e. what would its semantics be?
Keep in mind that TokenStream is nothing more than the source code verbatim except as a series of token values rather than bothersome text.

[SimonSapin]

The extended API for TokenStream (with TokenTree) is stable in 1.29. Importing function-like proc macros will be stable in 1.30.

[CodeSandwich]

It seems that since rustc 1.30.0-nightly (63d51e89a 2018-09-28) it is no longer possible to traverse code inside module in separate file. If you process mod module;, you get TokenStream containing mod module;, WYSIWYG.

I understand, that this is necessary in order to preserver spans, hygiene and consistency with processed code. Is there or will be any way to work with content of modules in separate files? Its lack may be confusing for end users when trivial refactoring of module organization causes change in macro behavior.

[alexcrichton]

Ok this issue is massive and has gotten to the point that I don't think it's too useful to keep open any more and track APIs. To that end I've opened #54728 which splits this issue into a number of more fine-grained tracking issues:

• Tracking issue for the quote! macro in proc_macro #54722 - the quote! macro
• Tracking issue for proc_macro::Ident::new_raw #54723 - Ident::new_raw
• Tracking issue for Span::def_site() #54724 - Span::def_site
• Tracking Issue: Procedural Macro Diagnostics (RFC 1566) #54140 - proc_macro and diagnostics
• Tracking issue for proc_macro::Span inspection APIs #54725 - inspecting a proc_macro::Span
• Tracking issue for custom inner attributes #54726 - allowing custom inner attributes
• Tracking issue for procedural macros and "hygiene 2.0" #54727 - a sort of catch-all for "hygiene 2.0" reports, may want to be sharded itself.

At this point I'm going to close this, but if I've forgotten to split off any other tracking issues please let me know! I can certainly open up some more follow-ups

[XX]

@alexcrichton What about of sub-attributes for attribute-like macro?
#38356 (comment)
Is there a issue for this?

[petrochenkov]

@XX
Such sub-attributes don't necessarily have to be a language feature?
In case of derive registration of custom attributes is needed because derive macros cannot remove attributes from their input (the input is immutable).
Attribute macros can remove #[other_attribute]s from its input, so they will never get to name resolution and never report the "unresolved attribute" error.

(Beside that classic legacy unstable custom attributes mentioned in #38356 (comment) are now compatible with proc macros.)

[XX]

@petrochenkov Yes, thanks for the clarification.