Add more complex functionlike!(...) procedural macro example(s)

[JeanMertz]

I'm learning how to use syn, and after starting with some simple examples, I'm now running into issues that relate to blocks, specifically, how to:

• store the TokenStream of a block for later
• skip over a block in the stream and continue with the next token after the closing brace
• how to parse the tokens inside a block

I've got this simple example here:

https://gist.github.com/JeanMertz/ac3b5a1fcc351c8497aa6b3b68d94c87

The error reported there is I believe because Block::parse_within is the wrong thing to do, as it actually expects valid statements, while I'm using custom syntax.

But there are also other examples such as is_empty() that show how to loop over the contents of a block:

let content;
let _ = braced!(content in input);
while !content.is_empty() {
    let _ = content.parse()?;
}

But if I do that with the above example, I get:

error[E0277]: the trait bound `(): syn::parse::Parse` is not satisfied
  --> src/lib.rs:51:33
   |
51 |                 let _ = content.parse()?;
   |                                 ^^^^^ the trait `syn::parse::Parse` is not implemented for `()`

I haven't been able to find any good resources on how to handle structures like in my example. The lazy-static example is straight-forward, as it parses tokens in sequence from start to finish, and all returned tokens implement Parse, whereas (apparently) in my example that's not the case, but I'm unable to figure out how to detect that that's not the case, and/or how to deal with such situations (ie. being able to advance the cursor in such a case).

[dtolnay]

Thanks, this is useful feedback.

Regarding Block::parse_within, yes the documentation for that method says that it parses "the body of a block as zero or more statements, possibly including one trailing expression." In your case SkipThis { AndAlsoThis => That } is not syntactically a Rust statement / expression so Block::parse_within will not parse it.

• how to store the TokenStream of a block for later

You would parse the input into a TokenStream the same as you would parse it into any other syntax tree node:

let tts: TokenStream = content.parse()?;

Examples of this are syn::Attribute and syn::Macro which both store part of their representation as a TokenStream.

• how to skip over a block in the stream and continue with the next token after the closing brace

The API requires you to parse every token in the input, but you can effectively ignore some part of the input by parsing it to TokenStream and dropping the TokenStream.

let name: Ident = input.parse()?;

let content;
braced!(content in input);
let _: TokenStream = content.parse()?; // ignore content

// keep going
let name2: Ident = input.parse()?;

• how to parse the tokens inside a block

I think you mostly got this. But when you wrote this line:

let _ = content.parse()?;

there is no way to infer what type this is supposed to parse. Is it parsing an expression? Is it parsing an attribute? Is it parsing an enum...? It is ambiguous, which is why we refused to compile it.

[JeanMertz]

Thanks! This makes things much more clear.

The key points that helped me here:

• You have to parse each token, just throw it away if you don't care
• The error returned by the compiler can be a bit cryptic if it is dealing with an ambiguous parsing request
• the object returned by braced! can be parsed as one big token, returning a TokenStream

When implementing your suggestion I first ran into an error that TokenStream did not implement Parse. Then I realised I read somewhere that dyn uses proc_macro2, while the proc macro function itself takes and returns a proc_macro::TokenStream, so I had to explicitly set the value type to proc_macro2::TokenStream in this case.

One final questions:

• In some examples I see storing braced!(content in input) in a struct. What's the use-case to store what's returned by this macro? In other words, what's the Brace token useful for? The description shows {...}, so I assume it's not just the single { token at the start of a block, but it represents an entire block as a token, without its contents?

[dtolnay]

If the caller may want to trigger errors pointing to the brace, they need to be able to access the span of the brace token.

return Err(Error::new(m.brace_token.span, "this error points to the brace token"));

error: this error points to the brace token
  --> src/main.rs:10:14
   |
10 |     SkipThis { AndAlsoThis => That }
   |              ^^^^^^^^^^^^^^^^^^^^^^^

Also we need the brace token for transforming the syntax tree back to a tokenstream when interpolated within a quote! invocation.

syn/src/item.rs

Lines 2101 to 2112 in 3db288c

	impl ToTokens for ItemEnum {
	fn to_tokens(&self, tokens: &mut TokenStream) {
	tokens.append_all(self.attrs.outer());
	self.vis.to_tokens(tokens);
	self.enum_token.to_tokens(tokens);
	self.ident.to_tokens(tokens);
	self.generics.to_tokens(tokens);
	self.generics.where_clause.to_tokens(tokens);
	self.brace_token.surround(tokens, |tokens| {
	self.variants.to_tokens(tokens);
	});
	}

[JeanMertz]

Ah, got it. Thanks 👍 I'll close this issue, but maybe it's worth considering adding some of the pointers you gave somewhere in the documentation, as it instantly helped me move from struggling to find the correct solution to grasping the (simple parts of the) API and how it works.

[dtolnay]

Will do!

[JeanMertz]

I don't really know if there's a better place to ask ad-hoc questions (somewhere on Discord perhaps?), so I'll just ask it here, and perhaps it can feed into some extended documentation:

I have a TokenStream and want to wrap it into a module as mod my_mod { TokenSteamHere }.

I've gotten to:

token::Brace::default().surround(&mut stream, |stream| {
    stream;
});

which I believe wraps the braces around the TokenStream, but I still need to prepend it with mod my_mod, and have yet to find the right solution.

I've looked at something like:

let wrapped = quote!("mod");
wrapped.extend(stream);

But that doesn't work as expected.

Any hints/pointers would be appreciated. Don't feel obliged to answer though, if you get around to it great, if not, I'll continue to search around in the API docs to figure it out.

[dtolnay]

Quote is the easiest way. https://github.com/dtolnay/quote

let wrapped = quote! {
    mod my_mod {
        #stream
    }
};

[JeanMertz]

Ah of course, I can actually put a TokenStream inside quote!, that makes sense. Thanks, this was the missing link 👍

[JeanMertz]

Another "learn as you go" question that might also help others:

I have a TokenStream, and I like to convert it to a ParseStream so that I can use it in my tests.

Basically, I'm trying to do this:

let tokens: TokenStream = quote!{ HelloWorld };
// something to go from TokenStream to ParseStream
let output = MyObject::parse(&stream).unwrap();

assert_eq!(output, MyObject { ... })

I tried the following, but quickly realised I was fighting with internals, and eventually ended up at a private API:

// duplicating implementation from https://docs.rs/syn/0.15/src/syn/parse.rs.html#1075-1080
let buf = syn::buffer::TokenBuffer::new2(tokens);
let scope = Span::call_site();
let cursor = buf.begin();
let unexpected = Rc::new(Cell::new(None));

// ending up with a private API
let stream = syn::parse::ParseBuffer {
    scope: scope,
    cell: Cell::new(unsafe { mem::transmute::<Cursor, Cursor<'static>>(cursor) }),
    marker: PhantomData,
    unexpected: unexpected,
};

// field `scope` of struct `syn::parse::ParseBuffer` is private
// ...

Either I'm missing an obvious API that helps me achieve this, or this isn't possible (yet), I suspect it's the former.

[dtolnay]

I believe the syn::parse doc covers this. If MyObject::parse is a Parse impl:

let tokens = quote! { HelloWorld };
let output: MyObject = syn::parse2(tokens).unwrap();

Otherwise, effectively the function that you linked to:

let tokens = quote! { HelloWorld };
let output = MyObject::parse.parse2(tokens).unwrap();