[macro_metavar_expr_concat] Add support for literals

[c410-f3r]

Adds support for things like ${concat($variable, 123)} or ${concat("hello", "_world")} .

cc #124225

[rustbot]

r? @petrochenkov

rustbot has assigned @petrochenkov.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[petrochenkov]

concat_idents! doesn't currently support literal, but the motivation here is pretty clear, you want to combine ident and 0 into ident0, but 0 is not an identifier lexically.

However, restricting literals to simple string literals "abc" (no raws, no suffixes) would be enough for that.
All possible identifier parts should fit into that.

[petrochenkov]

Also, is there any validation for the concatenation result, like in https://doc.rust-lang.org/stable/proc_macro/struct.Ident.html#method.new ?

When concatenating arbitrary strings you can get something that is

• not lexically an identifier
• not NFC normalized (https://doc.rust-lang.org/reference/identifiers.html?highlight=nfc#normalization)

Both lexer and proc macro interface prevent creation of such ill-formed identifiers.

[c410-f3r]

Also, is there any validation for the concatenation result, like in https://doc.rust-lang.org/stable/proc_macro/struct.Ident.html#method.new ?

When concatenating arbitrary strings you can get something that is

* not lexically an identifier

* not NFC normalized (https://doc.rust-lang.org/reference/identifiers.html?highlight=nfc#normalization)

Both lexer and proc macro interface prevent creation of such ill-formed identifiers.

Yeah, this is bad :|

The current code allows the creation of invalid identifiers so I need investigate how to perform such verification.

[c410-f3r]

Thanks for the review

@rustbot review

[c410-f3r]

Nice!

@rustbot review

[petrochenkov]

Where is this error coming from?
I don't see any additional checking in transcribe.rs that would result in this.

Also, will concat!("\x41") produce a or an invalid identifier \x41?

[c410-f3r]

Where is this error coming from?

rust/compiler/rustc_parse/src/lexer/unescape_error_reporting.rs

Lines 36 to 41 in 8a8ad34

	EscapeError::LoneSurrogateUnicodeEscape => {
	dcx.emit_err(UnescapeError::InvalidUnicodeEscape { span: err_span, surrogate: true })
	}
	EscapeError::OutOfRangeUnicodeEscape => {
	dcx.emit_err(UnescapeError::InvalidUnicodeEscape { span: err_span, surrogate: false })
	}

Looks like the parse eagerly halts when an invalid unicode literal is found. Changed to "\u{00BD}" (½).

Also, will concat!("\x41") produce a or an invalid identifier \x41?

At the current time it is not possible to accept literal parameters but I expect to see a ${concat(..)} is not generating a valid identifier error like in https://github.com/rust-lang/rust/pull/126841/files#diff-c527d06b3a89d2cf1181cf0b3122091572d2992346b42217dc290a18d2e2366cR75

cc #126841 (comment)

[petrochenkov]

Okay, the current behavior looks acceptable to me - either we have a literal that doesn't have \ in it so its escaped and unescaped values are the same, or it has \ and then it's not a valid identifier and we produce an error.

[c410-f3r]

@rustbot review

[petrochenkov]

As usual, literals turned out more complex than it may seem.
@bors r+

[bors]

📌 Commit c990e00 has been approved by petrochenkov

It is now in the queue for this repository.