Fix double handling in `collect_tokens` #129346

nnethercote · 2024-08-21T06:41:42Z

Double handling of AST nodes can occur in collect_tokens. This is when an inner call to collect_tokens produces an AST node, and then an outer call to collect_tokens produces the same AST node. This can happen in a few places, e.g. expression statements where the statement delegates HasTokens and HasAttrs to the expression. It will also happen more after #124141.

This PR fixes some double handling cases that cause problems, including #129166.

r? @petrochenkov

nnethercote · 2024-08-21T06:43:20Z

The last three commits are tricky and hard to review. Even after multiple weeks of working with this code, I still find it hard to understand and reason about, and I wish it could be made simpler. But the good news is that these commits are enough to the get final failing test (tests/ui/proc-macros/macro-rules-derive-cfg.rs) working in #124141.

nnethercote · 2024-08-21T07:46:09Z

I think this should get a crater run, just because I don't fully trust that the test suite is thoroughly testing cfg_attr stuff.

And why not do a perf run, just to check there's no regressions.

@bors try @rust-timer queue

bors · 2024-08-21T07:47:21Z

⌛ Trying commit 7d65809 with merge 8b22b49...

…ollect_tokens, r=<try> Fix double handling in `collect_tokens` Double handling of AST nodes can occur in `collect_tokens`. This is when an inner call to `collect_tokens` produces an AST node, and then an outer call to `collect_tokens` produces the same AST node. This can happen in a few places, e.g. expression statements where the statement delegates `HasTokens` and `HasAttrs` to the expression. It will also happen more after rust-lang#124141. This PR fixes some double handling cases that cause problems, including rust-lang#129166. r? `@petrochenkov`

bors · 2024-08-21T09:41:52Z

☀️ Try build successful - checks-actions
Build commit: 8b22b49 (8b22b49ea046aed5d405c5bda40972c039aa70b5)

rust-timer · 2024-08-21T11:46:32Z

Finished benchmarking commit (8b22b49): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.2%, 0.7%]	43
Regressions ❌ (secondary)	0.4%	[0.1%, 0.6%]	16
Improvements ✅ (primary)	-0.3%	[-0.3%, -0.3%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.3%	[-0.3%, 0.7%]	44

Max RSS (memory usage)

Results (secondary -2.7%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.7%	[-3.4%, -2.0%]	2
All ❌✅ (primary)	-	-	0

Cycles

Results (secondary -2.3%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.3%	[-2.3%, -2.3%]	1
All ❌✅ (primary)	-	-	0

Binary size

Results (secondary 0.0%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.0%	[0.0%, 0.0%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Bootstrap: 748.876s -> 749.555s (0.09%)
Artifact size: 338.91 MiB -> 338.99 MiB (0.02%)

tests/ui/conditional-compilation/invalid-node-range-issue-129166.rs

petrochenkov · 2024-08-21T15:36:44Z

Which change is a source of the perf regression, 7d65809 probably?
All the others look like optimizations.

petrochenkov · 2024-08-21T15:38:05Z

I'll start crater once #129346 (comment) is addressed and CI is green.
@rustbot author

nnethercote · 2024-08-22T07:04:55Z

I have adding a commit that avoids cloning ret.attrs() except when necessary. This gets back part of the perf regression. The rest is due to Parser::clone being more expensive because of the new FxHashSet in CaptureState. I tried changing it to a GrowableBitSet which seemed like it should have helped but didn't.

Let's see what the new status is:
@bors try @rust-time queue

…ollect_tokens, r=<try> Fix double handling in `collect_tokens` Double handling of AST nodes can occur in `collect_tokens`. This is when an inner call to `collect_tokens` produces an AST node, and then an outer call to `collect_tokens` produces the same AST node. This can happen in a few places, e.g. expression statements where the statement delegates `HasTokens` and `HasAttrs` to the expression. It will also happen more after rust-lang#124141. This PR fixes some double handling cases that cause problems, including rust-lang#129166. r? `@petrochenkov`

bors · 2024-08-22T07:06:07Z

⌛ Trying commit cab3178 with merge a70947b...

bors · 2024-08-22T09:05:06Z

☀️ Try build successful - checks-actions
Build commit: a70947b (a70947b8062a88e6f76266f2540de6ea6c2bfba5)

lqd · 2024-08-22T09:22:07Z

@rust-timer build a70947b

rust-timer · 2024-08-22T10:52:15Z

Finished benchmarking commit (a70947b): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.2%, 0.7%]	25
Regressions ❌ (secondary)	0.4%	[0.2%, 0.6%]	17
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.3%	[0.2%, 0.7%]	25

Max RSS (memory usage)

Results (primary -0.8%, secondary 2.4%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.9%	[0.5%, 1.3%]	2
Regressions ❌ (secondary)	2.4%	[2.1%, 2.7%]	2
Improvements ✅ (primary)	-4.3%	[-4.3%, -4.3%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-0.8%	[-4.3%, 1.3%]	3

Cycles

Results (secondary 2.3%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.3%	[2.3%, 2.3%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

Results (secondary -0.1%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.1%	[-0.1%, -0.1%]	1
All ❌✅ (primary)	-	-	0

Bootstrap: 750.761s -> 751.313s (0.07%)
Artifact size: 338.92 MiB -> 339.03 MiB (0.03%)

bors · 2024-09-08T05:35:27Z

⌛ Testing commit d4bf28c with merge 6d05f12...

bors · 2024-09-08T08:03:50Z

☀️ Test successful - checks-actions
Approved by: petrochenkov
Pushing 6d05f12 to master...

rust-timer · 2024-09-08T09:17:59Z

Finished benchmarking commit (6d05f12): comparison URL.

Overall result: ❌✅ regressions and improvements - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.3%	[0.3%, 0.3%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.4%	[-0.4%, -0.3%]	2
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results (primary -1.9%, secondary 1.9%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.6%	[2.6%, 2.6%]	1
Regressions ❌ (secondary)	3.0%	[2.3%, 4.2%]	7
Improvements ✅ (primary)	-3.4%	[-3.5%, -3.4%]	3
Improvements ✅ (secondary)	-0.7%	[-0.7%, -0.6%]	3
All ❌✅ (primary)	-1.9%	[-3.5%, 2.6%]	4

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 754.658s -> 756.515s (0.25%)
Artifact size: 341.04 MiB -> 341.16 MiB (0.04%)

danielhenrymantilla · 2024-10-18T10:47:43Z

FWIW, this did regress the following test: https://docs.rs/crate/cfg_eval/0.1.2/source/tests/item.rs

https://github.com/danielhenrymantilla/cfg_eval.rs/actions/workflows/future-proof.yml

I don't know why crater did not see it 😔

danielhenrymantilla · 2024-10-18T10:50:33Z

tests/ui/proc-macro/macro-rules-derive-cfg.rs

-        struct Foo {
-            val: [bool; {
-                let a = #[cfg_attr(not(FALSE), rustc_dummy(first))] $expr;
-                0
-            }]
-        }


This case stopped working?

Fixed in #132587.

Noratrieb · 2024-10-18T10:51:51Z

even if crater found your crate, it was only a check run

It caused a test regression in the `cfg_eval.rs` crate. (The bugfix in rust-lang#129346 was in a different commit; this commit was just a code simplification.)

lqd · 2024-11-04T11:46:40Z

even if crater found your crate, it was only a check run

and since beta runs are test runs, the regression is indeed present in the 1.83 unexpected test failures: https://crater-reports.s3.amazonaws.com/beta-1.83-1/beta-2024-10-19/gh/danielhenrymantilla.cfg_eval.rs/log.txt

…placement-ranges, r=petrochenkov Revert "Avoid nested replacement ranges" from rust-lang#129346. It caused a test regression in the `cfg_eval.rs` crate. (The bugfix in rust-lang#129346 was in a different commit; this commit was just a code simplification.) r? `@petrochenkov`

…kingjubilee Rollup of 10 pull requests Successful merges: - rust-lang#129884 (mark some target features as 'forbidden' so they cannot be (un)set with -Ctarget-feature) - rust-lang#132153 (Stabilise `const_char_encode_utf16`.) - rust-lang#132473 ([core/fmt] Replace checked slice indexing by unchecked to support panic-free code) - rust-lang#132571 (add const_eval_select macro to reduce redundancy) - rust-lang#132587 (Revert "Avoid nested replacement ranges" from rust-lang#129346.) - rust-lang#132596 ([rustdoc] Fix `--show-coverage` when JSON output format is used) - rust-lang#132598 (Clippy: Move some attribute lints to be early pass (post expansion)) - rust-lang#132601 (Update books) - rust-lang#132606 (Improve example of `impl Pattern for &[char]`) - rust-lang#132609 (docs: fix grammar in doc comment at unix/process.rs) r? `@ghost` `@rustbot` modify labels: rollup

…placement-ranges, r=petrochenkov Revert "Avoid nested replacement ranges" from rust-lang#129346. It caused a test regression in the `cfg_eval.rs` crate. (The bugfix in rust-lang#129346 was in a different commit; this commit was just a code simplification.) r? `@petrochenkov`

…kingjubilee Rollup of 11 pull requests Successful merges: - rust-lang#131153 (Improve duplicate derive Copy/Clone diagnostics) - rust-lang#132025 (fix suggestion for diagnostic error E0027) - rust-lang#132303 (More tests for non-exhaustive C-like enums in FFI) - rust-lang#132492 (remove support for extern-block const intrinsics) - rust-lang#132587 (Revert "Avoid nested replacement ranges" from rust-lang#129346.) - rust-lang#132596 ([rustdoc] Fix `--show-coverage` when JSON output format is used) - rust-lang#132598 (Clippy: Move some attribute lints to be early pass (post expansion)) - rust-lang#132601 (Update books) - rust-lang#132606 (Improve example of `impl Pattern for &[char]`) - rust-lang#132608 (document `type_implements_trait`) - rust-lang#132609 (docs: fix grammar in doc comment at unix/process.rs) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#132587 - nnethercote:revert-avoid-nested-replacement-ranges, r=petrochenkov Revert "Avoid nested replacement ranges" from rust-lang#129346. It caused a test regression in the `cfg_eval.rs` crate. (The bugfix in rust-lang#129346 was in a different commit; this commit was just a code simplification.) r? `@petrochenkov`

It caused a test regression in the `cfg_eval.rs` crate. (The bugfix in rust-lang#129346 was in a different commit; this commit was just a code simplification.) (cherry picked from commit 981dc02)

[beta] backports - rustdoc: skip stability inheritance for some item kinds rust-lang#132481 - Avoid use imports in thread_local_inner! in static rust-lang#132101 - Also treat `impl` definition parent as transparent regarding modules rust-lang#132453 - Revert "Avoid nested replacement ranges" from rust-lang#129346. rust-lang#132587 r? cuviper

rustbot assigned petrochenkov Aug 21, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Aug 21, 2024

This comment has been minimized.

Sign in to view

nnethercote force-pushed the fix-double-handling-in-collect_tokens branch from a0f9bfd to 7d65809 Compare August 21, 2024 07:44

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Aug 21, 2024

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Aug 21, 2024

petrochenkov reviewed Aug 21, 2024

View reviewed changes

tests/ui/conditional-compilation/invalid-node-range-issue-129166.rs Show resolved Hide resolved

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 21, 2024

nnethercote force-pushed the fix-double-handling-in-collect_tokens branch from 7d65809 to aeaa62b Compare August 21, 2024 22:54

This comment has been minimized.

Sign in to view

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 7, 2024

bors added the merged-by-bors This PR was explicitly merged by bors. label Sep 8, 2024

bors merged commit 6d05f12 into rust-lang:master Sep 8, 2024
7 checks passed

rustbot added this to the 1.83.0 milestone Sep 8, 2024

bors mentioned this pull request Sep 8, 2024

interpret: make typed copies lossy wrt provenance and padding #129778

Merged

rustbot removed the perf-regression Performance regression. label Sep 8, 2024

nnethercote deleted the fix-double-handling-in-collect_tokens branch September 8, 2024 22:13

danielhenrymantilla reviewed Oct 18, 2024

View reviewed changes

nnethercote mentioned this pull request Nov 4, 2024

Revert "Avoid nested replacement ranges" from #129346. #132587

Merged

workingjubilee mentioned this pull request Nov 4, 2024

Rollup of 10 pull requests #132616

Closed

workingjubilee mentioned this pull request Nov 5, 2024

Rollup of 11 pull requests #132626

Merged

jieyouxu mentioned this pull request Nov 7, 2024

Add a regression test for #132587 #132727

Open

cuviper mentioned this pull request Nov 7, 2024

[beta] backports #132733

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix double handling in `collect_tokens` #129346

Fix double handling in `collect_tokens` #129346

nnethercote commented Aug 21, 2024

nnethercote commented Aug 21, 2024

This comment has been minimized.

nnethercote commented Aug 21, 2024

This comment has been minimized.

bors commented Aug 21, 2024

This comment has been minimized.

bors commented Aug 21, 2024

This comment has been minimized.

rust-timer commented Aug 21, 2024

petrochenkov commented Aug 21, 2024

petrochenkov commented Aug 21, 2024

nnethercote commented Aug 22, 2024

bors commented Aug 22, 2024

bors commented Aug 22, 2024

lqd commented Aug 22, 2024

This comment has been minimized.

rust-timer commented Aug 22, 2024

bors commented Sep 8, 2024

bors commented Sep 8, 2024

rust-timer commented Sep 8, 2024

danielhenrymantilla commented Oct 18, 2024

danielhenrymantilla Oct 18, 2024

nnethercote Nov 4, 2024

Noratrieb commented Oct 18, 2024

lqd commented Nov 4, 2024

Fix double handling in collect_tokens #129346

Fix double handling in collect_tokens #129346

Conversation

nnethercote commented Aug 21, 2024

nnethercote commented Aug 21, 2024

This comment has been minimized.

nnethercote commented Aug 21, 2024

This comment has been minimized.

bors commented Aug 21, 2024

This comment has been minimized.

bors commented Aug 21, 2024

This comment has been minimized.

rust-timer commented Aug 21, 2024

Overall result: ❌ regressions - ACTION NEEDED

petrochenkov commented Aug 21, 2024

petrochenkov commented Aug 21, 2024

nnethercote commented Aug 22, 2024

bors commented Aug 22, 2024

bors commented Aug 22, 2024

lqd commented Aug 22, 2024

This comment has been minimized.

rust-timer commented Aug 22, 2024

Overall result: ❌ regressions - ACTION NEEDED

bors commented Sep 8, 2024

bors commented Sep 8, 2024

rust-timer commented Sep 8, 2024

Overall result: ❌✅ regressions and improvements - no action needed

danielhenrymantilla commented Oct 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Noratrieb commented Oct 18, 2024

lqd commented Nov 4, 2024

Fix double handling in `collect_tokens` #129346

Fix double handling in `collect_tokens` #129346