fix(core): optimize daemon output glob matching #27775

AgentEnder · 2024-09-04T18:46:54Z

Current Behavior

Nonexistant outputs are considered globs
Glob outputs are walked from workspace root, which is slow on large repos

[NX Daemon Server] - 2024-09-06T19:44:39.674Z - Done responding to the client outputsHashesMatch
[NX Daemon Server] - 2024-09-06T19:44:39.674Z - Handled OUTPUTS_HASHES_MATCH. Handling time: 94. Response time: 0.
[NX Daemon Server] - 2024-09-06T19:44:39.775Z - [REQUEST]: Responding to the client. recordOutputsHash
[NX Daemon Server] - 2024-09-06T19:44:39.775Z - Done responding to the client recordOutputsHash
[NX Daemon Server] - 2024-09-06T19:44:39.775Z - Handled RECORD_OUTPUTS_HASH. Handling time: 100. Response time: 0.
[NX Daemon Server] - 2024-09-06T19:44:39.818Z - [REQUEST]: Responding to the client. PROCESS_IN_BACKGROUND
[NX Daemon Server] - 2024-09-06T19:44:39.818Z - Done responding to the client PROCESS_IN_BACKGROUND
[NX Daemon Server] - 2024-09-06T19:44:39.818Z - Handled PROCESS_IN_BACKGROUND. Handling time: 14. Response time: 0.

Expected Behavior

Nonexistant outputs are only globs if they should be
Globs are a bit faster

[NX Daemon Server] - 2024-09-06T19:43:36.899Z - Handled OUTPUTS_HASHES_MATCH. Handling time: 0. Response time: 0.
[NX Daemon Server] - 2024-09-06T19:43:36.900Z - [REQUEST]: Responding to the client. recordOutputsHash
[NX Daemon Server] - 2024-09-06T19:43:36.900Z - Done responding to the client recordOutputsHash
[NX Daemon Server] - 2024-09-06T19:43:36.900Z - Handled RECORD_OUTPUTS_HASH. Handling time: 0. Response time: 0.
[NX Daemon Server] - 2024-09-06T19:43:36.944Z - [REQUEST]: Responding to the client. PROCESS_IN_BACKGROUND
[NX Daemon Server] - 2024-09-06T19:43:36.944Z - Done responding to the client PROCESS_IN_BACKGROUND
[NX Daemon Server] - 2024-09-06T19:43:36.944Z - Handled PROCESS_IN_BACKGROUND. Handling time: 13. Response time: 0.
[NX Daemon Server] - 2024-09-06T19:43:36.949Z - Uploading file artifacts
[NX Daemon Server] - 2024-09-06T19:43:36.949Z - Done uploading file artifacts

Note timings are from Nx repo, close enough to be comparable. No real improvement was expected here, mainly checking that things didn't get worse.

Related Issue(s)

Fixes #

vercel · 2024-09-04T18:47:08Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Updated (UTC)
nx-dev	⬜️ Ignored (Inspect)	Visit Preview	Sep 5, 2024 10:07pm

nx-cloud · 2024-09-04T18:50:30Z

☁️ Nx Cloud Report

CI is running/has finished running commands for commit e70bbbd. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution

✅ Successfully ran 4 targets

Sent with 💌 from NxCloud.

FrozenPandaz · 2024-09-04T18:54:11Z

packages/nx/src/migrations/update-15-0-0/prefix-outputs.ts

@@ -13,6 +13,7 @@ import {
 } from '../../tasks-runner/utils';
 import { updateJson } from '../../generators/utils/json';
 import { PackageJson } from '../../utils/package-json';
+import { getTransformableOutputs } from 'nx/src/native';


Relative import

FrozenPandaz · 2024-09-04T18:56:05Z

packages/nx/src/native/cache/expand_outputs.rs

+            let root_path = directory.join(&root);
+            let glob_set = build_glob_set(&patterns)?;
+            trace!("walking directory: {:?}", root_path);
+            println!("walking directory: {:?}", root_path);


FrozenPandaz · 2024-09-04T18:56:10Z

packages/nx/src/native/cache/expand_outputs.rs

+                    println!("file: {}", &file.normalized_path);
+                    println!("matches: {}", glob_set.is_match(&file.normalized_path));


FrozenPandaz · 2024-09-04T18:56:52Z

packages/nx/src/native/cache/expand_outputs.rs


-        files.extend(found_paths);
+            files.extend(found_paths);
+        }
    }

    if !directories.is_empty() {


Can we use the multi-threaded walker below?

We can do this some other time.

FrozenPandaz · 2024-09-04T18:57:20Z

packages/nx/src/native/glob.rs

+pub(crate) use glob_transform::partition_glob;
+


never got used?

FrozenPandaz · 2024-09-04T18:59:54Z

packages/nx/src/native/glob/glob_transform.rs

+    while let Some(group) = remove_first(&mut groups) {
+        if group.is_empty() {
+            continue;
+        }
+        match &group[0] {
+            GlobGroup::NonSpecial(value) => {
+                if !contains_glob_pattern(&value) && pattern_segments.is_empty() {
+                    leading_dir_segments.push(value.to_string());
+                } else {
+                    pattern_segments.push(group);
+                }
+            }
+            _ => {
+                pattern_segments.push(group);
+            }
+        }
+    }


This is written weirdly... can we not go through it as an iter?

FrozenPandaz · 2024-09-04T19:01:59Z

packages/nx/src/tasks-runner/utils.ts

 }

 export function transformLegacyOutputs(
  projectRoot: string,
-  error: InvalidOutputsError
+  outputs: string[],
+  transformableOutputs: Set<string>


What if we determine what is transformable here in this function rather than passing it in?

FrozenPandaz · 2024-09-04T21:29:53Z

packages/storybook/src/plugins/plugin.ts

@@ -291,7 +291,7 @@ function normalizeOutputPath(projectRoot: string): string | undefined {
  if (projectRoot === '.') {


We don't need this if anymore?

FrozenPandaz · 2024-09-04T21:38:10Z

packages/nx/src/native/glob/glob_transform.rs

@@ -107,6 +113,33 @@ fn build_segment(
    }
 }

+pub fn partition_glob(glob: &str) -> (String, Vec<String>) {
+    let (negated, groups) = parse_glob(glob).unwrap();


Make this a Result and use ? here.

Suggested change

let (negated, groups) = parse_glob(glob).unwrap();

let (negated, groups) = parse_glob(glob)?;

FrozenPandaz · 2024-09-04T21:39:45Z

packages/nx/src/native/glob/glob_transform.rs

+    let mut leading_dir_segments: Vec<String> = vec![];
+    let mut pattern_segments = vec![];
+    groups
+        .into_iter()
+        .filter(|group| !group.is_empty())
+        .for_each(|group| match &group[0] {
+            GlobGroup::NonSpecial(value) => {
+                if !contains_glob_pattern(&value) && pattern_segments.is_empty() {
+                    leading_dir_segments.push(value.to_string());
+                } else {
+                    pattern_segments.push(group);
+                }
+            }
+            _ => {
+                pattern_segments.push(group);
+            }
+        });


This could use .partition_map?

FrozenPandaz · 2024-09-04T21:41:32Z

packages/nx/src/native/glob/glob_transform.rs

+    #[test]
+    fn should_partition_glob_with_leading_dirs_and_no_patterns() {
+        let (leading_dirs, globs) = super::partition_glob("dist/app/");
+        assert_eq!(leading_dirs, "dist/app");
+        assert_eq!(globs, [] as [String; 0]);
+    }


Please test with something like... dist/app/**/something/**/*.js

This should be dist/app | **/something/**/*.js .

github-actions · 2024-09-06T20:03:57Z

Failed to publish a PR release of this pull request, triggered by @FrozenPandaz.
See the failed workflow run at: https://github.com/nrwl/nx/actions/runs/10744325239

FrozenPandaz · 2024-09-05T18:57:04Z

packages/nx/src/native/cache/expand_outputs.rs


-        files.extend(found_paths);
+            files.extend(found_paths);
+        }
    }

    if !directories.is_empty() {


We can do this some other time.

## Current Behavior - Nonexistant outputs are considered globs - Glob outputs are walked from workspace root, which is slow on large repos [NX Daemon Server] - 2024-09-06T19:44:39.674Z - Done responding to the client outputsHashesMatch [NX Daemon Server] - 2024-09-06T19:44:39.674Z - Handled OUTPUTS_HASHES_MATCH. Handling time: 94. Response time: 0. [NX Daemon Server] - 2024-09-06T19:44:39.775Z - [REQUEST]: Responding to the client. recordOutputsHash [NX Daemon Server] - 2024-09-06T19:44:39.775Z - Done responding to the client recordOutputsHash [NX Daemon Server] - 2024-09-06T19:44:39.775Z - Handled RECORD_OUTPUTS_HASH. Handling time: 100. Response time: 0. [NX Daemon Server] - 2024-09-06T19:44:39.818Z - [REQUEST]: Responding to the client. PROCESS_IN_BACKGROUND [NX Daemon Server] - 2024-09-06T19:44:39.818Z - Done responding to the client PROCESS_IN_BACKGROUND [NX Daemon Server] - 2024-09-06T19:44:39.818Z - Handled PROCESS_IN_BACKGROUND. Handling time: 14. Response time: 0. ## Expected Behavior - Nonexistant outputs are only globs if they should be - Globs are a bit faster [NX Daemon Server] - 2024-09-06T19:43:36.899Z - Handled OUTPUTS_HASHES_MATCH. Handling time: 0. Response time: 0. [NX Daemon Server] - 2024-09-06T19:43:36.900Z - [REQUEST]: Responding to the client. recordOutputsHash [NX Daemon Server] - 2024-09-06T19:43:36.900Z - Done responding to the client recordOutputsHash [NX Daemon Server] - 2024-09-06T19:43:36.900Z - Handled RECORD_OUTPUTS_HASH. Handling time: 0. Response time: 0. [NX Daemon Server] - 2024-09-06T19:43:36.944Z - [REQUEST]: Responding to the client. PROCESS_IN_BACKGROUND [NX Daemon Server] - 2024-09-06T19:43:36.944Z - Done responding to the client PROCESS_IN_BACKGROUND [NX Daemon Server] - 2024-09-06T19:43:36.944Z - Handled PROCESS_IN_BACKGROUND. Handling time: 13. Response time: 0. [NX Daemon Server] - 2024-09-06T19:43:36.949Z - Uploading file artifacts [NX Daemon Server] - 2024-09-06T19:43:36.949Z - Done uploading file artifacts > Note timings are from Nx repo, close enough to be comparable. No real improvement was expected here, mainly checking that things didn't get worse. ## Related Issue(s)  Fixes # --------- Co-authored-by: FrozenPandaz <[email protected]> (cherry picked from commit ac9da41)

github-actions · 2024-09-12T00:09:15Z

This pull request has already been merged/closed. If you experience issues related to these changes, please open a new issue referencing this pull request.

AgentEnder requested review from a team as code owners September 4, 2024 18:46

AgentEnder requested review from xiongemi and FrozenPandaz September 4, 2024 18:46

FrozenPandaz requested changes Sep 4, 2024

View reviewed changes

AgentEnder requested a review from FrozenPandaz September 4, 2024 19:44

FrozenPandaz and others added 4 commits September 4, 2024 16:21

chore(repo): wip

15f848d

fix(misc): fix

d931e75

fix(core): optimize glob output resolution time

818fbd5

chore(core): code review

1c3b07d

AgentEnder force-pushed the fast-walk branch from 632ad9a to 1c3b07d Compare September 4, 2024 20:23

fix(storybook): use correct outputs

f601907

AgentEnder requested a review from a team as a code owner September 4, 2024 21:28

AgentEnder requested a review from Coly010 September 4, 2024 21:28

FrozenPandaz requested changes Sep 4, 2024

View reviewed changes

chore(core): fixes

c453042

FrozenPandaz force-pushed the fast-walk branch from debda00 to c453042 Compare September 5, 2024 18:59

chore(storybook): update snapshots

e70bbbd

AgentEnder requested a review from FrozenPandaz September 6, 2024 19:35

FrozenPandaz approved these changes Sep 6, 2024

View reviewed changes

FrozenPandaz merged commit ac9da41 into master Sep 6, 2024
6 checks passed

FrozenPandaz deleted the fast-walk branch September 6, 2024 21:07

github-actions bot locked as resolved and limited conversation to collaborators Sep 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(core): optimize daemon output glob matching #27775

fix(core): optimize daemon output glob matching #27775

AgentEnder commented Sep 4, 2024 •

edited

Loading

vercel bot commented Sep 4, 2024 •

edited

Loading

nx-cloud bot commented Sep 4, 2024 •

edited

Loading

FrozenPandaz Sep 4, 2024

FrozenPandaz Sep 4, 2024

FrozenPandaz Sep 4, 2024

FrozenPandaz Sep 4, 2024

FrozenPandaz Sep 5, 2024

FrozenPandaz Sep 4, 2024

FrozenPandaz Sep 4, 2024

FrozenPandaz Sep 4, 2024

FrozenPandaz Sep 4, 2024

FrozenPandaz Sep 4, 2024

FrozenPandaz Sep 4, 2024

FrozenPandaz Sep 4, 2024

github-actions bot commented Sep 6, 2024

FrozenPandaz Sep 5, 2024

github-actions bot commented Sep 12, 2024

		println!("file: {}", &file.normalized_path);
		println!("matches: {}", glob_set.is_match(&file.normalized_path));

		@@ -291,7 +291,7 @@ function normalizeOutputPath(projectRoot: string): string \| undefined {
		if (projectRoot === '.') {

	let (negated, groups) = parse_glob(glob).unwrap();
	let (negated, groups) = parse_glob(glob)?;

fix(core): optimize daemon output glob matching #27775

fix(core): optimize daemon output glob matching #27775

Conversation

AgentEnder commented Sep 4, 2024 • edited Loading

Current Behavior

Expected Behavior

Related Issue(s)

vercel bot commented Sep 4, 2024 • edited Loading

nx-cloud bot commented Sep 4, 2024 • edited Loading

☁️ Nx Cloud Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Sep 6, 2024

Choose a reason for hiding this comment

github-actions bot commented Sep 12, 2024

AgentEnder commented Sep 4, 2024 •

edited

Loading

vercel bot commented Sep 4, 2024 •

edited

Loading

nx-cloud bot commented Sep 4, 2024 •

edited

Loading