Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize flattening in apache airflow workspace #11313

Merged
merged 2 commits into from
Feb 7, 2025

Conversation

konstin
Copy link
Member

@konstin konstin commented Feb 7, 2025

Motivation

No-op uv lock in apache airflow (891c67f210ab7c877d1f00ea6ea3d3cdbb0e96ef) is slow, which makes uv run slow, too.

Reference project:

$ hyperfine "uv run python -c \"print('hi')\""
Benchmark 1: uv run python -c "print('hi')"
Time (mean ± σ):      16.3 ms ±   1.5 ms    [User: 9.8 ms, System: 6.4 ms]
Range (min … max):    13.0 ms …  20.0 ms    186 runs

Apache airflow before:

$ hyperfine "uv run python -c \"print('hi')\""
Benchmark 1: uv run python -c "print('hi')"
Time (mean ± σ):     161.0 ms ±   5.2 ms    [User: 135.3 ms, System: 24.1 ms]
Range (min … max):   155.0 ms … 176.3 ms    18 runs

Optimization

FlatRequiresDist::from_requirements is taking 50% of main thread runtime.

Before:

image

After both commits:

image

Apache airflow after the first commit:

$ hyperfine "uv-profiling run python -c \"print('hi')\""
Benchmark 1: uv-profiling run python -c "print('hi')"
  Time (mean ± σ):     122.3 ms ±   5.4 ms    [User: 96.1 ms, System: 24.7 ms]
  Range (min … max):   114.0 ms … 133.2 ms    23 runs

Apache airflow after the second commit:

$ hyperfine "uv-profiling run python -c \"print('hi')\""
Benchmark 1: uv-profiling run python -c "print('hi')"
  Time (mean ± σ):     108.5 ms ±   3.4 ms    [User: 83.2 ms, System: 24.2 ms]
  Range (min … max):   103.6 ms … 119.9 ms    28 runs

@konstin konstin added the performance Potential performance improvement label Feb 7, 2025
@konstin konstin requested a review from ibraheemdev February 7, 2025 11:30
@@ -1002,11 +1003,19 @@ impl MarkerTree {
///
/// ASSUMPTION: There is one `extra = "..."`, and it's either the only marker or part of the
/// main conjunction.
pub fn top_level_extra_name(self) -> Option<ExtraName> {
pub fn top_level_extra_name(self) -> Option<Cow<'static, ExtraName>> {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we didn't go through the DNF for the general top level marker case this could be a &'static ExtraName

@charliermarsh charliermarsh merged commit 6e5479f into main Feb 7, 2025
73 checks passed
@charliermarsh charliermarsh deleted the konsti/top-level-marker branch February 7, 2025 22:08
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Feb 11, 2025
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.5.29` -> `0.5.30` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>astral-sh/uv (astral-sh/uv)</summary>

### [`v0.5.30`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0530)

[Compare Source](astral-sh/uv@0.5.29...0.5.30)

##### Python

The managed PyPy distributions have been updated for PyPy v7.3.18, which includes:

-   PyPy3.10, which updates the standard library from Python 3.10.14 to 3.10.19
-   PyPy3.11, which adds beta support for Python 3.11.11

See the [PyPy release](https://pypy.org/posts/2025/02/pypy-v7318-release.html) for more details.

##### Enhancements

-   Add `uv sync --dry-run` ([#&#8203;11299](astral-sh/uv#11299))
-   Ignore `#egg` fragment in HTML Simple API response ([#&#8203;11340](astral-sh/uv#11340))

##### Configuration

-   Add `NO_BINARY` and `NO_BINARY_PACKAGE` environment variables ([#&#8203;11399](astral-sh/uv#11399))

##### Performance

-   Avoid re-cloning name when populating ambiguous set ([#&#8203;11401](astral-sh/uv#11401))
-   Optimize flattening in large workspaces ([#&#8203;11313](astral-sh/uv#11313))

##### Bug fixes

-   Allow dynamic packages to be overloaded ([#&#8203;11400](astral-sh/uv#11400))
-   Fix credential caching for index roots when URL ends in `simple/` ([#&#8203;11336](astral-sh/uv#11336))
-   Fix marker merging for requirements.txt for psycopg ([#&#8203;11298](astral-sh/uv#11298))
-   Set 777 permissions on locked files ([#&#8203;11328](astral-sh/uv#11328))
-   Support extras in `@` requests for tools ([#&#8203;11335](astral-sh/uv#11335))
-   Upgrade `astral-tokio-tar` to v0.5.1 ([#&#8203;11359](astral-sh/uv#11359))
-   Avoid missing logging for no-op upgrade events ([#&#8203;11301](astral-sh/uv#11301))
-   Use refined specifiers when logging narrowed Python range ([#&#8203;11334](astral-sh/uv#11334))
-   Don't use popup-generating `eprintln` in trampoline warnings ([#&#8203;11295](astral-sh/uv#11295))
-   Patch pkg-config files to be relocatable ([#&#8203;11291](astral-sh/uv#11291))
-   Fix a case of duplicate `torch` packages when using conflicting extras ([#&#8203;11323](astral-sh/uv#11323))

##### Documentation

-   Add docs for `uv tool install --editable` ([#&#8203;11280](astral-sh/uv#11280))
-   Fix broken anchors in README and docs index ([#&#8203;11338](astral-sh/uv#11338))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xNjQuMSIsInVwZGF0ZWRJblZlciI6IjM5LjE2NC4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Potential performance improvement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants