Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updates for Pyodide builds after pyodide-build was unvendored #2090

Merged
merged 9 commits into from
Nov 22, 2024
Merged

Updates for Pyodide builds after pyodide-build was unvendored #2090

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
aa3bf38
Fix typos and capitalise
agriyakhetarpal Nov 21, 2024
f2bdd7f
Add docstring for `install_xbuildenv` function
agriyakhetarpal Nov 21, 2024
d54ab6f
Include Emscripten, Pyodide version in logger messages
agriyakhetarpal Nov 21, 2024
4b97ee8
Mark separation of pyodide-build and Pyodide
agriyakhetarpal Nov 21, 2024
8037155
Bump to Pyodide 0.26.4 (py3.12.7) + pyodide-build 0.29.0
agriyakhetarpal Nov 21, 2024
5ce4951
Use pyodide-build version as the source of truth
agriyakhetarpal Nov 21, 2024
f9b789b
Update Pyodide constraints
agriyakhetarpal Nov 21, 2024
15f5784
Use 3.12 as the Python version instead of 3.12.7
agriyakhetarpal Nov 21, 2024
35467fc
chore: appease the formatter
henryiii Nov 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ jobs:
run: uv run pytest --run-emulation ${{ matrix.arch }} test/test_emulation.py

test-pyodide:
name: Test cibuildwheel building pyodide wheels
name: Test cibuildwheel building Pyodide wheels
needs: lint
runs-on: ubuntu-24.04
timeout-minutes: 180
Expand Down
18 changes: 13 additions & 5 deletions cibuildwheel/pyodide.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ class PythonConfiguration:
version: str
identifier: str
pyodide_version: str
pyodide_build_version: str
emscripten_version: str
node_version: str

Expand All @@ -65,10 +66,15 @@ def install_emscripten(tmp: Path, version: str) -> Path:
return emcc_path


def install_xbuildenv(env: dict[str, str], pyodide_version: str) -> str:
def install_xbuildenv(env: dict[str, str], pyodide_build_version: str, pyodide_version: str) -> str:
"""Install a particular Pyodide xbuildenv version and set a path to the Pyodide root."""
# Since pyodide-build was unvendored from Pyodide v0.27.0, the versions of pyodide-build are
# not guaranteed to match the versions of Pyodide or be in sync with them. Hence, we shall
# specify the pyodide-build version in the root path, which will set up the xbuildenv for
# the requested Pyodide version.
pyodide_root = (
CIBW_CACHE_PATH
/ f".pyodide-xbuildenv-{pyodide_version}/{pyodide_version}/xbuildenv/pyodide-root"
/ f".pyodide-xbuildenv-{pyodide_build_version}/{pyodide_version}/xbuildenv/pyodide-root"
Copy link
Contributor

@henryiii henryiii Nov 21, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will not respect overrides (like latest or pip installing a new version), but will always be the one hard-coded. Was that intentional?

Copy link
Contributor Author

@agriyakhetarpal agriyakhetarpal Nov 21, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is intentional, since we don't have any ways to override here (no environment variables or the like, etc.). We should be fine with the hardcoded version. We know that users can still override pyodide-build using their constraint files if they like, but that won't be of much use (unless there's a bug in pyodide-build or if they want to use a fork or for any other reasons to use a constraint). I plan to rebase #2002 after this gets reviewed/merged.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, just saw your edit. How do we ensure that we use the version of pyodide-build that users install themselves, e.g., in before-build?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Constraint flags apply when it is setting up the environment. If a user installs it themselves, that will happen after the constrained version has already set up a new environment. Do we need the pyodide-build version in the env path?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if I understood that correctly 😅 If I got it right,

the constrained version has already set up a new environment

Is "environment" here the cibuildwheel venv that will be used for the build? If these changes constrain the pyodide-build version in this environment and constraints files will have no effect, I'd definitely like to try out a different method. Could you give me a pointer on how I could proceed?

Also, a side note – after thinking a bit more about this, I think it would be great not to hardcode the Pyodide version, too (not in this PR, though). In case we release a new version of pyodide-build, say, X, that drops compatibility with the Pyodide version Y listed in build-platforms.toml, overriding the version here with constraints to use X would make it unable to work with Pyodide version Y.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean the xbuildenv gets set up with the constrained versions, but before-build runs after the xbuildenv is setup, so it can't affect the version used to setup the xbuildenv. After that I believe you can change it, so it's only the pyodide-build version used for the xbuildenv setup that's a little harder to control.

)
with FileLock(CIBW_CACHE_PATH / "xbuildenv.lock"):
if pyodide_root.exists():
Expand Down Expand Up @@ -165,13 +171,15 @@ def setup_python(
env=env,
)

log.step("Installing emscripten...")
log.step(f"Installing Emscripten version: {python_configuration.emscripten_version} ...")
emcc_path = install_emscripten(tmp, python_configuration.emscripten_version)

env["PATH"] = os.pathsep.join([str(emcc_path.parent), env["PATH"]])

log.step("Installing Pyodide xbuildenv...")
env["PYODIDE_ROOT"] = install_xbuildenv(env, python_configuration.pyodide_version)
log.step(f"Installing Pyodide xbuildenv version: {python_configuration.pyodide_version} ...")
env["PYODIDE_ROOT"] = install_xbuildenv(
env, python_configuration.pyodide_build_version, python_configuration.pyodide_version
)

return env

Expand Down
2 changes: 1 addition & 1 deletion cibuildwheel/resources/build-platforms.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,5 +190,5 @@ python_configurations = [

[pyodide]
python_configurations = [
{ identifier = "cp312-pyodide_wasm32", version = "3.12.1", pyodide_version = "0.26.1", emscripten_version = "3.1.58", node_version = "v20" },
{ identifier = "cp312-pyodide_wasm32", version = "3.12", pyodide_version = "0.26.4", pyodide_build_version = "0.29.0", emscripten_version = "3.1.58", node_version = "v20" },
henryiii marked this conversation as resolved.
Show resolved Hide resolved
]
22 changes: 6 additions & 16 deletions cibuildwheel/resources/constraints-pyodide312.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,6 @@ charset-normalizer==3.4.0
# via requests
click==8.1.7
# via typer
cloudpickle==3.1.0
# via loky
cmake==3.31.0.1
# via pyodide-build
distlib==0.3.9
Expand All @@ -40,8 +38,6 @@ idna==3.10
# requests
leb128==1.0.8
# via auditwheel-emscripten
loky==3.4.1
# via pyodide-build
markdown-it-py==3.0.0
# via rich
mdurl==0.1.2
Expand All @@ -56,26 +52,24 @@ pip==24.3.1
# via -r .nox/update_constraints/tmp/constraints-pyodide.in
platformdirs==4.3.6
# via virtualenv
pydantic==2.9.2
pydantic==2.10.0
# via
# pyodide-build
# pyodide-lock
pydantic-core==2.23.4
pydantic-core==2.27.0
# via pydantic
pygments==2.18.0
# via rich
pyodide-build==0.26.1
pyodide-build==0.29.0
# via -r .nox/update_constraints/tmp/constraints-pyodide.in
pyodide-cli==0.2.4
# via
# auditwheel-emscripten
# pyodide-build
pyodide-lock==0.1.0a6
pyodide-lock==0.1.0a7
# via pyodide-build
pyproject-hooks==1.2.0
# via build
pyyaml==6.0.2
# via pyodide-build
requests==2.32.3
# via pyodide-build
resolvelib==1.1.0
Expand All @@ -95,13 +89,11 @@ sniffio==1.3.1
# via
# anyio
# httpx
typer==0.13.0
typer==0.13.1
# via
# auditwheel-emscripten
# pyodide-build
# pyodide-cli
types-requests==2.32.0.20241016
# via pyodide-build
typing-extensions==4.12.2
# via
# pydantic
Expand All @@ -110,9 +102,7 @@ typing-extensions==4.12.2
unearth==0.17.2
# via pyodide-build
urllib3==2.2.3
# via
# requests
# types-requests
# via requests
virtualenv==20.27.1
# via
# build
Expand Down
4 changes: 2 additions & 2 deletions noxfile.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,10 +106,10 @@ def update_constraints(session: nox.Session) -> None:
pyodides = build_platforms["pyodide"]["python_configurations"]
for pyodide in pyodides:
python_version = ".".join(pyodide["version"].split(".")[:2])
pyodide_version = pyodide["pyodide_version"]
pyodide_build_version = pyodide["pyodide_build_version"]
output_file = resources / f"constraints-pyodide{python_version.replace('.', '')}.txt"
tmp_file = Path(session.create_tmp()) / "constraints-pyodide.in"
tmp_file.write_text(f"pip\nbuild[virtualenv]\npyodide-build=={pyodide_version}")
tmp_file.write_text(f"pip\nbuild[virtualenv]\npyodide-build=={pyodide_build_version}")
session.run(
"uv",
"pip",
Expand Down
2 changes: 1 addition & 1 deletion test/test_emscripten.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@

def check_node():
# cibuildwheel adds a pinned node version to the PATH
# check it's in the PATH then, check it's the one that runs pyoodide
# check it's in the PATH then, check it's the one that runs Pyodide
cibw_cache_path = Path(sys.argv[1]).resolve(strict=True)
# find the node executable in PATH
node = shutil.which("node")
Expand Down
Loading