cryptography-in-pyodide.md

Cryptography in Pyodide

Today I was evaluating if the Python cryptography package was a sensible depedency for one of my projects.

It's clearly the most responsible package to use for implementing encryption in Python, but I was nervous about adding it as a dependency to a project that could work without using it at all (with some design changes).

My key concern: I want this project to work running in Pyodide in WebAssembly in the future, and was nervous that cryptography, being written partially in Rust, wouldn't work in that environment.

It turns out it works just fine!

I tested it in the Pyodide REPL at https://pyodide.org/en/stable/console.html

The following import code worked without errors:

import micropip
await micropip.install("cryptography")
from cryptography.fernet import Fernet

Then I tested it like this:

key = Fernet.generate_key()
cipher_suite = Fernet(key)
encrypted_text = cipher_suite.encrypt(b"Secret message")
print(encrypted_text)
# b'gAAAAABlY7AZ-OuJAEv1J4KufF8vpreyehPeejdPqluXwD0G_mfFg-Zl1AvEza6F2DXbWMEkIcKc4B0Hb0qJ457pje5FhO5Vyw=='
decrypted_text = cipher_suite.decrypt(encrypted_text)
print(decrypted_text)
# b'Secret message'

From sniffing around in the browser DevTools network panel, it turns out Pyodide provides its own packaged version of the Cryptography package in a file called cryptography-39.0.2-cp311-cp311-emscripten_3_1_45_wasm32.whl.

I found the source of this custom package in the Pyodide repository, in the packages/cryptography folder.

More packages

That packages/ folder has a whole bunch of other useful modules that have been custom packaged to work with Pyodide. A few that caught my eye:

• Jinja2
• Pillow
• Pygments
• biopython
• fastparquet
• ffmpeg
• gdal
• geopandas
• geos
• html5lib
• jsonschema
• libheif
• libwebp
• lxml
• msgpack
• shapely
• sqlalchemy
• xgboost
• sqlite3

Each of these comes with a meta.yaml file that defines how it should be compiled, plus a test Python module to verify it and optionally a set of patches to apply before compilation.

Here's what packages/sqlite3/meta.yaml looks like:

package:
  name: sqlite3
  version: 1.0.0 # Nonsense
  tag:
    - always
  top-level:
    - sqlite3
    - _sqlite3
source:
  sha256: $(PYTHON_ARCHIVE_SHA256)
  url: $(PYTHON_ARCHIVE_URL)
build:
  type: cpython_module
  script: |
    export FILES=(
      "Modules/_sqlite/blob.c"
      "Modules/_sqlite/connection.c"
      "Modules/_sqlite/cursor.c"
      "Modules/_sqlite/microprotocols.c"
      "Modules/_sqlite/module.c"
      "Modules/_sqlite/prepare_protocol.c"
      "Modules/_sqlite/row.c"
      "Modules/_sqlite/statement.c"
      "Modules/_sqlite/util.c"
    )

    embuilder build sqlite3 --pic

    for file in "${FILES[@]}"; do
      emcc $STDLIB_MODULE_CFLAGS -c "${file}" -o "${file/.c/.o}"  \
        -sUSE_SQLITE3 -DMODULENAME=sqlite
    done

    OBJECT_FILES=$(find Modules/_sqlite/ -name "*.o")
    emcc $OBJECT_FILES -o $DISTDIR/_sqlite3.so $SIDE_MODULE_LDFLAGS \
       -sUSE_SQLITE3 -lsqlite3

    cd Lib && tar --exclude=test -cf - sqlite3 | tar -C $DISTDIR -xf -

One thing I don't understand is why some pure-Python packages such as Click also get this treatment - since those should work if installed directly from PyPI using micropip.install().

Update: John Ott figured this out. There's an open issue, Remove pure Python packages from the repository, which says:

Currently, we have dozens of pure Python packages in the repository. Some packages require a patch, and some packages don't have a wheel, but many packages exist only because other packages with C-extension depend on them in runtime.