Allow lightning.qubit to skip C++ compilation and provide a Python-only binary #129

trbromley · 2021-08-11T13:08:24Z

Context:

PennyLane-Lightning can now be installed without compiling its C++ binaries and will fall back
to using the default.qubit implementation. Skipping compilation is achieved by setting the
SKIP_COMPILATION environment variable, e.g., export SKIP_COMPILATION=True. This feature
is intended for building a pure-Python wheel of PennyLane-Lightning as a backup for platforms
without a dedicated wheel.

Possible Drawbacks:

Users installing the non-compiled version will obviously have a device that is as fast as default.qubit. A warning has been added for this eventuality.

…navailable

trbromley · 2021-08-11T13:09:32Z

setup.py

@@ -116,7 +116,7 @@ def build_extensions(self):
        build_ext.build_extensions(self)


-if not os.environ.get("MOCK_DOCS", False):
+if not os.environ.get("SKIP_COMPILATION", False):


Reminder - I believe this will need the environment variable to be updated in RTD.

Also, maybe we should add docs testing in the CI tests 🤔 I expected the tests to fail because of this.

I think I just added following these instructions.

Building the docs in a PR would be quite useful.

trbromley · 2021-08-11T13:10:10Z

tests/test_apply.py

@@ -665,7 +666,7 @@ def circuit(x):
        for _ in range(100):
            runs.append(circuit(p))

-        assert np.isclose(np.mean(runs), -np.sin(p), atol=1e-3, rtol=0)
+        assert np.isclose(np.mean(runs), -np.sin(p), atol=1e-2, rtol=0)


This was occasionally failing 🤔 So I thought it'd be sufficient to lower the tolerance.

github-actions · 2021-08-11T13:10:40Z

Test Report (C++) on Ubuntu

    1 files ±0     1 suites ±0 0s ⏱️ ±0s
164 tests ±0 164 ✔️ ±0 0 💤 ±0 0 ❌ ±0
694 runs ±0 694 ✔️ ±0 0 💤 ±0 0 ❌ ±0

Results for commit 333ed09. ± Comparison against base commit 333ed09.

♻️ This comment has been updated with latest results.

codecov · 2021-08-11T13:11:01Z

Codecov Report

Merging #129 (bcf65af) into master (e5c980e) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##            master      #129   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            3         3           
  Lines           56        71   +15     
=========================================
+ Hits            56        71   +15

Impacted Files	Coverage Δ
pennylane_lightning/lightning_qubit.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e5c980e...bcf65af. Read the comment docs.

trbromley · 2021-08-11T14:15:17Z

.github/workflows/deploy.yml

+  build-pure-python-wheel:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout PennyLane-Lightning
+        uses: actions/checkout@v2
+        with:
+          path: main
+
+      - uses: actions/setup-python@v2
+        with:
+          python-version: '3.7'
+
+      - name: Build wheels
+        run: |
+          python -m pip install --upgrade pip wheel
+          cd main
+          python setup.py bdist_wheel
+        env:
+          SKIP_COMPILATION: True
+
+      - uses: actions/upload-artifact@v2
+        with:
+          name: pure-python-wheels.zip
+          path: main/dist/*.whl


This is the main change

trbromley · 2021-08-11T14:15:39Z

.github/workflows/deploy.yml

+
+  upload-wheels:
+    runs-on: ubuntu-latest
+    needs: [build-wheels, build-pure-python-wheel]


Also added job as a dependency

trbromley · 2021-08-11T14:18:56Z

.github/workflows/deploy.yml

-        with:
-          user: __token__
-          password: ${{ secrets.PYPI }}
+name: Deploy


Unfortunately the diff for this file is not great, I believe my editor changed from \r\n to \n for the new line style. The main changes are highlighted below.

trbromley · 2021-08-11T14:19:52Z

.github/workflows/deploy.yml

+
+  upload-source:
+    runs-on: ubuntu-latest
+    needs: [build-wheels, build-pure-python-wheel]


Added as dependency, but not sure it's strictly needed.

trbromley · 2021-08-11T14:21:07Z

.github/workflows/tests_without_binary.yml

@@ -0,0 +1,58 @@
+name: Testing without binary


Very similar to the tests.yml workflow but with the c++ tests removed and a few tweaks to test only for without binaries.

trbromley · 2021-08-11T14:47:23Z

setup.py

@@ -168,7 +168,6 @@ def build_extensions(self):
 requirements = [
    "numpy",
    "pennylane>=0.15",
-    "pybind11",


Note this removal. This means that all the dependencies of lightning are also dependencies of PennyLane.

Just so I'm clear, this change requires the user to ensure they have pybind11 available manually, rather than as an auto-pulled dep when we try to build? As in, explictly running pip install -r requirements.txt is needed?

Yes, agreed - users building from source will need to do pip install -r requirements.txt. I added this in the installation instructions: 3c54052

There might be one additional use-case, not sure how common it is though:

A user wants to pip install lightning on an unsupported system and compile from source.

In this case, the user would have to:

Make sure all C++ dependencies are installed

Run

pip install pybind11 pennylane-lightning --no-binary :all:

Potentially this could occur in Docker workflows. Say a user is using the Alpine Linux image that uses musl; (as far as I am aware?) no wheels are available for this platform, but they

would like to avoid using git to reduce the docker image size

would like to use the C++ simulator

Good idea, have updated the instructions: 8e6ed64

trbromley · 2021-08-11T22:13:27Z

[ch8104]

mlxd · 2021-08-12T09:50:58Z

.github/workflows/deploy.yml

+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]


I wonder can we extract the artifacts create from the other wheel builder scripts, rather than rebuild here. Assuming a successful deploy, we fail to get the new architectures using this.

Can we get the new build scripts (wheel_linux_aarch64.yml, etc) to run also on the published trigger (e.g., like this deploy workflow). We could then have an additional job in each workflow to upload the wheels if the event is published (if: ${{ github.event_name == 'published' }}, taken from here)?

We can then:

Take the pure python build into it's own workflow, as you suggested below

Reduce this deploy workflow to just uploading the source?

Sure, that sounds like a good plan.

Nice! Are you ok if we do that in a separate PR?

Yea, perfectly fine with me.

mlxd · 2021-08-12T09:52:22Z

.github/workflows/deploy.yml

+          name: ${{ runner.os }}-wheels.zip
+          path: ./wheelhouse/*.whl
+
+  build-pure-python-wheel:


As with the above comment, I wonder if this should be offloaded into a separate build file, as the case for the other architectures. What do you think?

mlxd · 2021-08-12T09:58:25Z

setup.py

@@ -116,7 +116,7 @@ def build_extensions(self):
        build_ext.build_extensions(self)


-if not os.environ.get("MOCK_DOCS", False):
+if not os.environ.get("SKIP_COMPILATION", False):


Building the docs in a PR would be quite useful.

.github/CHANGELOG.md

Co-authored-by: Lee James O'Riordan <[email protected]>

trevor-vincent

Really good idea Tom, I have nothing to add beyond what Lee has already commented on.

mlxd

Great job on this @trbromley !

trbromley added 7 commits August 11, 2021 07:52

Makefile clean removes binary

252b0f8

Update lightning qubit to fallback to default qubit if binaries are u…

09d1fe3

…navailable

Tidy up test imports

7e6ecfd

Add skip to tests that do not make sense if lightning is default

54ec268

Add test

69e35f1

Rename

10772da

Rename environment variable to be clearer

b9f5f55

trbromley commented Aug 11, 2021

View reviewed changes

trbromley added 7 commits August 11, 2021 09:21

Add workflow for no binary

e9263ee

Update name of workflow

399820e

Update deploy

64d6941

Update workflow

17f1043

Update workflow

1c271eb

Update workflow

c651fdc

Update deploy workflow

0357dbd

trbromley commented Aug 11, 2021

View reviewed changes

trbromley added 3 commits August 11, 2021 10:29

Add to changelog

aa6af3e

Remove pybind11 as a package dependency of lightning

f8026e1

Update makefile

2ce5a66

trbromley commented Aug 11, 2021

View reviewed changes

trbromley marked this pull request as ready for review August 11, 2021 14:56

trbromley requested review from mlxd and trevor-vincent August 11, 2021 14:57

mlxd reviewed Aug 12, 2021

View reviewed changes

trbromley and others added 3 commits August 12, 2021 14:32

Add mention of requirements.txt in installation instructions

3c54052

Apply suggestions from code review

edb3350

Co-authored-by: Lee James O'Riordan <[email protected]>

Fix line spacing

5e52b35

trevor-vincent approved these changes Aug 13, 2021

View reviewed changes

Update installation instructions

8e6ed64

trbromley requested a review from mlxd August 13, 2021 11:59

mlxd approved these changes Aug 13, 2021

View reviewed changes

trbromley and others added 2 commits August 13, 2021 08:20

Merge branch 'master' into provide_universal_binary

df3bdef

Update python version for RTD

bcf65af

trbromley merged commit 333ed09 into master Aug 13, 2021

trbromley deleted the provide_universal_binary branch August 13, 2021 15:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow lightning.qubit to skip C++ compilation and provide a Python-only binary #129

Allow lightning.qubit to skip C++ compilation and provide a Python-only binary #129

trbromley commented Aug 11, 2021 •

edited

Loading

trbromley Aug 11, 2021

trbromley Aug 11, 2021

trbromley Aug 11, 2021

mlxd Aug 12, 2021

trbromley Aug 11, 2021

github-actions bot commented Aug 11, 2021 •

edited

Loading

codecov bot commented Aug 11, 2021 •

edited

Loading

trbromley Aug 11, 2021

trbromley Aug 11, 2021

trbromley Aug 11, 2021

trbromley Aug 11, 2021

trbromley Aug 11, 2021

trbromley Aug 11, 2021

mlxd Aug 12, 2021

trbromley Aug 12, 2021 •

edited

Loading

josh146 Aug 13, 2021 •

edited

Loading

trbromley Aug 13, 2021

trbromley commented Aug 11, 2021

mlxd Aug 12, 2021

trbromley Aug 12, 2021

mlxd Aug 13, 2021

trbromley Aug 13, 2021

mlxd Aug 13, 2021

mlxd Aug 12, 2021

mlxd Aug 12, 2021

trevor-vincent left a comment

mlxd left a comment

Allow lightning.qubit to skip C++ compilation and provide a Python-only binary #129

Allow lightning.qubit to skip C++ compilation and provide a Python-only binary #129

Conversation

trbromley commented Aug 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Aug 11, 2021 • edited Loading

Test Report (C++) on Ubuntu

codecov bot commented Aug 11, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trbromley Aug 12, 2021 • edited Loading

Choose a reason for hiding this comment

josh146 Aug 13, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trbromley commented Aug 11, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trevor-vincent left a comment

Choose a reason for hiding this comment

mlxd left a comment

Choose a reason for hiding this comment

trbromley commented Aug 11, 2021 •

edited

Loading

github-actions bot commented Aug 11, 2021 •

edited

Loading

codecov bot commented Aug 11, 2021 •

edited

Loading

trbromley Aug 12, 2021 •

edited

Loading

josh146 Aug 13, 2021 •

edited

Loading