add cl-quil-benchmarking system quilc-perf for perf. benchmarking #733

ghost · 2021-09-24T19:53:19Z

This is part of issue "Investigate performance at 72-150 qubits #565".

new modules:
benchmarking/quilc-perf.lisp - primarily exports
benchmark-nq function to get timings as nQ increases for various
chip and program types

benchmarking/quilc-mon-prof.lisp - builds on quilc-perf.lisp and is
for monitoring and profiling, as opposed to simple benchmarks, and
is a WIP.

Makefile - add targets to run benchmark-nq a couple of ways

cl-quil-benchmarking.asd - add new module quilc-perf and depend on 3
additional systems: metering, qvm-app, sb-sprof

ghost · 2021-09-24T19:54:07Z

Here's timing output from make benchmark-nq for BASELINE:

+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| STATIC/FULLY-CONNECTED 10  |     2.36 |     0 |        2 |
| STATIC/FULLY-CONNECTED 30  |    17.72 |     0 |        2 |
| STATIC/FULLY-CONNECTED 50  |    49.54 |     0 |        2 |
| STATIC/FULLY-CONNECTED 70  |    98.66 |     0 |        2 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| BELL/FULLY-CONNECTED 10    |     4.70 |     0 |        9 |
| BELL/FULLY-CONNECTED 30    |    27.75 |     0 |       29 |
| BELL/FULLY-CONNECTED 50    |    65.21 |     0 |       49 |
| BELL/FULLY-CONNECTED 70    |   119.45 |     0 |       69 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| QFT/FULLY-CONNECTED 10     |     9.22 |     0 |       37 |
| QFT/FULLY-CONNECTED 30     |    93.06 |     0 |      117 |
| QFT/FULLY-CONNECTED 50     |   350.25 |     0 |      197 |
| QFT/FULLY-CONNECTED 70     |  1121.33 |     0 |      277 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| HADAMARD/FULLY-CONNECTED 10 |     1.58 |     0 |        0 |
| HADAMARD/FULLY-CONNECTED 30 |    15.39 |     0 |        0 |
| HADAMARD/FULLY-CONNECTED 50 |    44.10 |     0 |        0 |
| HADAMARD/FULLY-CONNECTED 70 |    89.73 |     0 |        0 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| STATIC/LINEAR 10           |     0.69 |     0 |        2 |
| STATIC/LINEAR 30           |     1.46 |     0 |        2 |
| STATIC/LINEAR 50           |     2.15 |     0 |        2 |
| STATIC/LINEAR 70           |     2.84 |     0 |        2 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| BELL/LINEAR 10             |     2.90 |     0 |        9 |
| BELL/LINEAR 30             |    10.22 |     0 |       29 |
| BELL/LINEAR 50             |    18.93 |     0 |       49 |
| BELL/LINEAR 70             |    31.77 |     0 |       69 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| QFT/LINEAR 10              |     6.20 |    61 |       79 |
| QFT/LINEAR 30              |    19.30 |   641 |      265 |
| QFT/LINEAR 50              |    93.24 |  1821 |      451 |
| QFT/LINEAR 70              |   427.92 |  3601 |      637 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| HADAMARD/LINEAR 10         |     0.06 |     0 |        0 |
| HADAMARD/LINEAR 30         |     0.26 |     0 |        0 |
| HADAMARD/LINEAR 50         |     0.61 |     0 |        0 |
| HADAMARD/LINEAR 70         |     1.16 |     0 |        0 |
+----------------------------+----------+-------+----------+

Environment info:

Run on Fri Sep 17 20:40:24 PDT 2021
Hardware:
  Model Name:	MacBook Pro
  Model Identifier:	MacBookPro16,1
  Processor Name:	6-Core Intel Core i7
  Processor Speed:	2.6 GHz
  Number of Processors:	1
  Total Number of Cores:	6
  L2 Cache (per Core):	256 KB
  L3 Cache:	12 MB
Software:
  System Version:	macOS 10.15.7 (19H1217)
  Kernel Version:	Darwin 19.6.0

ghost · 2021-09-30T17:00:53Z

Here's timing output for make benchmark-nq-2x. Note that the 2nd run shows generally vastly improved timing. This is important to note. When doing interactive benchmarking, you have to keep this in mind to not get fooled into believing you've made huge improvements when you run benchmarks the 2nd time in same Lisp env. And also it will be interesting to observe whether forthcoming proposed optimizations improve both 1st and 2nd run timings.

+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| STATIC/FULLY-CONNECTED 10  |     2.80 |     0 |        2 |
| STATIC/FULLY-CONNECTED 30  |    21.03 |     0 |        2 |
| STATIC/FULLY-CONNECTED 50  |    56.66 |     0 |        2 |
| STATIC/FULLY-CONNECTED 70  |   107.45 |     0 |        2 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| BELL/FULLY-CONNECTED 10    |     4.89 |     0 |        9 |
| BELL/FULLY-CONNECTED 30    |    30.24 |     0 |       29 |
| BELL/FULLY-CONNECTED 50    |    73.84 |     0 |       49 |
| BELL/FULLY-CONNECTED 70    |   138.98 |     0 |       69 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| QFT/FULLY-CONNECTED 10     |    10.56 |     0 |       37 |
| QFT/FULLY-CONNECTED 30     |   104.07 |     0 |      117 |
| QFT/FULLY-CONNECTED 50     |   395.29 |     0 |      197 |
| QFT/FULLY-CONNECTED 70     |  1246.30 |     0 |      277 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| HADAMARD/FULLY-CONNECTED 10 |     1.80 |     0 |        0 |
| HADAMARD/FULLY-CONNECTED 30 |    18.00 |     0 |        0 |
| HADAMARD/FULLY-CONNECTED 50 |    50.90 |     0 |        0 |
| HADAMARD/FULLY-CONNECTED 70 |   100.89 |     0 |        0 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| STATIC/LINEAR 10           |     0.84 |     0 |        2 |
| STATIC/LINEAR 30           |     1.63 |     0 |        2 |
| STATIC/LINEAR 50           |     2.51 |     0 |        2 |
| STATIC/LINEAR 70           |     3.37 |     0 |        2 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| BELL/LINEAR 10             |     3.41 |     0 |        9 |
| BELL/LINEAR 30             |    12.04 |     0 |       29 |
| BELL/LINEAR 50             |    21.79 |     0 |       49 |
| BELL/LINEAR 70             |    33.54 |     0 |       69 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| QFT/LINEAR 10              |     6.42 |    61 |       79 |
| QFT/LINEAR 30              |    20.04 |   641 |      265 |
| QFT/LINEAR 50              |    94.63 |  1821 |      451 |
| QFT/LINEAR 70              |   498.13 |  3601 |      637 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| HADAMARD/LINEAR 10         |     0.07 |     0 |        0 |
| HADAMARD/LINEAR 30         |     0.31 |     0 |        0 |
| HADAMARD/LINEAR 50         |     0.72 |     0 |        0 |
| HADAMARD/LINEAR 70         |     1.32 |     0 |        0 |
+----------------------------+----------+-------+----------+


+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| STATIC/FULLY-CONNECTED 10  |     0.23 |     0 |        2 |
| STATIC/FULLY-CONNECTED 30  |     1.64 |     0 |        2 |
| STATIC/FULLY-CONNECTED 50  |     4.46 |     0 |        2 |
| STATIC/FULLY-CONNECTED 70  |     8.90 |     0 |        2 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| BELL/FULLY-CONNECTED 10    |     0.52 |     0 |        9 |
| BELL/FULLY-CONNECTED 30    |     3.22 |     0 |       29 |
| BELL/FULLY-CONNECTED 50    |     8.31 |     0 |       49 |
| BELL/FULLY-CONNECTED 70    |    17.01 |     0 |       69 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| QFT/FULLY-CONNECTED 10     |     1.54 |     0 |       37 |
| QFT/FULLY-CONNECTED 30     |    27.34 |     0 |      117 |
| QFT/FULLY-CONNECTED 50     |   176.36 |     0 |      197 |
| QFT/FULLY-CONNECTED 70     |   854.18 |     0 |      277 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| HADAMARD/FULLY-CONNECTED 10 |     0.20 |     0 |        0 |
| HADAMARD/FULLY-CONNECTED 30 |     1.70 |     0 |        0 |
| HADAMARD/FULLY-CONNECTED 50 |     4.81 |     0 |        0 |
| HADAMARD/FULLY-CONNECTED 70 |     9.68 |     0 |        0 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| STATIC/LINEAR 10           |     0.11 |     0 |        2 |
| STATIC/LINEAR 30           |     0.22 |     0 |        2 |
| STATIC/LINEAR 50           |     0.30 |     0 |        2 |
| STATIC/LINEAR 70           |     0.39 |     0 |        2 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| BELL/LINEAR 10             |     0.45 |     0 |        9 |
| BELL/LINEAR 30             |     1.85 |     0 |       29 |
| BELL/LINEAR 50             |     4.16 |     0 |       49 |
| BELL/LINEAR 70             |     7.59 |     0 |       69 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| QFT/LINEAR 10              |     1.74 |    61 |       79 |
| QFT/LINEAR 30              |    19.79 |   641 |      265 |
| QFT/LINEAR 50              |    95.34 |  1821 |      451 |
| QFT/LINEAR 70              |   468.71 |  3601 |      637 |
+----------------------------+----------+-------+----------+
+----------------------------+----------+-------+----------+
|        NAME                | TIME (s) | SWAPS | 2Q DEPTH |
+----------------------------+----------+-------+----------+
| HADAMARD/LINEAR 10         |     0.07 |     0 |        0 |
| HADAMARD/LINEAR 30         |     0.29 |     0 |        0 |
| HADAMARD/LINEAR 50         |     0.67 |     0 |        0 |
| HADAMARD/LINEAR 70         |     1.27 |     0 |        0 |
+----------------------------+----------+-------+----------+

Environment info:

Run on Mon Sep 27 09:11:52 2021
Hardware:
  Model Name:	MacBook Pro
  Model Identifier:	MacBookPro16,1
  Processor Name:	6-Core Intel Core i7
  Processor Speed:	2.6 GHz
  Number of Processors:	1
  Total Number of Cores:	6
  L2 Cache (per Core):	256 KB
  L3 Cache:	12 MB
Software:
  System Version:	macOS 10.15.7 (19H1217)
  Kernel Version:	Darwin 19.6.0

braised-babbage

Overall, this looks good. Just a few small requests re: docs & comments.

Makefile

benchmarking/quilc-mon-prof.lisp

benchmarking/quilc-perf.lisp

stylewarning · 2021-09-30T18:41:13Z

A violent error occurred when compressing a subsequence.
The offending subsequence is:
RZ(-1.5707963267948966) 2
RX(1.5707963267948966) 2
RZ(1.1831992743510398) 2
RZ(1.3624223294733835) 3
RX(1.5707963267948966) 3
RZ(2.0589147502832232) 3
RX(-1.5707963267948966) 3
RZ(2.731137214780616) 3
RX(-1.5707963267948966) 3
RZ(-5.105088062083414) 3
RZ(3.141592653589793) 2
RZ(-1.1549785415648206) 4
RX(1.5707963267948966) 4
RZ(1.8395366515564946) 4
RX(-1.5707963267948966) 4
RZ(-1.6213192609014024) 4
CPHASE(1.1780972450961729) 4 3
RZ(-4.71238898038469) 4
RX(1.5707963267948966) 4
RZ(3.141592653589793) 4
RX(-1.5707963267948966) 4
RZ(-1.5707963267948966) 4
RZ(-1.5707963267948966) 3
RX(1.5707963267948966) 3
RZ(1.5707963267948966) 3
CPHASE(3.141592653589793) 3 4
RZ(3.141592653589793) 4
RZ(3.141592653589793) 3
RX(-1.5707963267948966) 3
RZ(-1.5707963267948966) 3
RZ(1.5707963267948966) 2
RX(1.5707963267948966) 2
RZ(1.5707963267948966) 2
The current compression context is:
#S(CL-QUIL::COMPILATION-CONTEXT :AQVM #S(CL-QUIL::ANTISOCIAL-QVM :WFS #(#(#C(1.0d0 0.0d0) #C(0.0d0 0.0d0)) #(#C(0.9492372568027305d0 -0.3145610120431765d0) #C(3.1916143723912226d-13 8.47528971714898d-13)) #(#C(-0.7056731905546375d0 -0.04500386797197184d0) #C(-0.7056731905547174d0 -0.04500386797071089d0)) #(#C(0.7368889545376084d0 -0.14544168966499027d0) #C(-0.2588955525596931d0 -0.1575400119864813d0) #C(-0.008257323146726144d0 -0.5438431303825365d0) #C(0.14575040233277808d0 -0.1640698249099481d0)) #(#C(0.7368889545376084d0 -0.14544168966499027d0) #C(-0.2588955525596931d0 -0.1575400119864813d0) #C(-0.008257323146726144d0 -0.5438431303825365d0) #C(0.14575040233277808d0 -0.1640698249099481d0))) :INTERNAL-INDICES #(0 0 0 0 1)) :CHIP-SPECIFICATION #<CHIP-SPECIFICATION of 5:4 objects>)
Unhandled SIMPLE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING
                                    {10005984F3}>:
  When attempting to read the slot's value (slot-value), the slot
  COMPILATION-TOLERANCE is missing from the object
  #<CL-QUIL::STATE-PREP-COMPRESSION-TOLERANCE-ERROR {100407B0A3}>.

Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {10005984F3}>
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #<SIMPLE-ERROR "~@<When attempting to ~A, the slot ~S is missing from the object ~S.~@:>" {1004249503}> #<unused argument> :QUIT T)
1: (SB-DEBUG::RUN-HOOK SB-EXT:*INVOKE-DEBUGGER-HOOK* #<SIMPLE-ERROR "~@<When attempting to ~A, the slot ~S is missing from the object ~S.~@:>" {1004249503}>)
2: (INVOKE-DEBUGGER #<SIMPLE-ERROR "~@<When attempting to ~A, the slot ~S is missing from the object ~S.~@:>" {1004249503}>)
3: (ERROR "~@<When attempting to ~A, the slot ~S is missing from the object ~S.~@:>" "read the slot's value (slot-value)" COMPILATION-TOLERANCE #<CL-QUIL::STATE-PREP-COMPRESSION-TOLERANCE-ERROR {100407B0A3}>)
4: ((:METHOD SLOT-MISSING (T T T T)) #<unused argument> #<CL-QUIL::STATE-PREP-COMPRESSION-TOLERANCE-ERROR {100407B0A3}> COMPILATION-TOLERANCE SLOT-VALUE NIL) [fast-method]
5: ((LAMBDA (SB-PCL::OBJECT) :IN SB-PCL::SLOT-MISSING-INFO) #<CL-QUIL::STATE-PREP-COMPRESSION-TOLERANCE-ERROR {100407B0A3}>)
6: ((FLET "H0" :IN TEST-COMPILER-HOOK) #<CL-QUIL::STATE-PREP-COMPRESSION-TOLERANCE-ERROR {100407B0A3}>)
7: (SB-KERNEL::%SIGNAL #<CL-QUIL::STATE-PREP-COMPRESSION-TOLERANCE-ERROR {100407B0A3}>)
8: (CERROR "continue with possibly incorrect compilation" CL-QUIL::STATE-PREP-COMPRESSION-TOLERANCE-ERROR :COMPILATION-TOLERANCE 5.0d-11 :COMPILATION-PRECISION 5.820299797676398d-11)
9: (CL-QUIL::COMPRESS-INSTRUCTIONS-IN-CONTEXT (#<RZ(-#1=pi/2) 2> #<RX(#1#/2) 2> #<RZ(1.1831992743510398) 2> #<RZ(1.3624223294733835) 3> #<RX(#1#/2) 3> #<RZ(2.0589147502832232) 3> #<RX(-#1#/2) 3> #<RZ(2.731137214780616) 3> #<RX(-#1#/2) 3> #<RZ(-13*#1#/8) 3> #<RZ(#1#) 2> #<RZ(-1.1549785415648206) 4> ...) #<CL-QUIL::CHIP-SPECIFICATION of 5:4 objects> #S(CL-QUIL::COMPILATION-CONTEXT :AQVM #S(CL-QUIL::ANTISOCIAL-QVM :WFS #(#(#C(1.0d0 0.0d0) #C(0.0d0 0.0d0)) #(#C(0.9492372568027305d0 -0.3145610120431765d0) #C(3.1916143723912226d-13 8.47528971714898d-13)) #(#C(-0.7056731905546375d0 -0.04500386797197184d0) #C(-0.7056731905547174d0 -0.04500386797071089d0)) #1=#(#C(0.7368889545376084d0 -0.14544168966499027d0) #C(-0.2588955525596931d0 -0.1575400119864813d0) #C(-0.008257323146726144d0 -0.5438431303825365d0) #C(0.14575040233277808d0 -0.1640698249099481d0)) #1#) :INTERNAL-INDICES #(0 0 0 0 1)) :CHIP-SPECIFICATION #<CL-QUIL::CHIP-SPECIFICATION of 5:4 objects>))
; 
; compilation unit aborted
;   caught 1 fatal ERROR condition
10: (CL-QUIL::COMPRESS-INSTRUCTIONS-WITH-POSSIBLY-UNKNOWN-PARAMS (#<RZ(-#1=pi/2) 2> #<RX(#1#/2) 2> #<RZ(1.1831992743510398) 2> #<RZ(1.3624223294733835) 3> #<RX(#1#/2) 3> #<RZ(2.0589147502832232) 3> #<RX(-#1#/2) 3> #<RZ(1.1603408879857193) 3> #<CPHASE(#1#) 3 2> #<RX(#1#/2) 3> #<RZ(#1#) 2> #<RX(#1#/2) 2> ...) #<CL-QUIL::CHIP-SPECIFICATION of 5:4 objects> #S(CL-QUIL::COMPILATION-CONTEXT :AQVM #S(CL-QUIL::ANTISOCIAL-QVM :WFS #(#(#C(1.0d0 0.0d0) #C(0.0d0 0.0d0)) #(#C(0.9492372568027305d0 -0.3145610120431765d0) #C(3.1916143723912226d-13 8.47528971714898d-13)) #(#C(-0.7056731905546375d0 -0.04500386797197184d0) #C(-0.7056731905547174d0 -0.04500386797071089d0)) #1=#(#C(0.7368889545376084d0 -0.14544168966499027d0) #C(-0.2588955525596931d0 -0.1575400119864813d0) #C(-0.008257323146726144d0 -0.5438431303825365d0) #C(0.14575040233277808d0 -0.1640698249099481d0)) #1#) :INTERNAL-INDICES #(0 0 0 0 1)) :CHIP-SPECIFICATION #<CL-QUIL::CHIP-SPECIFICATION of 5:4 objects>) NIL)
11: ((LABELS CL-QUIL::TRANSITION-GOVERNOR-STATE :IN CL-QUIL::COMPRESS-INSTRUCTIONS) :GLOBAL :GLOBAL :FLUSHING NIL)
12: ((LABELS CL-QUIL::TRANSITION-GOVERNOR-STATE :IN CL-QUIL::COMPRESS-INSTRUCTIONS) 1 1 :QUEUEING NIL)
13: ((LABELS CL-QUIL::PROCESS-INSTRUCTION :IN CL-QUIL::COMPRESS-INSTRUCTIONS) #<CPHASE(pi) 2 1> 1 1)
14: (CL-QUIL::COMPRESS-INSTRUCTIONS (#<RZ(#1=pi) 4> #<RZ(15*#1#/16) 3> #<RZ(-#1#) 2> #<RX(#1#/2) 4> #<RX(#1#/2) 3> #<RX(#1#/2) 2> #<RZ(#1#) 4> #<RZ(1.3781260374774809) 3> #<RZ(0.0) 2> #<RX(-#1#/2) 4> #<RX(-#1#/2) 3> #<RX(-#1#/2) 2> ...) #<CL-QUIL::CHIP-SPECIFICATION of 5:4 objects> :PROTOQUIL T)
15: ((LABELS CL-QUIL::PROCESS-BLOCK :IN COMPILER-HOOK) #<CL-QUIL::BASIC-BLOCK ENTRY-BLK-5100 len:6 in:0 out:uncond {10030551F3}> NIL)
16: (COMPILER-HOOK #<PARSED-PROGRAM of 6 instructions {1001CF60E3}> #<CL-QUIL::CHIP-SPECIFICATION of 5:4 objects> :PROTOQUIL T :REWIRING-TYPE NIL :DESTRUCTIVE NIL)
17: (COMPARE-COMPILED #P"/src/quilc/tests/compiler-hook-test-files/modifiers-galore.quil" :CPHASE)
18: ((LABELS TEST-COMPILER-HOOK :IN TEST-COMPILER-HOOK))
19: ((LABELS FIASCO::RUN-TEST-BODY :IN FIASCO::RUN-TEST-BODY-IN-HANDLERS))
20: (FIASCO::CALL-WITH-TEST-HANDLERS #<CLOSURE (LAMBDA NIL :IN FIASCO::RUN-TEST-BODY-IN-HANDLERS) {100445FC8B}>)
21: (FIASCO::PRETTY-RUN-TEST #<test TEST-COMPILER-HOOK> #<CLOSURE (LABELS TEST-COMPILER-HOOK :IN TEST-COMPILER-HOOK) {100445EACB}>)
22: ((LABELS #:BODY-SYM0 :IN TEST-COMPILER-HOOK))
23: (TEST-COMPILER-HOOK :PRINT-STATS NIL)
24: ((LABELS FIASCO-SUITES::CL-QUIL-TESTS :IN FIASCO-SUITES::CL-QUIL-TESTS))
25: ((LABELS FIASCO::RUN-TEST-BODY :IN FIASCO::RUN-TEST-BODY-IN-HANDLERS))
26: (FIASCO::CALL-WITH-TEST-HANDLERS #<CLOSURE (LAMBDA NIL :IN FIASCO::RUN-TEST-BODY-IN-HANDLERS) {10064EA04B}>)
27: (FIASCO::PRETTY-RUN-TEST #<test FIASCO-SUITES::CL-QUIL-TESTS :tests 224> #<FUNCTION (LABELS FIASCO-SUITES::CL-QUIL-TESTS :IN FIASCO-SUITES::CL-QUIL-TESTS) {54106FDB}>)
28: ((LABELS #:BODY-SYM1 :IN FIASCO-SUITES::CL-QUIL-TESTS))
29: (FIASCO-SUITES::CL-QUIL-TESTS)
30: (RUN-SUITE-TESTS #<test FIASCO-SUITES::CL-QUIL-TESTS :tests 224> :VERBOSE NIL :STREAM #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDOUT* {10000076F3}> :INTERACTIVE T)
31: (RUN-TESTS :CL-QUIL-TESTS :DESCRIBE-FAILURES T :VERBOSE NIL :STREAM #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDOUT* {10000076F3}> :INTERACTIVE T)
32: (RUN-CL-QUIL-TESTS :VERBOSE NIL :HEADLESS NIL)
33: ((SB-PCL::EMF ASDF/ACTION:PERFORM) #<unused argument> #<unused argument> #<ASDF/LISP-ACTION:TEST-OP > #<ASDF/SYSTEM:SYSTEM "cl-quil-tests">)
34: ((LAMBDA NIL :IN ASDF/ACTION:CALL-WHILE-VISITING-ACTION))
35: ((:METHOD ASDF/ACTION:PERFORM-WITH-RESTARTS :AROUND (T T)) #<ASDF/LISP-ACTION:TEST-OP > #<ASDF/SYSTEM:SYSTEM "cl-quil-tests">) [fast-method]
36: ((:METHOD ASDF/PLAN:PERFORM-PLAN (T)) #<ASDF/PLAN:SEQUENTIAL-PLAN {1002265DB3}>) [fast-method]
37: ((FLET SB-C::WITH-IT :IN SB-C::%WITH-COMPILATION-UNIT))
38: ((:METHOD ASDF/PLAN:PERFORM-PLAN :AROUND (T)) #<ASDF/PLAN:SEQUENTIAL-PLAN {1002265DB3}>) [fast-method]
39: ((:METHOD ASDF/OPERATE:OPERATE (ASDF/OPERATION:OPERATION ASDF/COMPONENT:COMPONENT)) #<ASDF/LISP-ACTION:TEST-OP > #<ASDF/SYSTEM:SYSTEM "cl-quil"> :PLAN-CLASS NIL :PLAN-OPTIONS NIL) [fast-method]
40: ((SB-PCL::EMF ASDF/OPERATE:OPERATE) #<unused argument> #<unused argument> #<ASDF/LISP-ACTION:TEST-OP > #<ASDF/SYSTEM:SYSTEM "cl-quil">)
41: ((LAMBDA NIL :IN ASDF/OPERATE:OPERATE))
42: ((:METHOD ASDF/OPERATE:OPERATE :AROUND (T T)) #<ASDF/LISP-ACTION:TEST-OP > #<ASDF/SYSTEM:SYSTEM "cl-quil">) [fast-method]
43: ((SB-PCL::EMF ASDF/OPERATE:OPERATE) #<unused argument> #<unused argument> ASDF/LISP-ACTION:TEST-OP :CL-QUIL)
44: ((LAMBDA NIL :IN ASDF/OPERATE:OPERATE))
45: ((:METHOD ASDF/OPERATE:OPERATE :AROUND (T T)) ASDF/LISP-ACTION:TEST-OP :CL-QUIL) [fast-method]
46: (ASDF/SESSION:CALL-WITH-ASDF-SESSION #<CLOSURE (LAMBDA NIL :IN ASDF/OPERATE:OPERATE) {10021A398B}> :OVERRIDE T :KEY NIL :OVERRIDE-CACHE T :OVERRIDE-FORCING NIL)
47: ((LAMBDA NIL :IN ASDF/OPERATE:OPERATE))
48: (ASDF/SESSION:CALL-WITH-ASDF-SESSION #<CLOSURE (LAMBDA NIL :IN ASDF/OPERATE:OPERATE) {100207227B}> :OVERRIDE NIL :KEY NIL :OVERRIDE-CACHE NIL :OVERRIDE-FORCING NIL)
49: ((:METHOD ASDF/OPERATE:OPERATE :AROUND (T T)) ASDF/LISP-ACTION:TEST-OP :CL-QUIL) [fast-method]
50: (ASDF/OPERATE:TEST-SYSTEM :CL-QUIL)
51: (SB-INT:SIMPLE-EVAL-IN-LEXENV (ASDF/OPERATE:TEST-SYSTEM :CL-QUIL) #<NULL-LEXENV>)
52: (EVAL (ASDF/OPERATE:TEST-SYSTEM :CL-QUIL))
53: (SB-IMPL::PROCESS-EVAL/LOAD-OPTIONS ((:LOAD . "/root/quicklisp/setup.lisp") (:EVAL . "(push (truename \".\") asdf:*central-registry*)") (:EVAL . "(push (truename \"../\") ql:*local-project-directories*)") (:EVAL . "(ql:quickload :cl-quil-tests)") (:EVAL . "(asdf:test-system :cl-quil)") (:QUIT)))
54: (SB-IMPL::TOPLEVEL-INIT)
55: ((FLET SB-UNIX::BODY :IN SB-EXT:SAVE-LISP-AND-DIE))
56: ((FLET "WITHOUT-INTERRUPTS-BODY-14" :IN SB-EXT:SAVE-LISP-AND-DIE))
57: ((LABELS SB-IMPL::RESTART-LISP :IN SB-EXT:SAVE-LISP-AND-DIE))

unhandled condition in --disable-debugger mode, quitting
make: *** [test-cl-quil] Error 1
Makefile:131: recipe for target 'test-cl-quil' failed

ecpeterson · 2021-10-01T01:33:29Z

nice: the actual error is a missing slot, and not some bug in state prep compilation. phew!

ghost · 2021-10-01T01:54:39Z

nice: the actual error is a missing slot, and not some bug in state prep compilation. phew!

I think I see it: Quil package error condition quil::state-prep-compression-tolerance-error's slot names need to be package-qualified in %with-loose-state-prep-compression macro, which wraps code in test-compiler-hook.
Note: this is just the second-level error. But not exactly sure why first error happens all of a sudden and so consistently but only in this PR branch, apparently.

stylewarning · 2021-10-01T03:56:25Z

@ecpeterson Unfortunately if you look at line 3 and 8 in the stack trace, you see two errors: (1) the violent numerical error and (2) an error printing the error. sad_trombone.mp3

ecpeterson · 2021-10-01T04:01:34Z

😫

ghost · 2021-10-05T07:05:55Z

@kilimanjaro I addressed your requests and made some additional/related fixes and improvements in commit 4e4bef7. I marked each one as resolved -- not completely sure that's the right thing, but feel free to unresolve any or all.

ghost · 2021-10-05T23:42:01Z

Please do not merge it until I've had a chance to rebase and squash manually. I added "WIP:" to title just to prevent this going in as is. Hope this is OK with everyone.

stylewarning · 2021-10-06T04:36:28Z

You can change it to an unmergeable "draft PR" by clicking the right button somewhere on the page.

braised-babbage

Just two random stylistic things, but other than that it looks good to me!

benchmarking/quilc-mon-prof.lisp

ghost · 2021-10-15T04:23:29Z

EDIT: never mind, since I just did a push of a couple of commits (c6a6de3 and c1bfa5a), which kicked off the CI again, and this time it passed. 🤷

Hey folks. In draft PR #744 I just have the same commits as in this PR but it's rebased with master, and in particular takes in the commit fixing the missing slot secondary error (d176882). That PR passed all tests. Therefore, I suggest rerunning the tests (manually kicking off) and/or having me force push on this branch the rebase with master. What do you think @notmgsk or @stylewarning ?

stylewarning · 2021-10-19T20:13:03Z

@mhdavid-hrl I'm ready to merge. All good?

ghost · 2021-10-19T20:46:24Z

@mhdavid-hrl I'm ready to merge. All good?

Almost, this really needs a squash and fixup of commit messages.

This is part of issue "Investigate performance at 72-150 qubits #565". new modules: benchmarking/quilc-perf.lisp - primarily exports benchmark-nq function to get timings as nQ increases for various chip and program types. This also introduces a "chip preparation" step before running any benchmarks. This removes any overhead from running the first compile on a chip out of the first benchmark run. Currently, there is significant but not tremendous amount of initial overhead, but a PR under consideration might add significant additional preparation overhead, and these should not be be charged to the first run. benchmarking/quilc-mon-prof.lisp - builds on quilc-perf.lisp and is for monitoring and profiling, as opposed to simple benchmarks, and is a WIP of sorts in that, while it's been useful to run for probing and experimenting in a REPL, we ;; so far lack really good theory of operation and associated modes of ;; running. We hope with time to get there. Also, note that some of this is SBCL-only. Makefile - add targets to run benchmark-nq a couple of ways: `make benchmark-nq` calls benchmark-nq one time, and `make benchmark-nq-2x` calls it twice. The idea is to capture the sometimes significant jump in performance observed between a first and second run within the same Lisp environment, due to the effects of "warming", caching, and the like. cl-quil-benchmarking.asd - add new module quilc-perf and depend on 3 additional systems: metering, qvm-app, sb-sprof

ghost · 2021-10-19T22:16:35Z

Done with rebase/squash/fix commit log, and all checks have passed. Good to merge now.

stylewarning requested review from notmgsk, stylewarning and braised-babbage September 24, 2021 20:02

braised-babbage suggested changes Sep 30, 2021

View reviewed changes

ghost mentioned this pull request Oct 5, 2021

compiler hook test gets 2nd-level error: COMPILATION-TOLERANCE slot missing #736

Closed

stylewarning requested a review from braised-babbage October 5, 2021 20:08

braised-babbage reviewed Oct 8, 2021

View reviewed changes

benchmarking/quilc-mon-prof.lisp Outdated Show resolved Hide resolved

benchmarking/quilc-mon-prof.lisp Outdated Show resolved Hide resolved

This was referenced Oct 12, 2021

reimplement logical-scheduler walker more efficiently #741

Merged

draft: Draft benchmark nq and rebase #744

Closed

stylewarning merged commit 93b3ba3 into quil-lang:master Oct 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add cl-quil-benchmarking system quilc-perf for perf. benchmarking #733

add cl-quil-benchmarking system quilc-perf for perf. benchmarking #733

ghost commented Sep 24, 2021

ghost commented Sep 24, 2021

ghost commented Sep 30, 2021 •

edited by ghost

Loading

braised-babbage left a comment

stylewarning commented Sep 30, 2021

ecpeterson commented Oct 1, 2021

ghost commented Oct 1, 2021 •

edited by ghost

Loading

stylewarning commented Oct 1, 2021

ecpeterson commented Oct 1, 2021

ghost commented Oct 5, 2021

ghost commented Oct 5, 2021

stylewarning commented Oct 6, 2021

braised-babbage left a comment

ghost commented Oct 15, 2021 •

edited by ghost

Loading

stylewarning commented Oct 19, 2021

ghost commented Oct 19, 2021

ghost commented Oct 19, 2021

add cl-quil-benchmarking system quilc-perf for perf. benchmarking #733

add cl-quil-benchmarking system quilc-perf for perf. benchmarking #733

Conversation

ghost commented Sep 24, 2021

ghost commented Sep 24, 2021

ghost commented Sep 30, 2021 • edited by ghost Loading

braised-babbage left a comment

Choose a reason for hiding this comment

stylewarning commented Sep 30, 2021

ecpeterson commented Oct 1, 2021

ghost commented Oct 1, 2021 • edited by ghost Loading

stylewarning commented Oct 1, 2021

ecpeterson commented Oct 1, 2021

ghost commented Oct 5, 2021

ghost commented Oct 5, 2021

stylewarning commented Oct 6, 2021

braised-babbage left a comment

Choose a reason for hiding this comment

ghost commented Oct 15, 2021 • edited by ghost Loading

stylewarning commented Oct 19, 2021

ghost commented Oct 19, 2021

ghost commented Oct 19, 2021

ghost commented Sep 30, 2021 •

edited by ghost

Loading

ghost commented Oct 1, 2021 •

edited by ghost

Loading

ghost commented Oct 15, 2021 •

edited by ghost

Loading