From 31a26c9f159894fb4101bdba22d4260bc2b0fe3d Mon Sep 17 00:00:00 2001 From: Daniel Egger Date: Fri, 24 Feb 2023 23:01:11 +0100 Subject: [PATCH 01/12] * Added paper --- .github/workflows/draft-pdf.yml | 23 +++ docs/paper/framework.pdf | Bin 0 -> 22650 bytes docs/paper/paper.bib | 239 ++++++++++++++++++++++++++++++++ docs/paper/paper.md | 132 ++++++++++++++++++ docs/paper/qv.pdf | Bin 0 -> 21877 bytes 5 files changed, 394 insertions(+) create mode 100644 .github/workflows/draft-pdf.yml create mode 100644 docs/paper/framework.pdf create mode 100644 docs/paper/paper.bib create mode 100644 docs/paper/paper.md create mode 100644 docs/paper/qv.pdf diff --git a/.github/workflows/draft-pdf.yml b/.github/workflows/draft-pdf.yml new file mode 100644 index 0000000000..02e8c8b68f --- /dev/null +++ b/.github/workflows/draft-pdf.yml @@ -0,0 +1,23 @@ +on: [push] + +jobs: + paper: + runs-on: ubuntu-latest + name: Paper Draft + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Build draft PDF + uses: openjournals/openjournals-draft-action@master + with: + journal: joss + # This should be the path to the paper within your repo. + paper-path: docs/paper/paper.md + - name: Upload + uses: actions/upload-artifact@v1 + with: + name: paper + # This is the output path where Pandoc will write the compiled + # PDF. Note, this should be the same directory as the input + # paper.md + path: docs/paper/paper.pdf diff --git a/docs/paper/framework.pdf b/docs/paper/framework.pdf new file mode 100644 index 0000000000000000000000000000000000000000..74c87e10295b98286f799ea715da5b7b9610c674 GIT binary patch literal 22650 zcmdVC1z1$ww+D=L2@;BgFtju?14DN!ARQ`54LNkTbV^DqNC_yCQqtYsAR#TSq=@j% zV7}`A-tW85z3=mU!#SLD*4byDz1MH;wb$8e?M<&FDa{6C=fl zSQ7(rYx@g#NzH~QBAyv%EYh z7VNb(K%Pz9(o@n@@nx5{C29imszHNFaI3&AKCGr;l(!y_c6Z&)hH{P?c1?nJ8+WLh zS2@m@n!Y?F5N@KK$KulvT?Cxny=fx}IIxx{o4^+I463l6cXA>1N3$SKs&LP>PVTIK zxhH;%lyOBJeUgi#~VKl~o3Zo>G}$eOX*xpV06l9#Lqrv!o$= z9(jQQd7UzQK_8_Z zU~R03^NgX^uFzR~C|vSPa8ipH*}j%k#UoI1ULilFrm~k)SR-^akJ5=|->(mN!ZQQB z(wc1KJ`ZaouGX1ck*I1hE+<;!EeYg*Q)(axhJnW)>nEr^P@(utc;|)t-PD|&n$6>; zsM{0^TtRo)UNq&201wxTV{78jbtDa?10N?3tnQX&!5*%%y!+&aHdKbaWHtEI)Oj*Q z;!}NmzkQU1Ya)-m0p$i{89P3Hz1H;3HGTf2hq8qa42XlriSFEO$h`7;M=!z5C91Jehvlz5yeB~2IK<3w{Pd%0N_R2K+quo z`1Xy?0|0)1Cy*Ba-@eiL0Ko6>1#tr48-flmJ$(Dd1;i^6+(6V*&D919;84;xfdc3O zzzeP@>DxiA9N-m)KX^e0 zz*9*=oy-iODl*~-;x|N8p!U{|c7{-UzywE21F0a!ifaj=QVwXpMp{9vmE3<^o|;=`ISL;`wW^ap_iQlkt@&8WSV~ zOA@nq*==8Pr1$tj6Vs(W07FchKoVenTi;NqKlmU&M0oRcHlszB1vD#wd-#=#>B_FV z@a$WHl3Owu8kqtwhluwsKyJ_vu|d@6D?cxo z@IPoh(s1$kMHGzLIN9JL`W3gYi2jsPgmC|k-hbUm{BNT7Ret}0UMLW+iSr8TXZP0l z^j+uj+fV5Z)9y>; zn{BT)h@Gq+o7)cYk3G+h*$%GgxmO~2-5~b;yZi)~PPQd>jM(Vw{C~mY7i9+Ig#5tc zXD|A{h6fOV$AyYPWC^eSPdpF`=65_Ul+MM=|JG^$Z{l&G>i%H{NQKwJ@xb+W7737X zmo*m<;i3gxgJi*i({HoTg`y_8xFe&yWph4JJVN>aBpwc5aHL?%qAamAQ1M{rmG`~v zr-OFG^R?D3vG-2rFZez`mh9oA`!F;*w~pPs*@DG9M%wops=3w|%`hg)qmfOLqOvHi z>BKC=*Bf#ua;D?uvUpplDZL46cCdmz`!AS%XBV6aK(3!{=%-ZwuVMCIv>`-C@Ec~} ziw@ym!R(?N`Uhs}T`%#PM6U$si`)U)X>2+e?^wv0+t6)bB8!nrViBW8_FqG*Ke4l; z1Wh+e5Dt3NYBM#CdFa zN2`M3 z+9GT;k;Z8mO`EAVH!t5!mNmdFne?83c_t`aw{h(n`xne0JjUZokf|FRe}U+`;evn& z#PuDbi&B5(*}nqO&u;L?#|!wUx$}Rl2*E#$oL|g}VuT_@9hiCWZjdx)N^zK%B{<+R z6S;w??5pe7r<%?U8Mfn{E0A-5JaJfjAxSF>iLWE1Z0^}qg4qQq_=B_GdWjvjofEU< zT{TvJd_P7bN6)*FQzjyOo3y|IH)9`HIk6n@h*B{#QY3DcdYIDc3vLb`$uTksJ5}9Z zQ2O32U7*DC10_Vk|3#F(%gQgGnBX5B+yAxk2mWfP{KI@`N0<+|qJuLzyNS3?mhD)b zp(EMkF+43u5(3ncRrQke==%vrbAU}~B7}Hs_SFbVucGeNqsuwf^M^k651pL$faqNW zUo=zfU(py(#BJrcG`)_AdS;Msqi~xcZof-yh<47vhi@MFL0ANt$zO2!ZVn@G0sj0X z`fgl&{RHCpKep;FzHxq6aQ~Va0se~_5%dxdSqvA+vp&O4^YysXTs*3ML{{CQJ0R)8RK(uh=r;}$9|N!JnKJs{Jm-kMJhS{3pHrKX`D#*@b1yW* zyp+a;vQqGRB*%Sh&gx7^TJL-d!tC^luh;PLcXy zn>-&+^Z3#uF*55vfw`zYf+ZKtIZYiJr90v}%nat?(0q`x}$E=j+)ZADARlImzc_li5AxcjjQdE8N45TlFG+K8-0 z32eJ>t;PFALKZkGPNp#L8dBeuL^B~lW0z%m^?h+J*Pz@e?w*gA8QeRfG#@a?xV&b0 zN;$*DV> zUe6|q-&d>j<(2z{x@ioUc0JB#7}Vnn%#Wg5<+}w395hYdt+ov#&+5M3p^Ja1;cOC~ zeI6ID{}7ToQ*srl?~CGaa#TZ;XSPLU3msYfO?nS>i}?vIii`=4I>Q^42Shgnp1#L1 z45fPW7nuHG6Yzf5au+547h(FTRDU+6f6XS~`io7_dGQI0IzMyg-5_Pm)F-p2#c^5_ zQ6d? zAwcyNKtaYkOUvlE%>dq!3r=S1>RPXL<=F~=01qYvxbx-GR|b}IK#wghgx$9cU^`Qn|ol_=SNj<($@(DMVLBl*7f6fkUnWX^HaD zz?D+E&MTbf@6URB$Fut!Yk_U$nNaAsJ|>WRt+_8|f!|OQZIxsbXKkKMtB*AM(dU=a zdrTqz!SQ!ff=ydWM0xlS}7t?MbL)Kc#Irzl_CF6=q?z9zz6{y*H1n8lb8R-7wm;k;H#wkgzjH6D!Bh*RP?0d zHN(GPm9`5x1A3~<-AMtYx-OM;f8)PS~Ll}p*{Lv-iNPv zMv&jfc>4~=^gV^vU(os0m;nCJeisG)_Zk-W&kyCF==}G_#8(Fg5WvF)AF==S!*fg9 z)rNf4(R;3%Zh}IYRiflj3oOI>CzR!R-76->BZM(6f8nRq?*H=mNP1A)O?+l$>EF|;NdYRF2nIry1eH}ylg4YBTx@L=hy+k zQJYy`Dv*ljj%sH-KhJ)i{osBZfTVPpL=>sNt8;mH?!kE%=EG5ZQL4MHn*{hmdDh59 z4;o&c;H3%Msg&+6a}^XKM@3TdeK>x%lyawUq3Sl$LRDr^JbtnSiQ*+*{-}8VP}D7Q zU;J@Kn=KKykJ0v?-npMN?~4R%*fT1=V%_m_mbuoDXLx#tJxxFX?0_1h6FFy1iw16Fw)lz<(4iEvPHck7U9ZKVSUfZEA3Vvf@Mrl4j?3#-#G_pVx z`UGoHzM+#>Q~1RhC{(Swkg&7s?Rv1OBFj^^`en#!0$ruoMc6^_=h1KBU8;~m#QKYzpobF{> z(=N=>a3waDA}QLoh=rcVotdNqth^FFJ>pIAvW%~dnQof3vUpdWt2UWjJEN&TmwSgJ zDK)=7dW0?VxzBmxfU|m1PT@5r6t{_SPcOi7py`o*>(gA0EzJO>P+VuMgK(t>4A~u+ zG>N2Qeq&BPWUtufwfd=)BDq*{l59q7I(xBzgEE947?|(ks(nPG&oqk2ag&n~{YaV{ zrj@(~Q-n??g(_t06zr>QYNaqA;Ww+QHupKz-#nSlGk-2b)z|uUhnxf#*9~ZTLJA=3wIuVc)|VwCBMFa_1!d=t}~pG zeC$Cs$eW@{-gYuG=wthWf8cGLg*E#rrH#n6PrFDu4-h@P$Zzs3s`2D~`V{hsQrx(q z%3G+kwAaIi00iKGNfOZ`S^5QyeY{5y{l|R*FbKvOx;>v;U89PH*?ds@&fEAcYOeX& zpm}HQ4u9&8UWd5`{x5EE)Ytp)Ha26XcJa>pzi^wxO2qmy&(tc0k~6w+Gd@XOG#e&J znV+Ey>H;bWH!J0qxY8pGKe1H9;#(VAx2*w>~8x@kQp4M3<(P zR=c*gDt$M*x-=&8G+w&lLwN@%EP$Rn=Z*g}G+2R3y5O z%&wbc$nYDbd#gr2)~i958!&0W{jA-fU0F<-m~Kv*et5~Q#6N3=`u@Fs$m{Dp(v;~~ z4*+J5W3b{~mDH5Qea_j5Qy=m57@G7^grRp7=dWpjZSrxgp8@Z((Z-uw z(>IoqT|IrC8HwV6?VV%kaF`!Ka*v_nu!y5UUBOrShm3{g=%eO~YUU>rRKoU^IUG|Q zZ7rUYUu3FrM65zTv$e~d3!-TS6qSWe@+(jF_EV4o|I+#*w3 z8>SVUo1K0)Xa??6dCw|?+KdYSk;&Zw6w)zvCM+>zbdF%Xrab*6}B>U#{_D%A1)M5&gP533H1ALTuXZz)P}oQX=>cw-pwsQE5muz(uD~l-0?k?(e42sQ{M3fP=ccyhvTxlr z^Xn|7?YWlh+R*(1tb1$X*Eeq09@^EEg3my`B7!&M<)pU5ut{+4)S+Cp53hs-oJRz07_PVl+Xymvj33 z!{D-wW^-sACV7xboh8A&+ig4|HAm@st9r%K*Lc0O^ICR8O*A{!rtXG^-;0*&99@2x zUOB@(PKpO%IKe{M_0@P>#I^0__NnVA{@leRVwTl^HO;%s=8>uMyMrq-(_c`R4UWH@ zYMin^7Hn^!GPI1-)oElYsa+rV!7|WGqjtNM#rROqUYj-H zHjN=Nn8TBGl7QAa=vL>AMyyZCEPMo$wFVQ)GfB4&YQmUyNTQ1??ROIMxWT-V>3WEgUcu(*6GU$?;R6AXgL(~+G}ueuri^B%_P;-A4XK^rWHCDWEz(DuWqDl zT~ADzx0{;w6X^H<(aTrb7?k( z_t9Z`#^bZmlYx`~&2F;6q|{fIxu=wVZ_`u{Cr;ohB`uFKb zaJ&^W0_s<->9OaerIG!4Xw_sG0a$)feuW^x67uJKV1c`}=Whh@bi4t=%66#nZS@8> zfG3H}tcH)VtOju-kWCWxhjc`dwQF#(*xk4zNWDM~D0kY5j)8+gk840#T+7II8wsBM z8PS7x`I)GugEvDV5%ZFAMiYTM36~hhtOzU2wI2<2ggv&2ELEBLv=>Pgi8{5=ST1R3 zz!ihX#OB%hK8{u*H5(9?2#9%U+h2VtB zs}jW=Uq9ub>=#Ul2U9`fu=Fwht@4u(`PGwTd#&O|v>T3mSLvlu&F zu0Tmu>Bu5D_Rh{RJRmLeQ^)L`t=9Su|Cjd8?UNBRPYVRBkvnc`A{{>0!e{0WFeH+W zdKkA!R3vv81+|Mf<#$c4g9kw&rzYlf+jnkEzg^PXph;qYE20{^+3;0x_8$J(6_2*d zAL&0rKVCi6RjVtUKO(DYo6|mjAW~UtYD(h%`I%Dwy#kxMoiAnAwHWgC5~&BL9^#>C zVH`~4tPK>R-On2oSKs`?dbmx@c6grF6?qx_fe;o%94?Okb0@X6`~E!O@nz&-lF=erFUBv zN%YE*e5F7cn%2wEG8_{VZM>3onF7yu?xNC)-RC z32aYt-I+Q>Gcj{B)lbLylzW}(>T2nmhZGSk6cP4#9rJj)SrPJXWB#I6UPq@>0s5;` zhwspG#y3c{tCLe6RP%4{v$M{tC&y&*rivJA8jWT2BwtZ<03`_RSxi5*C@OlxTu8Sw z6)>v4%DkSQKmKn08nVwT|1IoGY>m9wy|f!k;t^?Y0IpBN0j~5MT~(an{RN@&+euVn zaCY6r<{-)JW77-uq6_}kbYuz?-lx-A`v%yC(L%68LZMJBIM-!v~m*pf(ttVr0=Ifr5*0Ucfl;o=&D)a;7!5WmEgEyTXK$P`2A0p>!|B#epXS`4)y#jAi2vkKU|0oNFC8 zl+~R`9z5uNqu-X?xaNI*QFrWRve=y|KT_XkX@NfJ-uc1$O>Q_o3I+2v=PRuPg<~cxc=YS^i8o(kZ8q-8(mY|Cl}^VI>x0zfx}-d81Y?_J zT`g2@)F-w@%1Ded3MG4f<*cRq*i?@SUEi5hDomEyf4&{ao=mUR2|CE zGKRsTl%36H&`r&@u&xeW?pT?)4V(Mox=WeiR*S25m<(@_Q$dfq7nz_QK9NZ0D1rq= z#eW%|_+=2Cixb3+NP!rZ;Qot`i2J`D*8X*TKqsc#su^%a>c|7F+kxN{fdN@(iCQ)} z$BEL67g8X#)C*u!Uz7KPzz~%@oRse69!^f;)w8N2tL0YrCkIF@6@4j~-5!s~jxkAA zwbktx_uW#K>r>VAq_8Q=p#gbEPR5qqVUe#Z^KBQa8Nrde4ZN$Z_qeV+t>2v)cLi8f z2JWH|9mgf)Z&Y4N8LG|QeZep4Sk5guJti|jbKGPm@E_B{dj=z!kk zm_C{htm4Y~;%$V6%Bp?n_Mw4^=$>+Q@+Y!A1f+6xk`D%S%^JGv;pX0@x$fHf6EZ_{ zOE0ymoC$^1cW2v86r9Z7^@g4$f`5VXmvMG5_m3f5?q3}gU*k{_5tEWodqX=j8wYE< zi_t*@Nm1Vt9=3a1^^TZ|HjA9Nf)Whsq6&|o{gFUg-_i``3b+Zs2SD5eAnr1Ky$#bh zfd|Wiex&^t&B_LXa01vs@WFm~a1A0#86HRcHL6$6K_6yjC}w2>g911&9uu=SyoeD- zh#`C|6CRlRmB0q#`AU$`w~>XKnV7=kh~fEMq)~N%T514zzh;2Ako|svs1ZDF1`2}5 zYQv*MRDOv%ya*2cK34gYC0{fA9+#^I4--b*yP%5e+d~l<{%7`>+1WctnCjbo=YhQb zw`(BiYXtN^nGU#sGfJ4*FO3cdM>^tG&)tY8T4yN|-Flo?5s0m^; z{D1xjfnxxH^8*5pB!=(+0o=S`05>;cq!{tXeGw`4YwnN>kM*yVf1Y#0gOs>=;ZJhH z^FbUh#?ZMz2yX)DVqlsZ!~+2FfB?Ka@DvbU_zq76Pss@YLwEpOygUFfd@K}khZDeq zxX%fv@_iE$c;K|Px|JDXwg$YI@D#tI$o<`A@ry_2qAFjx`k!K^zK>}Fxp{bg8`Mm8 zc2OIB(>6X8n~Iy{Plt-0h$TV8>py@c1+{H1$;^oBUybTqkYkpHJks(1$Y3qFpwlQpGu*$Sj;uGva{DS3fS zEziGzL(tBer!{)zd^oB5%7ciB*#WKL0sF>>v%$(H^CUEC!9)tr{nCXvr)+?V@+9CE z#}o9NRiJ)6J&jy=0=~$T#1!&q$}|5Sg5;uZ5CL^ok8Nw9>57i$i4mXEBfM#YP?ojF z?*&SmDom3*_`TMSeZm)mBHALmd6~h&{B~=>%hM>fiy&!A*S(HieGN@LOFb<=L~u3E5NCG#jP&?;gjyImA3!-n4IU1vI-hgjhXk4s?g!zx6~+9G^{4kh%w` zmkT)we+{ISiAcaeoeLwyX24h23z^qW%=W%vgSO z`j^iw42jL+Wp$k&-V>>^WtXZl;%$*wVdH}Xr2X;9QiPrr30pUOerNlIF!rQXMzZDT z_N`4460cLg0m}9|oUIQQin3n{HB*`xdOFEW-v~2{<6z-zY^J^a+=Io0R%+mWS3JZ0 z?m`Sr)8e85`f?v%9k94?J=5BN=D^W|e>+(2{`7%;2hT!O@U>chiKQ1+b1X4sWf}Fs zgMC*&xZsX$W8anwf8CbccTJmCrVeD_-@}{X#Ti$vG!MEJqdbBEP0Ur}<}phQjHZnd zKX2#`ASG(YSMSIPV&>5of8!BofAEP^!u}lXdT~qQC=n&C_-oREvnF48eB0EunCueq zf|!k^xGAqK3HM~rM9-QpXOA!G=&Vl?O&f*mr`K?6%@QpyzUe+crvE@n)kVJCYscbM zj^=&1xa#`pG_CyfeEfVXL694FE_7z?g{q2_L9}U;?i2$glx=~zSknW*-0D_I-U6|$ zsMK6rko(~4vg&Q8#i$X)P5^waLnFUWoUx%bK1k&3r??qYZ&DyW0&Y2Tf8ITGFlts+ zsCa)A$~Q=QjUywx+;|I6GS|JR)Bc+Es`De|;&LqEG1Cv0$74kJPba*lD2T)CfcJU` z>;vwo&7ZuKda4ziq|LhCN?1G+9G?NZG;2Z7EAxtZMCI)0Znq?gDRFOxRDQ!96c1HS z6=?UPN~!#oe&*u4`2?$>Wf|PZbweHbQk`V9x@cqiqZJ{0t!ROU**$Ky=@SpD%jIC# zNWPdkC|B%-g|*UG_9ujPpAA^^(Z8V((F|t!Ab6jxpz&z_z=hq4kS8s^wy$IEW_jqn z4?Vdh7Ex8BHZS&p#Sa8oDT(nzWWuZRoAL+4cWXw->?a=bX@#-e_Z|v1JJWFLALGAY z{RZN_drOnryHSo!19eq-^nPUW2+j<~=2ftIHW>~F)i5Otc_oo*G< znm!6~dg8Y&(wVj?OGS)--C2?31`WU|_-SQqWUx-gaL2j!WPR5T<_gM8(u_f*3-O%} zA_FR? zp$2<2;b-Zi+f$Ni9gH+$o0}({Tp0Xo$Dh+38Lf6srqt_fCG1rfKL$c?Cya3DzAmy7 zvs7?gK6SNU?O{K1dkf8`ZNtvvmXlUiP7KFg*G{KECqdXfHMKsYFKxA3%KOHkGGD}#?)lIdEM zAA?-8so%#oXH%M{d|?fq>=kvj4XM`Dn)?>FLdQEoC^9*OSY_~ePD(h`46l()3j}WN zJs2`0h!ZxEFUx#m5Fx{({w#X#@#>NF(BhL5J(&VwmtK?3$|W+-U|OC#&in(~`f^7Y zvO-*=3q_y3iLcizt|jCcGy5hLsdYz+YdeO_4Ex(1A{#bi+IliEnX{5OzmcRzu6;Ej zQLLkO7o{og-Tr#=ow&`#7|9A|v&ES06=9jsH|nl+^E;t$jA~yEr3#~#TAw~&AX#>r zdA#y6|Ap#ZhI`QUo2VuBYgjVHu} z!9&m1zmr!mYAI~No!Wo5x;;6gx+YP$uvv9c-MLeL5QD0%J^Ol^>FcJlry40NFIn*K zJSuYMdfEAM`x@b)M-5*IV@8ATG}M1R!uZZ>_bhcAZjW04j?E{Es~!Z$o|FIvg&0nrh)|G?YM_w8`d@YmnLW&LfR8BY6;*f(DJt|z z?fMPUh|l!p-8Z_`+9jnZ!6SWZrq!sLJ#>>&=27!T19*w~W3Gj&z2)1vCKVZd+p6}; z>BgL_&BwuAV5T7{>`=Y|yItHBfma^)9=U`PI0vcTVPT&)2qH_EdoV*SpSP}D?j+mn zzjA+b)P%nluXgCYtn#8f)er%-fl8MLpVrGy6WbJ@uHsddscxp`A#tvR!)DiSg>yzxQPo2< zkQKk!edvTMnPgI-gP9OLs-{7&uBYj$zoBq^JFLs})M^V0Q`8+&N(GP1@v+AuL_?(FHVKYt2&AcN`TdjiG}Urhny$VN z9qb`{u9l75u_-nq%jDD|Tl(7RSwA7|4AV6KZqU|@^d0)fWXs$edQ7EU}>d^Sa8rPrUk=PZq*qI+0r>Bk-0rCdkOkSet4;5Y_%d^h(K`n-qq)(! z1>+t*B>N;yUN$?Bi`{=Wpt(raaKm(OT{q`Msb9RXSeNZh;`;T%pls!|bhC9oP|cZ3 zJEq}Y=Ib=ZoSKp zb7z8(zq(vNBj?F}(`@DneVgF<91@Gt9kuE785H`r8_` zdD$9y)@ZqI^Wm$#sX2AbH-`CNlPXGNr=OYOdH9*;VN*EBYY8(QniswZpe7-8pQO@= zVN_RnFf6_5)r}fq#F>%uew^e9*_dO{Cu|CI)u16#CAd;>xiH|R+guj4~_3gNwRm3tji=cVERxAe3J z?s(#j(G>1}pR)UYuYn1PyfDwW^;cbmcx(2aianA~L}sn2R@<)$lFU(3o2i$29KTSR zlXY6u(9|N(C&X4;l9Evag4ETs=$}cKwoC$f1P_kFKUDX~#nFcA0!s)Zbz9WK53+jf zataQ*B-7uxXsc|$^X$qB#o?@F7gX06ohq<^rp|8xr$}<2Yt-qF;F2y1&{fvGskc0K zH;uSze-h++a<8!HgN&7Sh2!MuM2JY}EJ0gZ0p%Bit?pZMOj>+<S~pFa)gvAmU6;_ni+o$q`M!}c`QBR zFJ%<0Al?#5dK`dBQbIe#Ri);xJIw(KP;h6@r3#VX@_aWo>sN>3$&+g>G3Q%{<%!~U z<%HjXV;^g6aiHD*p1&zou<|q9#5X$Jl|(gI;;jbzbW&VXx1*m4(c6h}o1$y_DUEm{ zL!C2Zm^P;Z1ZZQIxdin#Chs?_ct9*+eTJnG3M?sQus&#MV6Qpk3 z+Pd4CS12H*6P1&Xi0&M3CwIb(_o z#1_U1Gnm}hGc+b@W=K{bPLYKWz?vrgi?N_YO>HgGhP0Ex&!Tl;ibZ^VD?E_;627*T zG?s^JN^{rUL&bn4OV8Vub zOeuFJDD>?Ud)r}n#h-$EgJ8mXd%`J46E)o{Hz3xfu3^0?uvtC(sNM~juHhbY3JOeD ze-D(BFj3R7f(Gd?b&crtg7lZT2KElaQVsU#QaoXYC9b}`i!e&Vy_+dQFsIV@Mx|lB zv@lBJ(%4=+$R)@Pm;(gA*fpvb1-5N6+_xeOIV!0N>UDxRmsW-KK80=T*$bphPrwZA zSyEoXtn}@{DJc_gJ66ykO7$=P4 z?9bLtMbAG(K|=dzwMo{fP|UJT**dG}EMzq>P80(3vG==u^69k2hQy)0#N5$bDt4~V z9ze2uj7U3GUciGXQjKh%SyLJjbc)CD$Dq-b`P5ZKk%hjHdm+m#$1LkXwkq6q&ShcWHyGgE;2& zVcC@!lA_eho%Eg5ozF`;NBX2T}zyso}v8|p>-DQ@X{+?Uvl2eMuapF)<}MeRtJf<*0@ zmfS<@Xg|d-ku!fz&M1_Pi1CnvK8QN?i%E+;eY#{#=bg>$oxTPR70zBGilLyd)8WFFTvVyraC&-YMXtyDB=1x1WsW8n&cxt4_==YRNHXJh$s59mdU1*qz;U z9MLth>bd~e)R_M0O-9%FCDNFjycJbqA-R%BBaS#Bv5&m;RJ5*-jqp3IViaFgg=NUd zQZh`+mC!~#M#DmlF_z$$b%+U)rKEFxxa1Y%6!RHbZlZ9xcBET~ZaQz%DyCX?o36@l z$th-9wmkant*4gRx>nRxNnKVnRnbf6F&lY}j>I~*?JjqYsySUfu%qw1s^-Mk38hPj zfl*iao9oI)r!CRM^wPm~4>}^unk4E8m8^rr6mrN!__C`{3oy!& z?$w9~TbV$DkpiHYtRy+PkVads#5XG-D=91FjaAl4w;UOAhe~d?Bl^kB(IS z>17%|Ke|VB+*fjp%>W^OClsQd&y`G^)?1~A(xwh*-`_T%e=jj4pqGRrho`k7=7z?) zM=eJeLWe~sB)5@5{98rigjhz2W zmhz$Jyae~40+jbGKVLr6w81?ks(KLhne-IZ3snSgE_93soufHJ_qbe#EE9cB=;2f1 z4LGAXy>WW^w5%O@`hnZXq%caF@0|vMB|Hfc^q>ZrU=e>Wq*;s|ALrgM%bmoa>Dj?-Y>Py4|mDDfo=%Vwx%%q=2yO{($V;%r}Cw^QFlxCz3yn;P_4}LP4lwL1tHs4x7fC$;{D@St_|a}mDEbDly~T}sSvTnUvm#* zdOkU`W@T>e5Z0E_RxVu&H1eHwsZ=I<1bR_wF&80iaqf|8VVOA@Q~$QMXJ(+Tr}ni| zh&us#C8xblb%Cc(HKcS(XIOWupex@&)9B>w9(eW5YFWQyeu%KPgl^&CN#*!>t+?A8x672t95JkNhjgkLZ_2aYLq7!?FUR0fS>Aui))^g7m2%UN z$TH;a?0AS>CZ}BWWAVFy=a?gtOhlHSp0{V8D!WwY&EOA_JHUEMO%;$je_oJGUF)nK<+2qwgJFgDt}v<~!u zdF4+UkB!_*n0&khqRlMW2~W%@Ih`=Q_j;u-G?nyKRYs}ZN1eg<%^%OF9k-Rn<lJyV#V2M!gLkLHBCnC<<3Qpdxm`6;K7OCgJ$0VQ^KohK)^DrgM z=RU}sSLH8#0^l$CQbb*5Kq2Wsj&0D^qe(<+^RopIg`*iZm~vu=qtZ6maN-(0Ob6V+ zy`*Tva2*%j2P{e^i-zKZf{Vn|fCqafvW9Vj%(8~!=4Uj3L+2wIh2-Yzas`7GaBUyS z_zFsf7;!Kv0f55aN9hWv3TKDm*(EIh-1v@8qIurq%cs~8KHk0$8v5rMcWI9=dEnQf z%=ub3jL);!5PM+NA)Owj8i_GCm(y%wrS41U7688X_J8&CM*-?urV_M16Iwhl*t(tX8B)ek zd`LeAC!Ra+G@rlY*r|6%f*%G)9EKX@YhzaXiO$h;$r?jrv&bvh46tnzefwe zeR|*L;s2Ok0Qov41L4U1n=9@g(E?G8=J2T+fjeHxFU$+RXLQKg+b)9;y;W7xp> zz+5HDPb6mRBFOnC8qwdMi+rkB%KKv4$rwd~_q-XkFBbZwsRup1y)l+|M6Xuvp1Kuy z{iac1HS`v5`I4^^_7ZoP>H?b64rKqB21(SbSn80gVS9B%=Kw$I^T`05gZl@d;NgN=M!naut2%|z2*ytuS{pEhb!#FEy{#ETo} z;4t~AqdHZX^G3nQNv#P57`#dc#h`CGsbushU<#~MS$=z|JILz&pPanl4#EWu=MM<< zV?hJR_t_!{pXN``^nZ5Nt7vG+DX>VuUFimPW*7eA3%~a_NBuYYf8(O(<^%!Q!0<)G z;4XT^qGIs1dj1<1JrDO+hx<>>_uzRW zb~tPP>--V#hNt6!X9$766vTJ2%pb=^G0m*4BoWI3-jw7Aae}~{ARrfz559~OgoP8x z$jQm{D{sC>W&KnG68i9X8EcaZVZGq=x3%=(dh}b!mWv(K7@HFw-hmA-`L`eZiW|Oe zh%w+R4aCWdP&|kq!0H=~lar4VuGW8}K@c-Pf1`0C)(rfOhFGQUHyRLLuRmzq@Ia5> z)A9XD192f@bN=}b5HBL|?4L9+JS6B38ienUcW?nY|47HhjaV!3pZRg|A?7dslLn7% zflKgjG#5pGBK29A2bN}ALWBr%%&>yKwaa8>(zejuJd>IQ_b^Z9%E;50BEBo?Jm|QRW);iBJ?Dpdsnfxb^suR_uIOpa#qIHfQ!EAqPMem W(6@8A(5&$MAYg2IdZ|0o*#8fEu54KV literal 0 HcmV?d00001 diff --git a/docs/paper/paper.bib b/docs/paper/paper.bib new file mode 100644 index 0000000000..733c9b7685 --- /dev/null +++ b/docs/paper/paper.bib @@ -0,0 +1,239 @@ +@article{Alexander2020, + doi = {10.1088/2058-9565/aba404}, + url = {https://doi.org/10.1088/2058-9565/aba404}, + year = 2020, + month = {aug}, + publisher = {{IOP} Publishing}, + volume = {5}, + number = {4}, + pages = {044006}, + author = {T. Alexander and N. Kanazawa and D. J. Egger and L. Capelluto and C. J. Wood and A. Javadi-Abhari and D. C. McKay}, + title = {Qiskit pulse: programming quantum computers through the cloud with pulses}, + journal = {Quantum Sci. Technol.}, +} + +@article{Cross2019, + title = {Validating quantum computers using randomized model circuits}, + author = {Cross, A. W. and Bishop, L. S. and Sheldon, S. and Nation, P. D. and Gambetta, J. M.}, + journal = {Phys. Rev. A}, + volume = {100}, + issue = {3}, + pages = {032328}, + numpages = {11}, + year = {2019}, + month = {Sep}, + publisher = {American Physical Society}, + doi = {10.1103/PhysRevA.100.032328}, + url = {https://link.aps.org/doi/10.1103/PhysRevA.100.032328} +} + +@article{Banaszek1999, + title = {Maximum-likelihood estimation of the density matrix}, + author = {Banaszek, K. and D'Ariano, G. M. and Paris, M. G. A. and Sacchi, M. F.}, + journal = {Phys. Rev. A}, + volume = {61}, + issue = {1}, + pages = {010304}, + numpages = {4}, + year = {1999}, + month = {Dec}, + publisher = {American Physical Society}, + doi = {10.1103/PhysRevA.61.010304}, + url = {https://link.aps.org/doi/10.1103/PhysRevA.61.010304} +} + +@article{Diamond2016, + author = {S. Diamond and S. Boyd}, + title = {{CVXPY}: {A} {P}ython-embedded modeling language for convex optimization}, + journal = {J. Mach. Learn. Res.}, + year = {2016}, + volume = {17}, + number = {83}, + pages = {1--5}, +} + +@article{Fischer2022, + title = {Ancilla-free implementation of generalized measurements for qubits embedded in a qudit space}, + author = {Fischer, Laurin E. and Miller, Daniel and Tacchino, Francesco and Barkoutsos, Panagiotis Kl. and Egger, Daniel J. and Tavernelli, Ivano}, + journal = {Phys. Rev. Res.}, + volume = {4}, + issue = {3}, + pages = {033027}, + numpages = {17}, + year = {2022}, + month = {Jul}, + publisher = {American Physical Society}, + doi = {10.1103/PhysRevResearch.4.033027}, + url = {https://link.aps.org/doi/10.1103/PhysRevResearch.4.033027} +} + +@misc{Greenaway2022, + doi = {10.48550/ARXIV.2211.16439}, + url = {https://arxiv.org/abs/2211.16439}, + author = {Greenaway, S. and Smith, A. and Mintert, F. and Malz, D.}, + keywords = {Quantum Physics (quant-ph), FOS: Physical sciences, FOS: Physical sciences}, + title = {Analogue Quantum Simulation with Fixed-Frequency Transmon Qubits}, + publisher = {arXiv}, + year = {2022}, + copyright = {Creative Commons Attribution 4.0 International} +} + +@misc{Forest, + author = {K. Gulshen and J. Combes and M. P. Harrigan and P. J. Karalekas and M. P. da Silva and M. Sohaib Alam and A. Brown and S. Caldwell and L. Capelluto and \emph{et al.}}, + title = {{F}orest {B}enchmarking: {QCVV} using {PyQuil}}, + year = {2019}, + doi = {10.5281/zenodo.3455847}, + url = {https://doi.org/10.5281/zenodo.3455847} +} + +@misc{Hamilton2022, + doi = {10.48550/ARXIV.2209.00678}, + url = {https://arxiv.org/abs/2209.00678}, + author = {Hamilton, K. E. and Laanait, N. and Francis, A. and Economou, S. E. and Barron, G. S. and Yeter-Aydeniz, K. and Morris, T. and Cooley, H. and Kang, M. and \emph{et al.}}, + keywords = {Quantum Physics (quant-ph), FOS: Physical sciences, FOS: Physical sciences}, + title = {An entanglement-based volumetric benchmark for near-term quantum hardware}, + publisher = {arXiv}, + year = {2022}, + copyright = {arXiv.org perpetual, non-exclusive license} +} + +@Article{Harris2020, + author = {Harris, C. R. and Millman, K. J. and van der Walt, S. J. and Gommers, R. and Virtanen, P. and Cournapeau, D. and Wieser, E. and Taylor, J. and Berg, S. and \emph{et al.}}, + title = {Array programming with NumPy}, + journal = {Nature}, + year = {2020}, + month = {Sep}, + day = {01}, + volume = {585}, + number = {7825}, + pages = {357-362}, + issn = {1476-4687}, + doi = {10.1038/s41586-020-2649-2}, + url = {https://doi.org/10.1038/s41586-020-2649-2} +} + +@misc{Qiskit, + author = {M. S. Anis and A. Mitchell and H. Abraham and A. Offei and R. Agarwal and G. Agliardi and M. Aharoni and I. Y. Akhalwaya and G. Aleksandrowicz and \emph{et al.}}, + title = {Qiskit: An Open-source Framework for Quantum Computing}, + year = {2021}, + doi = {10.5281/zenodo.2573505} +} + +@software{QuantumAI, + author = {{Quantum AI team and collaborators}}, + title = {ReCirq}, + month = Oct, + year = 2020, + publisher = {Zenodo}, + doi = {10.5281/zenodo.4091470}, + url = {https://doi.org/10.5281/zenodo.4091470} +} + +@article{Magesan2011, + title = {Scalable and Robust {Randomized} {Benchmarking} of Quantum Processes}, + author = {Magesan, E. and Gambetta, J. M. and Emerson, J.}, + journal = {Phys. Rev. Lett.}, + volume = {106}, + issue = {18}, + pages = {180504}, + numpages = {4}, + year = {2011}, + month = {May}, + publisher = {American Physical Society}, + doi = {10.1103/PhysRevLett.106.180504}, + url = {https://link.aps.org/doi/10.1103/PhysRevLett.106.180504} +} + +@misc{Maruyoshi2022, + doi = {10.48550/ARXIV.2208.00576}, + url = {https://arxiv.org/abs/2208.00576}, + author = {Maruyoshi, Kazunobu and Okuda, Takuya and Pedersen, Juan William and Suzuki, Ryo and Yamazaki, Masahito and Yoshida, Yutaka}, + title = {Conserved charges in the quantum simulation of integrable spin chains}, + publisher = {arXiv}, + year = {2022}, + copyright = {arXiv.org perpetual, non-exclusive license} +} + +@Article{Hunter2007, + Author = {Hunter, J. D.}, + Title = {Matplotlib: A 2D graphics environment}, + Journal = {Comput. Sci. Engineer.}, + Volume = {9}, + Number = {3}, + Pages = {90--95}, + publisher = {IEEE COMPUTER SOC}, + doi = {10.1109/MCSE.2007.55}, + year = 2007 +} + +@article{Newville2014, + title = {{LMFIT}: Non-Linear Least-Square Minimization and Curve-Fitting for {Python}}, + DOI = {10.5281/zenodo.11813}, + publisher = {Zenodo}, + author = {Newville, Matthew and Stensitzki, Till and Allen, Daniel B. and Ingargiola, Antonino}, + year = {2014}, + month = {Sep} + } + +@misc{Lebigot2016, + title = {{Uncertainties}: a Python package for calculations with uncertainties}, + author = {Lebigot, E. O.}, + url = {https://pythonhosted.org/uncertainties/}, +} + +@article{Sivarajah2021, + doi = {10.1088/2058-9565/ab8e92}, + url = {https://dx.doi.org/10.1088/2058-9565/ab8e92}, + year = {2020}, + month = {nov}, + publisher = {IOP Publishing}, + volume = {6}, + number = {1}, + pages = {014003}, + author = {S. Sivarajah and S. Dilkes and A. Cowtan and W. Simmons and A. Edgington and R. Duncan}, + title = {t|ket⟩: a retargetable compiler for NISQ devices}, + journal = {Quantum Sci. Technol.}, +} + +@misc{Smith2016, + title={A Practical Quantum Instruction Set Architecture}, + author={Smith, R. S. and Curtis, M. J. and Zeng, W. J.}, + journal={arXiv preprint arXiv:1608.03355}, + year={2016} +} + +@article{Tornow2022, + title = {Minimum Quantum Run-Time Characterization and Calibration via Restless Measurements with Dynamic Repetition Rates}, + author = {Tornow, Caroline and Kanazawa, Naoki and Shanks, William E. and Egger, Daniel J.}, + journal = {Phys. Rev. Appl.}, + volume = {17}, + issue = {6}, + pages = {064061}, + numpages = {17}, + year = {2022}, + month = {Jun}, + publisher = {American Physical Society}, + doi = {10.1103/PhysRevApplied.17.064061}, + url = {https://link.aps.org/doi/10.1103/PhysRevApplied.17.064061} +} + +@misc{Vazquez2022, + doi = {10.48550/ARXIV.2207.11268}, + url = {https://arxiv.org/abs/2207.11268}, + author = {Vazquez, A. C. and Egger, D. J. and Ochsner, D. and Woerner, S.}, + keywords = {Quantum Physics (quant-ph), FOS: Physical sciences, FOS: Physical sciences}, + title = {Well-conditioned multi-product formulas for hardware-friendly Hamiltonian simulation}, + publisher = {arXiv}, + year = {2022}, + copyright = {arXiv.org perpetual, non-exclusive license} +} + +@article{pyGSTi, + title = {pyGSTio/pyGSTi: Version 0.9.10.1}, + DOI = {10.5281/zenodo.6363115}, + publisher = {Zenodo}, + author = {E. Nielsen and S. Seritan and T. Proctor and K. Rudinger and K. Young and A. Russo and R. Blume-Kohout and R. P. Kelly and J. K. Gamble and L. Saldyt}, + year = {2022}, + month = {Mar}, +} + diff --git a/docs/paper/paper.md b/docs/paper/paper.md new file mode 100644 index 0000000000..f158d93d57 --- /dev/null +++ b/docs/paper/paper.md @@ -0,0 +1,132 @@ +--- +title: 'Qiskit Experiments: A Python package to characterize and calibrate quantum computers' +tags: + - Python + - Quantum computing + - Characterization + - Calibration +authors: + - name: Naoki Kanazawa + orcid: 0000-0002-4192-5558 + affiliation: 1 + - name: Daniel J. Egger + orcid: 0000-0002-5523-9807 + corresponding: true + affiliation: 2 + - name: Yael Ben-Haim + orcid: 0000-0000-0000-0000 + affiliation: 3 + - name: Helena Zhang + orcid: 0000-0002-7813-7133 + affiliation: 4 + - name: Will E. Shanks + orcid: 0000-0002-5045-8808 + affiliation: 4 + - name: Gadi Aleksandrowicz + orcid: 0000-0000-0000-0000 + affiliation: 3 + - name: Christopher J. Wood + orcid: 0000-0001-7606-7349 + affiliation: 4 +affiliations: + - name: IBM Quantum – IBM Research Tokyo, Tokyo, 103-8510, Japan + index: 1 + - name: IBM Quantum, IBM Research Europe - Zurich, Ruschlikon 8003, Switzerland + index: 2 + - name: IBM Quantum, IBM Research Israel, Haifa 31905, Israel + index: 3 + - name: IBM Quantum, IBM T.J. Watson Research Center, Yorktown Heights, NY 10598, USA + index: 4 +date: 21 February 2023 +bibliography: paper.bib + +--- + +# Summary + +Qiskit Experiments is a Python package for designing and running quantum computing experiments +with a focus on calibration and characterization of quantum devices. +It consists of a general purpose experiments framework which can be used by researchers to rapidly +implement new experiments, and a library of common experiments for calibration, characterization, +and verification of quantum devices. + +The core framework of `Qiskit Experiments` consists of three parts. +(i) An experiment class defines the quantum circuits to run. +(ii) A data container class named `ExperimentData` stores the data measured during the execution +of the quantum circuits. +(iii) An analysis class, attached to each experiment, defines how to analyze the measured data. +The analysis also attaches its results, such as fit results and figures, to the data container. +\autoref{fig:framework} summarizes this framework. +Importantly, this framework can interface with services to store, load, and share data. +The library of experiments includes common quantum computing experiments such as Randomized +Benchmarking [@Magesan2011], Quantum state and process tomography [@Banaszek1999], Quantum Volume [@Cross2019], +and gate error amplifying calibration sequences [@Tornow2022]. + +`Qiskit Experiments` is based on Qiskit [@Qiskit], a general purpose Python library for programming +quantum computers and simulators. +It uses many other open source packages. +These include Numpy [@Harris2020] for fast numerical computing, lmfit [@Newville2014] to fit complex +models to data, CVXPY [@Diamond2016] for convex optimization, Matplotlib [@Hunter2007] for plotting, +and uncertainties [@Lebigot2016] to provide measurements with a mean and a standard deviation. + + +# Statement of need + +Quantum computing processes information following the laws of quantum mechanics. +Quantum computers, like classical computers, must be programmed to perform quantum computations. +A quantum computer consists of qubits which store information in +quantum states, along with additional hardware elements, such as resonators to couple, control, +and readout the qubits. +The different elements in the quantum hardware have properties that must be characterized to +calibrate the quantum gates that process the information. +Furthermore, the quality of these gates has to be benchmarked to measure the overall performance +of the quantum computer. +This characterization and calibration requires an extensive set of experiments and analysis routines. + +Quantum development software packages such as Qiskit, ReCirq [@QuantumAI], tKet [@Sivarajah2021], and +Forest [@Smith2016] are part of the quantum stack to execute quantum circuits on hardware. +They also enable high-level applications that abstract away the quantum hardware. +Forest-benchmarking [@Forest] and pyGSTi [@pyGSTi] are tailored towards benchmarking of quantum hardware. +However, there is still a need for open-source software that enables researchers and hardware +maintainers to easily execute characterization and calibration experiments. +`Qiskit Experiments` is unique in this perspective as it provides low-level characterization +experiments that integrate with pulse-level control [@Alexander2020]. +In addition, `Qiskit Experiments` provides a calibration framework to manage device calibration. +This framework is usable with any hardware exposed as a Qiskit backend. +For example, the `Qiskit Experiments` framework has been used to explore measurements without qubit +reset [@Tornow2022], characterize positive operator value measures [@Fischer2022], quantum +states [@Hamilton2022], and time-evolutions [@Greenaway2022], as well as calibrate gates [@Vazquez2022]. + +![ +Conceptual framework of Qiskit Experiments. +The circuits are run as jobs on the quantum backends. +If an experiment exceeds the maximum circuit limit per job it is broken down in multiple jobs. +The raw data, figures and analysis results are contained in the `ExperimentData` class. +\label{fig:framework}](framework.pdf){ width=50% } + +# Example usage + +Here, we exemplify `Qiskit Experiments` with a Quantum Volume (QV) measurement [@Cross2019]. +We execute random SU(4) circuits on a noisy simulator of a quantum backend to quantify +the largest quantum circuit with equal width and depth that can be successfully run. +A depth $d$ QV circuit is successful if it has mean heavy-output probability greater +than two-thirds with a confidence level exceeding 0.977, and at least 100 trials have been run. +`Qiskit Experiments` only requires a few lines of code to run this standardized yet complex experiment. +The analysis classes of existing experiments automatically generate key figures with customizable +visualization options, as exemplified by the QV plot, see \autoref{fig:qv}. + +![Example result of a quantum volume measurement carried out with `Qiskit Experiments`. +The dashed line shows the two-thirds threshold. +Each dot shows an execution of a quantum circuit. +The shaded area is a $2\sigma$ confidence interval. +\label{fig:qv}](qv.pdf){ width=75% } + +# Documentation + +`Qiskit Experiments` documentation is available at [https://qiskit.org/documentation/experiments](https://qiskit.org/documentation/experiments). + +# Acknowledgements + +We acknowledge contributions from the Qiskit Community and feedback from our users. + +# References diff --git a/docs/paper/qv.pdf b/docs/paper/qv.pdf new file mode 100644 index 0000000000000000000000000000000000000000..27474459b4aecd34140154fe6f5953a5f1e7db07 GIT binary patch literal 21877 zcmeIa2T)Yavp6bQi4sIIOP08K$wgG9+7AW=Y2Ku{zK0wNhC3n)oJ zKtKcm6aTZQ-_OAPuK)XP)vJ2-u*R7)JE6O$&vZ{u_u+5C>0?U}A&7E9rK@t*#)-G0(ymN`KE~ zzUN9HQxI6k&fLo0#TJD6*4;qE!ok`S4H7&W{6FA8e?~yc#l;ov34%htK_CM_W9S#;3=4s9mL9FMR&CK%BJRC@!##*Eib<+acFa}M%HNirB0vuviAYW;pc{oSAw74 zFI%K9?=Mf}w7GuDA-nhF+&JE6KloQ33ccmRq7yR=haYX~K3VUrgI8X&_2q7Qw^c^S zGsO{3=h-^y^Rju5K-oXpN9+H1vqbmL$BuPmd9M zs=q;=a{(68=(8FBdVIJh2f4OYTjODv6U9I9@pDz{cEU2>S!*=`5#uNIBrEj;S>8U+ z3`r^bA5g_kE=9`e1&G;3^YY4v&c}Ju4|(Pw-)^gxq)&hE;Uh+@;?^O>AoL>dTF*Eg!3zK3V%7 zP#r?=4&eGsE#w*I20UzOI@q_1c_u>Yawgg(g@D9w(qd-w98K;`g)cKNUj(X2*r-0` znq1UWx8TSXvD80Y8Q{i(zni>J=r7`v8~S7*D<=M<$b3=AB+rYFt#7aND9jIEvEnGq zR6)h5(+$0ELXVnyj<~yWeYTR*QyQTE?EbZ=WMR>tup>E{yUQX#a&TvKa`tsu;R|(N z9=UU3lx*2&G`e*C&>F&L^%jXbo~@K&i4W>1%px@y_q-qLB=TtT)e9w4a9-}wId~_2 z=KZS4W3hO#v0=D4(!0)gwX{wrSNgRsvA2CSg}SG&FIvOvdi7zzxPSaj0M;8ASUI5X zNMRrK?XV?-O+`{~kG2gbqZz!x1w1jE+EIXK?wlGE1YC-ZD;sqx-@;HW=GOL^s`u zt|(K&M^bk>u`5IF!O%D6bBVXSJko9m((mhJ@5O{VRzK+GjV(NJ*L>S7pSZas7N6mI zHH<*UQML4;rq4%KL7dlHvN)F|V+>FN*b^C=J?;~D=S#G4wWlQ|8j>ix^d+gVx^%T^ z&PRJY$5qIC)N1*+B@SI*AYYY84LZoU#~Mj0HIF^Ll%Y{5m#0BhM>@(>D8%SXb(5Wk zbdSJ*!DcC2=q*#zE4*jaZmq47=Fj6idG$m-ig4ZV92(D)=_`dm{kZ0!X39{B-`hm;*syw8_6^c@U@b6s;gy{ zL_Q}@;Pz)0EExE$-DlRMA?>sZT^OG>*r|qAEU-1)@i|A6x1h->ph(UrG0qeZ%H=CP z&1-drj75|K|1}Zyv|!m~)p(k$$G*D;S~|BUq)Uw=yL`9jUYJDM&!NI`CZ?7|c*E|D z4lg%d3szaHkQ9_l5HzkLs!^(rmuQ7K?x|)Lv=ywLtHFwKfb3k0h<_npKKM%6|Ax+k z`!`v~9O)7`j4!)oWjLKt=e@_re9c=&nR6MNKu1K;(3DT8->R~GUV?<;j)1^;*WA_T zWZZ-J8dHz2`U^U&ggd6PJiTwlGvBCON%LMZU)R4B%CEp6a7}^X zv=Exq1;24Emqr<~Tb($*8c|A5P;KN6aeHRXqGTDZoI@>An>8l<483!gU%&PC3Z#34 zMoL8XMwFf4oP8o$ZSZ4>phI*D$RWXw6zPZkA>+bkVj29h5y|T)RVDLv9H&qjut5Z2 z^6lY%^E=pf7Ys*&IYJylaW>5|?u!R<*Y7u6xMw|sbvDVXM?fF@T+#hfp{FA3PhZKz zE=5?{Nnb2`Tk{b|XtH7Q+^qp+Np2Bi7UOic$!Gx!`g7OHB+ZLO{6eG18w?c34b+U& zam`PV=lifvzxY=C&T;s!JvYxvlBiAY;qHB8;sfv!y&@I8!!&RW7ZxZU1wpHi@Cpd~UHbCh^;*O<9tAZ#_Zr;oNd zCD`%`gk!Ozg?F$)By@X*?2(1_$5~4JAv|N}hhO|E=Q-!y#?SWJy&#*LeH~gGYS9(g zuc|8ISNY}Efn+KkKdVkl{0qlN(QQG+I2tV4dpLfTS9GFJDhg#1H*e0(j+=_J#y6S0 z)SU6&x%f#<&t_GW=v~JL`fE(5&z9A`zA+=qnmFI5`tmXrlf^k2|EMIXNYtVdg;?GY=VHed9c;tW7kbHRM#e%+B_(!ZaY^*msigwXs~T2c z>m;?;r^autxp~YZdbyD?Y6ET0_D#@cpJmdobw3{N7LqAGor$D1LE_Bw3`g8A~>%__>F^wJo?jBB_e z!}qmFl!6zd+f{wfKNirsB$M~Rj^PuOeClFB>*F@eOd)U3QAP&~2Y{?Kn-s2m~ou~BJI)h@~HMA=o*fkLcN z?Sz0VVx<_#Hv^~cB ze)zbQb~{zRK8hh`7KwaoP{`FLKcS;WBQA1O(0U*^tanZHl*jLa&NRpn5ZJo7Jd|I z@OmIKt?gcvTqQkh+A+oLG#>=QSX22V=`^RD7MIj#;wGUwM4$d^4cj%F;;pngD!jQ13Eg&gefixX&R99 zs8#!Q9F{#UYu%XK^zC~P9Z#6SlYrDpf?y+5BnfwU39I}gJvCG|nUYyAb&@42CX5P7 zj@Tl|<}~9vHyzrYxU$WC@ny~-0lQT9qUeT)iwKrIK?yofhYXN4c+4V{oHPYr{<&&|6Mas;aS7L# zpb^|?_uBpT z7TnB}fvRis%-PH=t0=L_lf%^ZKpNf!#_hw&l{R7ZuGj z7(dMf%&3sfc*_^>1_s%KDhiMmwAU#51|Pl|Em%af6)imAk!i-}G*~G0uxPEfRzBA< zdndk_I3uW0>HH`15+(L|hJwHfkD&9#vXv%?ImElU#x!SQywj&~Kk#XgA>V|PXU&D# zL50WVN7*OZU0NBtM^e1Cka}v6yEGQXV^}5O!8@KUnNE1Al1jaPK}Lwld8)izr&`#! zk=?wZfo2Rz$*EfYkU2A0in))p6l0kMy_*|b)S;%SUh>8K;8VDYy2G$Jgj8+)DCc=9 zS8N@=qE~PB`9WE8!!GZ)G$(?i)7N*G=-ggeC2&6oCgEVtT6zlW3+%8DSsK3YPSTmz zYI?(hQNPQ+ZM2tRAf;LzmoodU)S0J2qUNdU!pmm&QWJ&vtkQ28A!tUiSk4-uM90&1 z$b}~+f<)AbCY1}vs@G+mvFXO+03d=ZJ^i0*CKiN)>>@A5u# zHwwF#^T?7sW>6BFeyo|X1aDh>gd)GXC0vI( z71qLtw!9F_8K5O61MpefiFWFQW;6Vl5Jdl)3@QUZMgK-Qno%g}nbW@Z!1ns1zU{Y- zck11EPZpe>H8D8Nm~<)f6v#PS5P!;C-)4Qx(AWm+0p=f(U4YyI49S&Y!i}1H#9mdxt@6- z3Rxj9jY@cGYi~YIqd-T+6sC|}C--$(IjQu**hn1X{rcrktpjQ@A*q}Ri_@^+oK5y@ z#-2v z=Xu7R3`kUT&tJN(kr)bsKb{KSaiiNLbr_Tze`=bcP1swb7`fFos7`ukx|sMLAIx;w z&iDR?VM>ngq=Ogpw#M$Zeyqx^Dq3k#)=L>4AYOYj)eg7#n)rOAa@bC|ayIkFp%vnJ zt7KK2WUx#f9Q|Myz82H@4zKc!!`9s#ZF;3cM8x3yO(JuS%)KIksmzJQh1bOSDf#gw zj5jH%mT})15)VG+We2I;vlp>&b5DCb<~gF1GL@--J5P+fbm5R@d__p`)*H(42Tj@$ z9&v#Ua@A=3J6n-ni?rr`Eqe=|QLtGiVhf-2vnSyGQU)29!qz3`DtjiyD`WTVdkwiY1kOoCI%d&vZFD z+4iAN+pbg+G2roOQb&DxgXPe2G4WEi%t})XL?WzXll#%_co8mA5OU`&$BlK9_U!gs zG}SG+M5jL9_pB+nJVJXz&8ARD!SpK2nD4v#1myZw*^}YOiKgLUSMa_=Jas_JRvYf4 z3?Fy@gAXHb0}m!fINp6A#_GK+@_Bn{w*T!{hx&&1SRwMmPgSL37R`p@VuU1Qs?XE!y{Z+gFtxh8{W|0xDE9@; z`|9S)L(9*a%j%hXz3`_2?vg4w3dt*5!(#FVCEIJ;i}R;0<>GHTmJg@fZf~o7!H#Zx z-#ykPxao2+VdOwM=YswlbljQbdwvRqm#U!jDjN{P;`Ef&bmwre%BoY>2l~L0FXMy# zr^U#Obz8=$-YeODUY?$qxN7pKrMaoTu_AK9`~3T!joE~^pR~8tA}Pq2yDkekRSCQ@ z^gMZ5cTA6~E&n}5f=Vpy4!5|wMtb@q_m$1z$D?10&dau3i)p@d^Re@jrQJvMF|L(? zjh`pplBK(zzZh`%zJK@Aba_m|#G?(c)P66K_ScQW6H{u=1Q<){cjMl7o9U6!4`Urf zLIe>%45>#}N=&6OU~~M@@Mr6)^fyx_KNJE*f%xG_C+I}N(f^h{WmKk3@Stj z*734H9~p7wT|JzC7<7NwQGYf-A~lsraiAv~m(K*`X-_8a#p>=3eLXaiu^}ZVrKzHI zP)d_}#PSK`&h=i7RXXA_$wjG#SfLc%oXJe~`#gj%R4pj23TgYw{Zq!-+qlBm?>$DQ z+zGNExZQAY>xL{DJlVU#+cAS{7iXFG$?K-O z?;zOt3jd^52Lv{=PX}Y6Sx(%c{%XK&ym@|yQ#19Lg}_*of0${d%{{G;4Ax);8GQp; zJziOB2Xj3y9dj2?ehmvJdv`C40o=~q17o@dtjb5GXF{-?3t-U2Sf(*1b6I;E8*9K? zj4=-b>-dAp|Fn^iz$}!9CmJ&j2to+PO!(g|EYL*O+SAg*-VN>Q0fK%vpksQfnWH`I zkL=e15D3Qd4gCA31kA2#{(UAG5=jVFH+Kdw_!s2<;Sh|mAN~XQZ#2=jw?f+iSRYyT z34b!UW5D@ALMcy6jHO%{CXBf%a8tClx3xn9zk@OGV}9X>!BIzsa9h9%kNRu${~O2& z!AiijguSJdi>;G2W;!)tX7@U1DGm`iY=mgMZ-s2`vns-`++P z1OobBNQD2>G2l=k5CT|T1n{y8!G%C5IEqjhKub^v0)hcI6mVh4Q9=T6Vt&W8hohk1 za=0L-{l6u`AHTzawwQz&<3~cmfn5fIA%GA9NWehCLLfmw;291BA%V#SF$oC+!Gw{7 zKmwtVC=e0}41$7#1Q9@6ApigfP{!n#-{C+4E@=dT{+Ka^g@l3pTOB4pLIy5Kh{-WP zV35Kj6cm2c7YQK#Z496-Fz-S!biEVAOJ5=kAlJg1i#IT0Om!)0Ky1k z`XN9_AQu9Ng~^W+aPxy928IZj95esXqG1RKzyJ`I5Hlwd=yODrAHZSqBTmE2gupyN zFeJi&jX@2Q1Lxr~t@+u)&Yz`jG?B0&?W1 z5fFkJn>UH;s*>*{76T<@|`Cz{P1(az;*NtjDg{e9|?mw3L*roApk3e2aZX?z#K<> z0c?$^pVAMW_>lv=0KkVJJjyZUpVHA4o0mEE$ z{@4lsRT=~1{UUF*lw3|6$=k1z#Z_FJnYUN-zZ~9=&;sM&Oir?Iw#lQj1b+s&%+~FCjvuqU=o+ zdfz}s#8yt#OVGGmmKff(P+XhUZf=>_zRA<0Fh0vyr#>~RJX>J9u4LkUQN!8(5{(y? zAl`LSX3-{s1J>7k0wUL4KHT7rOf6y&;tyhu&et6ztt96Rv|;z@$yMJhL^OSn8rg03 zmr`)ywBo7Ll;_E2yBc=_3`!TPK>885FZ29&qD3rzyg%8U%1-)nS`yzLA_GZU62f`GTXF=OuA+* zw+P2o*DT`(?m8~DESHg<*3sU?8n!4JWqcLQk^6R!3*MIP=WNG55{Wq**hR{Vo|zxf zA>S2mx0Q%eX@2sdbShJ}0sJ&^BunL#8e(dA^C{JUm|#)j07y#PB*f7=@yna^LL3SR zw2n{GTfZQi!@ld;Be>7AWoB~|-MC{G?Popx-xduD$c3NjBc7;2(GDV(*~^wzIB^*C zzINnI$jnva-b{2Tm1QqjRv*2d8rk(EC2r5oQX`DtPeVeQdqpBTZBO`wE-L$_sF4m|!Laof*$%53=3e5}~)#Nr_JS=0iDjMKzGV>z~C> z>v#@%MqU5RcFDuhd?@qv*|YrRw0Epkhu6(-JMj~*#z|f7*&X$}`yN8R`5xz`l!VnW zDF3a81E6~Tf-+J|#YOgrB}~kUB=^~uGaW|d{12U-pVghBX0}Xz&Y{Els`x>1nsR-2 zdc2njOf9er=4eHm#M|Md>r;jXEZ!^7Qqn71=Xi*@37`wGA*)^$> z`n2h~<4b&Z@37G~anx4Lt#W?yKBCW`ax%0(be$qNAR*Igx+fg|RrJJ`NQlGe+ksB+ z_=&KWo>xPOXsSP26_&1&et4OgRb#jx%5L3c92HCJb+WzrLx4+6dau;#{!@QirZ?{i zNark{{YghauKk_VK%ssiBOH#KT6Bck!IG!h)N;s8_02>iJPf{Qq z&j1wJJ2Z5<2Zn-bLl6kOd#e3$`4*5kNkf%HDPheo!X~sP_9xIViu_~X!MMi!jXZ@z ze%%VilT=)ApukHd5iW4L_2X3T`=WCV885C*b8)=6#xO-K=aAN*YN*4Mh)52*`ew_m zI%iwdY0$=I&Zr4%3f`F+$c~e8K&ebp*k-8Nv;j;uNwc9t#`HE)+d`Kra zbB#ZcpKlt8EN_3h`po}xENsp})znG5+@D$OqU?p06#|85R?GCW z%ZpN{-Y7q|Gm1*REQcetv0dBae`;g;g~QW(1Z5cmqTDuBYiMl&m$Q*9Oi#Bf4)%PwzU($#8pKA6vcDK0xG{=0xqI{1K0-aF$brRBvj!RDJV~*woM-v(W^a z`54;&wJw7|{K5=!*VR1mSWbrR2G4^H41-?GlEU2utg!W z9-T_b3=06Szyj)EqFW+UnQj^nT};}o+u`QMbD})Gq8DW@KhQp(lYe`YPWXj@HWF=Z zb<;0Di>NO#>MpqSv53xetvvoPeF~Eybqq2HQx1YtR z{|+VHwJIkcsnKqB^7|w)Txp&xS_il3%S8mtk~tQ3f@bnkZ36tXmB=BX0TtbLeb}g> zt&PBJiof{%uuYYU1aDm0nbhgCV zvQ%EVYvd7$np)Dy1KCry4-gyFU#|`V8s{f6brxOnV+i)Q>J|(*h5SM=4dEnQ7UEvt za0$o-nrX>LB_$6~lLuEDk}}m3w^+JIC>b+-GTGRwY`Uh6tUxw%I;x$IKGc#?-rk#I zVoix5GLN|KHeRvlk{f^a&^vX|fM64~f6egX>B+~c{&6d?J;Vdw|#DIibXqMhz|*y|Oc>s+}pn;)k6yoCcv*N9MYEN7PYdHh09p#|%{ zo7i+7QirCq`%H3;D;yyDQ`F;H78}{G!Y(y?qnYlXqwti@k`ka~>3_p>>vWc8MNMIz zww7C&M9E6U&8x32neZP&ufNq1VTfPVW^ql?cv%+WHQ{jZ7W+1v8!3xCCl*&bpJ9&45AH_(&b6HpmeGDp>(FP6pkwu}`Z=tSJiZEuw&zlQc z#<$8r@w4BcZ;2Cd3V(V`80JZDy{@Oo6@8mEeia_ioy+5Y(}$Yx?NgL7SCaS=XkJ@8 zy1V#hO3!)co)kh6n_*l&iuyS#N;RS0MkVc7+7#F-(=Bh-D-~i-3q@9})CMZpGN(o~ zrZe{$-0Chpw^C2-LNoR4XV(sd3ZH4%TZ~2#sKq^BS>L0y=K*UsG8*LsdQqg#(k#{;fg_2hNs1bDX%gic6;?Mxiyyd$x6B&{zAyY||%KM;Yxh zo~CH%6UddG#QIk%rWEes-E|l58jIZu5HL2%^o?qaiQ|xXFq6K)^1u`=P%Xr$uuTNZ z;gK!vzdfA4FP#=nkf^?uPxSbT@Yh9fP}@Wm$-{YZm{b+}Yi{W$E?Tr%6jJ2o%h%78 z7HQrzeEd>tV{H$H=inM!Ok~NhbE+0k+F7v!dsXs;?0Smsoy2%scFIR*W^FAP%$A4m zVcn(Gl}+`do_8SERcMqCbiW_dRM->qXQ<96UXgWz*j|=)g%vZGe1o}zSsT~hwh6< z(`iJBBWtYz>y7&svT@g(JjI)F$5l5wg~!_|A3JU4jOX8j!+BUjo;O4|tVw#;w7hqF zaPvHTrNH@^HT`QP9I)s7yryaiQW$$T&ZckpO-YcX&!FA~vgyN1DwEzLL`>a5PFncE z9Fabe^z*~hWH04VXKWX@Mt1o?>%0j75?8QK8RP^|vqec6I(Xd$DjLRo^8*9!mk_0R18JeId&>HqV8om@1 ziV|ixfb*85o5aX$qKsZECWJR-{1m!Cuj6*#T+QooZ$TOU?YYt`?>^e;rJIEhtLz&1 zTnd-SD9iNTNRsxOR}*FJV^{A>nfk&SXFi$DM`+m@SiTgy9iA@ReuAOXa+H#(+TCaz-;j zvU%Qvzk3Fc%#G{n^C8{rw2{m9*Oe}C&lFMk9+KK}CYJgRTjuN$S$<*@I|joM{-ObzB9FQQw-HNXFWNTk5BX&ll{Ai?!Ei@iindrNlquUee_~9g16=*k} ze1usxjTwNrR&a=~N0$#5ee~|6y}h$EW^uQ$zyvJ9O3|0JST4yHAabuO`N0GD7xYEu zb(T%aLmr-G7jrCu+Z&Px+RX1p6Q;CQv+@&ocpaG>p6iF1k}FS32$aCoXPa@1=9^Q? z#h-7UXYv-=J_gyp)#{PI;tEH>f%6WeS{DLCX+cm3Yzs=vK_l-ZVDmXT@krkGc$nAY zX$EAlhtE#n(U8K9nf-6|+TYJOusTiPjDx_Uh@b{eTQ>Md_NJDy=z3{^vx>wWJs9Io zEk|}plxEu365#%?qlHjTU8P9>k$Wj96V{K^#snU z@)7i552v{ivgWp>;-9FuCE+?|+P_w~kV3y~m2%frJa9oLW%qJJ*n&PUi6tvrL+cMw zBm-6`@AT3yVajZ`DEOACd?8o5xBIOgytyo_CrXb;C3h~P`@%{!cDbldN2X)6!Uk*3U8ijiC+4t~#!H-{Yn7C~|9z)8%)y1H|jmXbZM=nv60*8fI#$P?3 zZJ!NCV$cT0joNdyoOIG|9WelfHd2O+J`agr35k5bF^ji(2iMl}^Ba^4l41$hD@d%N`0q&D*Bxr%9EGq4ke=vsb5MVZQf~-GcVp2EP3487&wq|+r6~=$YNIbMs#p@ z597)HRh^jwlxRUiL-*IBu9=F7K;HEW=Jtjvy{GDO8f_#`&FBa}FBd=0Oepf<7+n8a zYl8iLuxWJegb=SI@yl2x8&V1Z2%6Mkpv%!1x5x52~meQ)-9f>Rz)!xbG;xog~YtpT)rQ`e2M z*E|df4_&Tc)0u}0t=GiJ&iS_DsIq^x3UV5hFkW9>l|y+)2)!8!iV3>W&yFRfBUCQ& z(EO7_D%vmf>$?W?q+aa9#cVIfYm#PZp4^$m21U-c%7;nZ zA8ZxCd7#vI*;4r++lspd7e)A?REM!=%VNDx&vo*1*Gi9#mQ!Nt?(iGZo|!yK3no{q zeB*l$J_(Z5Tsg$LMC*Ty`3<;6`8(qXaP|KS+lgz|$KgVWdriaX3|jr(mf2FrsN6c- zQ9Ss-I!rC6&4bKL;s$%Aa2V>m5D7@@AoY!SGQ$nz{2-x~^7zFglG7Y654mQ=O&CvQa=XCWW$<%}`65i+gLl#(L?#NF@j2^- z=3_0ZxN*C;wBK)%rR1(xEJUnfVN285{0SOxiTRhS2mkc|r>5EM46NR#M&Qb?b%mjk zi6BPo-j}N~J*Se=?Nk=>w^NMMdaU=@gk8HEUdcqNr`m734E0x5e=Q`&_CD0qJ$;9x zTDi~UnM*Z72|U#<6qoLJKYqE)Kg_VuXKQhCggf|+IIPX4rdb$LERx_5IE4~l##TWeK~(&-$4?shI8Hb}v$@^P45zmytf=^ob`zO3)(W=ibCE@a za!CA%$1l&t?ta9-{rRHcF?9P|P3iZ08B{GGz99$*L6UrgWZLy;>1!-!6;E{Zy$U(? zI>-=TlCD6A;4Wt*ZzwOxiTPTRjAN$$TfGGeT+{!$&kIukrwd%q<`6DLlaRfN-urDI z56cL~6$;*9BN!T})q4D>>?Gb)%jY)?Ij-<2C&{)^CoyW4SVoDSQ&d&7Oj3Nn6v~!m zP-B{rsmoKNJzs0z`S{V4(b!YjIJdaZ(Qoi|KWnU-q-?2$Rwh9|(&#pchQ=!GUzD3M zg1-GK3#q34aLQGV6E7^fx@@Xt&@7^Fk8nb$>ph8Th~U~Wi2kjL1X!SdL6l!x#iJ8M zJncznng`z}SQ2}KZt!V7e2|nZCr@g&1AP?{tsGzosw*7tsEg~9448_|=mrS?WCI+!@Q#(yzT9tw9ck-x^4c(aUE zEU&1$kCSjAA)CpA2giIh>n_@Z{(^f+ZXfbSOdVXwX01iBI`grwsZh;VVjY|NzFdPI z1*2srH>+(`hK7Xi<~FAp+M#byHWh^pK2g_DZ_$UZu?U|zRc2wN-)2vhX>n?r;*yS* zPL?0-*H7CU??eN?V)1zfcpkGXf2%3}cEbdRqjUhx6@pE1>sG{{SCV~veZhQeM2Tq( z0y0FF5N8XgA~PEcd@Dl`6x4vxqcgH7agKrJZxu*5?ANye3D{5YfZLxz?uU}aTN(rJ zZ{Ln-0qiG~&1Y00W{ccl!)Gcjqa4%4?MueYdmPo$FB0Bn^=WtRQP)rn@>ePhlGpJQ z;%ZTf8)Smdu#OJytcaAlDc=m%KUqgb-xR8JBGh4@fTJH7lk@6zNZu>ZH=JlnwBs!Ci;r#{%B?9jf6~QdwELb+_f0_LF$?k6>gTU& z035R;2DMYk+(n#Ve2t@h%*cPM>_Oqb$<)Vq?s@n&D944; zQaoHYxIQFbT)+`0=L|Co?Spr9Wohwph7!7d+InQ&bCDHKdRb=j__jincp_LdV>0Nj z^IN;r1x=$!?%hk(DUPp-DTE|^sC7ESbg96Y)hp?IXa^5?X0d_sY4LdDi-MSzt3_DQ zQ^#QQ*ZR_LqJv*c$pc5~WaA+3PMA%}OUrcL$fd)HlRX6ZBr%d`qlLWKn;<^@rdQX? z?b=_$p&>8vA_I9RoDG&FbwNs&n#g?bcYZrelkOR(`jW@1h;1s!KF4y;taQ#pX>IE! ztVd6;n^uZvN_AJ5bZ}6fRUI>Dvn4&B2@|OG6c+?7%NwMdC%Wj-!71Vq;fgHbw{H{; z;7U=%-j$C-c~E^S1}C5+F2t2W*FE{2FAb^*IS#EGRBSJ$FqygwaGeELFzoE2aL#Jk zS8PivwtjIT+x5Jc{L)8dd4PIk(Z9cY`?0QB=Zy&$Nz~ZF%ciKsPqCHD%LkuvE}0oe z9Ye#vRj8nV^VQEHP%Tl3R0cTe>}5YbTN(y$T^4(EyTNYr(H$8Aui(#1AU83wA$N8G zwAD&%Jns#L#nu{-<-)MgVfRCc4ye>@0lz`Ich^)Oh38`!6#IdE_mo&cD_NJxNeZiU z=9X3vt7*c4-h0F3oxwWM*@T8#n@i#atz{m7tO#vhxzQSCaT^$%pyjlAt;v!R!60C%Axf2=K1^86>1yJ$Vp>cuhQ9p*2CViL6U2?2GYw7$QymCV3Bm;}TZLd>oi% zzW_`dcAtoh=aV9OQkKN_3_E3eCEwQV>@&`u9F$Xrs)aRDy*qc_x`r|}oS%(nAUp=Z zzt)xzzuxGGC$y2aLr%(U9uOW%7EN;J2WXTyL~a~bvZ*u@`C(heh7S$V-_B}}FuSqX z`HVx?ai6MFl0)!mrWt8*(KAMc>wXqOG$u^7^zjSTD<8=llWeXGJ&|tUunY39(6OJn zfQ7@_b10$bV30;;^K=srLY{`Le**W3c*L)#wj5 z7%gE^;HbR_&T1+p7vCi)x@ z%W1XzoEG=c9VwMN@Sg|^veJjBWp!;+a8F4dAb%QQ4vtaGkv23O)*oeSBl<^ACF z2Q16cAd7Rfd{X^oa+K80gIq<(N5E0*Mf%&>r#i(?(rAM5vGjx(CRQf4QpGhmhtX$X zqvgn6np2c=tYQc8x5g;fj1bRY6r)ebD;#i|M881PPn@81@5Q#dshm=?)ZtpaQ-iPn z@hjF%1>Lki=>xbf{^w2kk2|%rnwHit?$kfKQ3IBOqwj(I<0364M);c>HE>G`_H;A1 z1U@T*1cB{*-R!JgKwvj(4|`WD5O5F%`&oOqg21jWfOj+cvMb=@2?E;y-{=7TxAz9> zJnel5!JgLMm|ovJlEE(aKx33300;=M^nn$DFO2~1haj*z2&@5okOTzQ0s&5rU|qnE z6$I7;2?H(7L0}6I*pd)z1&jg=4fs_8E$l#Gd!Ph#bpn9_uX>iS2t#0^dmxIs{e2N!U3d@{KEfOrJo`O{>2|0BNMbSo6Pr+g(EP2cy0V&`Gdn@ z01f^Q?f(+u|Jn~60W9;z>j3t zkG)O?bmRc8jgcltZp}bR2?Q+K(MO`xK`_io4XD!q;Q3b41OYDoM|E0&k36OX$OX`m zLm<#j5AgW^_NzGvkbp<+EP&XAZzW3*V0}5NvjXBEzLl(jWyX{Uze_nFjNzz-4G7?X zZzYU81WbzG9xx*E=!O)S%?<=8Vn@~Al2sn{bE-mY1WFvJ@CHSm`s{}% zoF^$j$EdnLkwwPU3kYok7E{IE3KRARgsfoj|0nwI+jsPSqVvy8BLjT(4Q=o0B8&NK z-Z@zj7z74D0!2dMFrdH-fpI|~T-=1 z`u+zzA%RWC2J}xI<~_jtfn2`V0bm2+XusA$g#|I&)i3RUI>F!jLV*At;BEb_o$w#y z!GIP3X@6-4JfHw({;dv<*{Oc5Ln1Mui@(-EV1VNOTOAT1{5#CRyny8UtsM%1+2ns6 z4+Vq#(GDpD93g&dCx`%oH-4)FDE#|)P@zBJg9`mg1Hjw@gqi(19#j}`x&EyV2sRe} z13n;%@DEr7g)#dm=I@{Ng$O}}{uoaP;LtzXVZz&f?<@ExEW!}LHt}0KVHid)RdmZu*e4#MZANWFHfOK0#o2`mG%d0>`LGKl-9Q% Date: Tue, 14 Mar 2023 17:53:21 +0100 Subject: [PATCH 02/12] Update docs/paper/paper.md Co-authored-by: Helena Zhang --- docs/paper/paper.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/paper/paper.md b/docs/paper/paper.md index f158d93d57..62af5efee9 100644 --- a/docs/paper/paper.md +++ b/docs/paper/paper.md @@ -47,7 +47,7 @@ bibliography: paper.bib Qiskit Experiments is a Python package for designing and running quantum computing experiments with a focus on calibration and characterization of quantum devices. It consists of a general purpose experiments framework which can be used by researchers to rapidly -implement new experiments, and a library of common experiments for calibration, characterization, +implement new experiments and a library of common experiments for calibration, characterization, and verification of quantum devices. The core framework of `Qiskit Experiments` consists of three parts. From 6ce216a9a00baf2e0100604a02b471c581414ba3 Mon Sep 17 00:00:00 2001 From: "Daniel J. Egger" <38065505+eggerdj@users.noreply.github.com> Date: Tue, 14 Mar 2023 17:59:22 +0100 Subject: [PATCH 03/12] Update docs/paper/paper.md --- docs/paper/paper.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/paper/paper.md b/docs/paper/paper.md index 62af5efee9..472857c507 100644 --- a/docs/paper/paper.md +++ b/docs/paper/paper.md @@ -59,7 +59,7 @@ The analysis also attaches its results, such as fit results and figures, to the \autoref{fig:framework} summarizes this framework. Importantly, this framework can interface with services to store, load, and share data. The library of experiments includes common quantum computing experiments such as Randomized -Benchmarking [@Magesan2011], Quantum state and process tomography [@Banaszek1999], Quantum Volume [@Cross2019], +Benchmarking [@Magesan2011], Quantum State and Process Tomography [@Banaszek1999], Quantum Volume [@Cross2019], and gate error amplifying calibration sequences [@Tornow2022]. `Qiskit Experiments` is based on Qiskit [@Qiskit], a general purpose Python library for programming From a8c6f7cf24a0b842ab4cdd012121f2047cd400df Mon Sep 17 00:00:00 2001 From: "Daniel J. Egger" <38065505+eggerdj@users.noreply.github.com> Date: Tue, 14 Mar 2023 17:59:40 +0100 Subject: [PATCH 04/12] Update docs/paper/paper.md Co-authored-by: Helena Zhang --- docs/paper/paper.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/paper/paper.md b/docs/paper/paper.md index 472857c507..96f56acc9e 100644 --- a/docs/paper/paper.md +++ b/docs/paper/paper.md @@ -113,7 +113,7 @@ A depth $d$ QV circuit is successful if it has mean heavy-output probability gre than two-thirds with a confidence level exceeding 0.977, and at least 100 trials have been run. `Qiskit Experiments` only requires a few lines of code to run this standardized yet complex experiment. The analysis classes of existing experiments automatically generate key figures with customizable -visualization options, as exemplified by the QV plot, see \autoref{fig:qv}. +visualization options, as exemplified by the QV plot in \autoref{fig:qv}. ![Example result of a quantum volume measurement carried out with `Qiskit Experiments`. The dashed line shows the two-thirds threshold. From bcca743add5a6f12ed23f247a4763193649c9730 Mon Sep 17 00:00:00 2001 From: "Daniel J. Egger" <38065505+eggerdj@users.noreply.github.com> Date: Tue, 14 Mar 2023 18:00:02 +0100 Subject: [PATCH 05/12] Update docs/paper/paper.md Co-authored-by: Helena Zhang --- docs/paper/paper.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/paper/paper.md b/docs/paper/paper.md index 96f56acc9e..a22585495d 100644 --- a/docs/paper/paper.md +++ b/docs/paper/paper.md @@ -117,7 +117,7 @@ visualization options, as exemplified by the QV plot in \autoref{fig:qv}. ![Example result of a quantum volume measurement carried out with `Qiskit Experiments`. The dashed line shows the two-thirds threshold. -Each dot shows an execution of a quantum circuit. +Each dot shows an execution of a randomized quantum circuit aggregated over many shots. The shaded area is a $2\sigma$ confidence interval. \label{fig:qv}](qv.pdf){ width=75% } From 7cc3d20c85476b161be3864f7a015d73934c569a Mon Sep 17 00:00:00 2001 From: Daniel Egger Date: Tue, 14 Mar 2023 18:54:13 +0100 Subject: [PATCH 06/12] * Updated paper. --- docs/paper/paper.bib | 11 +++++++++++ docs/paper/paper.md | 8 +++----- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/docs/paper/paper.bib b/docs/paper/paper.bib index 733c9b7685..85071318b4 100644 --- a/docs/paper/paper.bib +++ b/docs/paper/paper.bib @@ -12,6 +12,17 @@ @article{Alexander2020 journal = {Quantum Sci. Technol.}, } +@article{Amico2023, + doi = {10.48550/ARXIV.2303.02108}, + url = {https://arxiv.org/abs/2303.02108}, + author = {Amico, M. and Zhang, H. and Jurcevic, P. and Bishop, L. S. and Nation, P. and Wack, A. and McKay, D. C.}, + keywords = {Quantum Physics (quant-ph), FOS: Physical sciences, FOS: Physical sciences}, + title = {Defining Standard Strategies for Quantum Benchmarks}, + publisher = {arXiv}, + year = {2023}, + copyright = {arXiv.org perpetual, non-exclusive license} +} + @article{Cross2019, title = {Validating quantum computers using randomized model circuits}, author = {Cross, A. W. and Bishop, L. S. and Sheldon, S. and Nation, P. D. and Gambetta, J. M.}, diff --git a/docs/paper/paper.md b/docs/paper/paper.md index a22585495d..ef2608ee0f 100644 --- a/docs/paper/paper.md +++ b/docs/paper/paper.md @@ -14,7 +14,6 @@ authors: corresponding: true affiliation: 2 - name: Yael Ben-Haim - orcid: 0000-0000-0000-0000 affiliation: 3 - name: Helena Zhang orcid: 0000-0002-7813-7133 @@ -23,7 +22,6 @@ authors: orcid: 0000-0002-5045-8808 affiliation: 4 - name: Gadi Aleksandrowicz - orcid: 0000-0000-0000-0000 affiliation: 3 - name: Christopher J. Wood orcid: 0000-0001-7606-7349 @@ -37,7 +35,7 @@ affiliations: index: 3 - name: IBM Quantum, IBM T.J. Watson Research Center, Yorktown Heights, NY 10598, USA index: 4 -date: 21 February 2023 +date: 15 February 2023 bibliography: paper.bib --- @@ -93,8 +91,8 @@ maintainers to easily execute characterization and calibration experiments. experiments that integrate with pulse-level control [@Alexander2020]. In addition, `Qiskit Experiments` provides a calibration framework to manage device calibration. This framework is usable with any hardware exposed as a Qiskit backend. -For example, the `Qiskit Experiments` framework has been used to explore measurements without qubit -reset [@Tornow2022], characterize positive operator value measures [@Fischer2022], quantum +For example, the `Qiskit Experiments` framework is used to explore measurements without qubit +reset [@Tornow2022], benchmarking [@Amico2023], characterize positive operator value measures [@Fischer2022], quantum states [@Hamilton2022], and time-evolutions [@Greenaway2022], as well as calibrate gates [@Vazquez2022]. ![ From 34b3076caed5d559421163a463ead97a76b436a3 Mon Sep 17 00:00:00 2001 From: Daniel Egger Date: Thu, 6 Apr 2023 13:56:57 +0200 Subject: [PATCH 07/12] * Update paper --- docs/paper/paper.bib | 23 +++++++++++++++++++++++ docs/paper/paper.md | 15 ++++++++++++--- 2 files changed, 35 insertions(+), 3 deletions(-) diff --git a/docs/paper/paper.bib b/docs/paper/paper.bib index 85071318b4..71b5785c89 100644 --- a/docs/paper/paper.bib +++ b/docs/paper/paper.bib @@ -23,6 +23,20 @@ @article{Amico2023 copyright = {arXiv.org perpetual, non-exclusive license} } +@article{Ball2021, + doi = {10.1088/2058-9565/abdca6}, + url = {https://dx.doi.org/10.1088/2058-9565/abdca6}, + year = {2021}, + month = {sep}, + publisher = {IOP Publishing}, + volume = {6}, + number = {4}, + pages = {044011}, + author = {H. Ball and M. J. Biercuk and A. R. R. Carvalho and J. C. and M. Hush and L. A. De Castro and L. Li and P. J Liebermann and H. J. Slatyer and C. Edmunds and V. Frey and C. Hempel and A. Milne}, + title = {Software tools for quantum control: improving quantum computer performance through noise and error suppression}, + journal = {Quantum Sci. Technol.} +} + @article{Cross2019, title = {Validating quantum computers using randomized model circuits}, author = {Cross, A. W. and Bishop, L. S. and Sheldon, S. and Nation, P. D. and Gambetta, J. M.}, @@ -123,6 +137,15 @@ @Article{Harris2020 url = {https://doi.org/10.1038/s41586-020-2649-2} } +@misc{Pasquale2023, + title = {Towards an open-source framework to perform quantum calibration and characterization}, + author = {A. Pasquale and S. Efthymiou and S. Ramos-Calderer and J. Wilkens and I. Roth and S. Carrazza}, + year = {2023}, + eprint = {2303.10397}, + archivePrefix = {arXiv}, + primaryClass = {quant-ph} +} + @misc{Qiskit, author = {M. S. Anis and A. Mitchell and H. Abraham and A. Offei and R. Agarwal and G. Agliardi and M. Aharoni and I. Y. Akhalwaya and G. Aleksandrowicz and \emph{et al.}}, title = {Qiskit: An Open-source Framework for Quantum Computing}, diff --git a/docs/paper/paper.md b/docs/paper/paper.md index ef2608ee0f..058f8eab10 100644 --- a/docs/paper/paper.md +++ b/docs/paper/paper.md @@ -85,12 +85,21 @@ Quantum development software packages such as Qiskit, ReCirq [@QuantumAI], tKet Forest [@Smith2016] are part of the quantum stack to execute quantum circuits on hardware. They also enable high-level applications that abstract away the quantum hardware. Forest-benchmarking [@Forest] and pyGSTi [@pyGSTi] are tailored towards benchmarking of quantum hardware. +Commercial solutions that provide quantum optimal control as a service now also exist [@Ball2021]. However, there is still a need for open-source software that enables researchers and hardware maintainers to easily execute characterization and calibration experiments. -`Qiskit Experiments` is unique in this perspective as it provides low-level characterization -experiments that integrate with pulse-level control [@Alexander2020]. +Recently, software packages have started to emerge to fill this gap [@Pasquale2023]. +`Qiskit Experiments` is unique in this perspective as it provides open-source low-level +characterization experiments that integrate with pulse-level control [@Alexander2020]. In addition, `Qiskit Experiments` provides a calibration framework to manage device calibration. -This framework is usable with any hardware exposed as a Qiskit backend. +This framework is usable with any hardware exposed as a Qiskit backend. +`Qiskit Experiments` greatly simplifies the execution of complex experiments. +Indeed, the experiments in the library run multiple quantum circuits and complex fitting +but only require a few code lines to run. +In addition, the base framework of `Qiskit Experiments` provides experimentalists a clear interface +to create new experiments. +They must (i) implement the abstract `circuits` method, (ii) define the experiment +options, and optionally (iii) implement the analysis class, if not already present in the library. For example, the `Qiskit Experiments` framework is used to explore measurements without qubit reset [@Tornow2022], benchmarking [@Amico2023], characterize positive operator value measures [@Fischer2022], quantum states [@Hamilton2022], and time-evolutions [@Greenaway2022], as well as calibrate gates [@Vazquez2022]. From c45ad3369aab6189d7b6f8b338f78e2d51446ab4 Mon Sep 17 00:00:00 2001 From: Daniel Egger Date: Thu, 6 Apr 2023 15:55:38 +0200 Subject: [PATCH 08/12] * Added link to manuals. --- docs/paper/paper.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/paper/paper.md b/docs/paper/paper.md index 058f8eab10..5cc6485636 100644 --- a/docs/paper/paper.md +++ b/docs/paper/paper.md @@ -131,6 +131,8 @@ The shaded area is a $2\sigma$ confidence interval. # Documentation `Qiskit Experiments` documentation is available at [https://qiskit.org/documentation/experiments](https://qiskit.org/documentation/experiments). +The documentation also includes [experiment manuals](https://qiskit.org/documentation/experiments/manuals/index.html) +that show how to run experiments such as the Quantum Volume presented above. # Acknowledgements From 8e5f8684da004da72725928d44495286a1819486 Mon Sep 17 00:00:00 2001 From: "Daniel J. Egger" <38065505+eggerdj@users.noreply.github.com> Date: Fri, 7 Apr 2023 15:50:33 +0200 Subject: [PATCH 09/12] Update docs/paper/paper.md Co-authored-by: Helena Zhang --- docs/paper/paper.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/paper/paper.md b/docs/paper/paper.md index 5cc6485636..514c15a5d6 100644 --- a/docs/paper/paper.md +++ b/docs/paper/paper.md @@ -131,7 +131,7 @@ The shaded area is a $2\sigma$ confidence interval. # Documentation `Qiskit Experiments` documentation is available at [https://qiskit.org/documentation/experiments](https://qiskit.org/documentation/experiments). -The documentation also includes [experiment manuals](https://qiskit.org/documentation/experiments/manuals/index.html) +The documentation also includes [experiment manuals](https://qiskit.org/documentation/experiments/manuals) that show how to run experiments such as the Quantum Volume presented above. # Acknowledgements From b85be99392fa9f48c307a62e21cb1f3c0621e949 Mon Sep 17 00:00:00 2001 From: "Daniel J. Egger" <38065505+eggerdj@users.noreply.github.com> Date: Fri, 7 Apr 2023 15:52:30 +0200 Subject: [PATCH 10/12] Update docs/paper/paper.md --- docs/paper/paper.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/paper/paper.md b/docs/paper/paper.md index 514c15a5d6..94b0c8d42a 100644 --- a/docs/paper/paper.md +++ b/docs/paper/paper.md @@ -98,7 +98,7 @@ Indeed, the experiments in the library run multiple quantum circuits and complex but only require a few code lines to run. In addition, the base framework of `Qiskit Experiments` provides experimentalists a clear interface to create new experiments. -They must (i) implement the abstract `circuits` method, (ii) define the experiment +They must (i) define how circuits are constructed, (ii) define the experiment options, and optionally (iii) implement the analysis class, if not already present in the library. For example, the `Qiskit Experiments` framework is used to explore measurements without qubit reset [@Tornow2022], benchmarking [@Amico2023], characterize positive operator value measures [@Fischer2022], quantum From 003630ea361c3adaf80856d1acf4f8e9997fb06d Mon Sep 17 00:00:00 2001 From: "Daniel J. Egger" <38065505+eggerdj@users.noreply.github.com> Date: Fri, 7 Apr 2023 15:55:30 +0200 Subject: [PATCH 11/12] Update docs/paper/paper.md --- docs/paper/paper.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/paper/paper.md b/docs/paper/paper.md index 94b0c8d42a..13d08102c7 100644 --- a/docs/paper/paper.md +++ b/docs/paper/paper.md @@ -100,7 +100,8 @@ In addition, the base framework of `Qiskit Experiments` provides experimentalist to create new experiments. They must (i) define how circuits are constructed, (ii) define the experiment options, and optionally (iii) implement the analysis class, if not already present in the library. -For example, the `Qiskit Experiments` framework is used to explore measurements without qubit +Experiments in the `Qiskit Experiments` library and custom extensions built on top of the +framework have been used to explore measurements without qubit reset [@Tornow2022], benchmarking [@Amico2023], characterize positive operator value measures [@Fischer2022], quantum states [@Hamilton2022], and time-evolutions [@Greenaway2022], as well as calibrate gates [@Vazquez2022]. From 984d8ab7dab62dd0e0860edbcc025cf80fa33d62 Mon Sep 17 00:00:00 2001 From: Daniel Egger Date: Fri, 7 Apr 2023 16:08:10 +0200 Subject: [PATCH 12/12] * Text streamline --- docs/paper/paper.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/paper/paper.md b/docs/paper/paper.md index 13d08102c7..766723f881 100644 --- a/docs/paper/paper.md +++ b/docs/paper/paper.md @@ -91,18 +91,18 @@ maintainers to easily execute characterization and calibration experiments. Recently, software packages have started to emerge to fill this gap [@Pasquale2023]. `Qiskit Experiments` is unique in this perspective as it provides open-source low-level characterization experiments that integrate with pulse-level control [@Alexander2020]. +`Qiskit Experiments` greatly simplifies the execution of complex experiments and is +usable with any hardware exposed as a Qiskit backend. +Indeed, a library provides many experiments which run multiple quantum circuits and complex fitting. +Crucially, each experiment only requires a few code lines to run with `Qiskit Experiments`. +In addition, the base framework of `Qiskit Experiments` provides experimentalists a clear +interface to create new experiments. +They must (i) define how to construct the circuits, (ii) define the experiment options, +and optionally (iii) implement the analysis class, if not already present in the library. In addition, `Qiskit Experiments` provides a calibration framework to manage device calibration. -This framework is usable with any hardware exposed as a Qiskit backend. -`Qiskit Experiments` greatly simplifies the execution of complex experiments. -Indeed, the experiments in the library run multiple quantum circuits and complex fitting -but only require a few code lines to run. -In addition, the base framework of `Qiskit Experiments` provides experimentalists a clear interface -to create new experiments. -They must (i) define how circuits are constructed, (ii) define the experiment -options, and optionally (iii) implement the analysis class, if not already present in the library. Experiments in the `Qiskit Experiments` library and custom extensions built on top of the -framework have been used to explore measurements without qubit -reset [@Tornow2022], benchmarking [@Amico2023], characterize positive operator value measures [@Fischer2022], quantum +framework have been used to explore measurements without qubit reset [@Tornow2022], +benchmarking [@Amico2023], positive operator value measures [@Fischer2022], quantum states [@Hamilton2022], and time-evolutions [@Greenaway2022], as well as calibrate gates [@Vazquez2022]. ![