[bfetch] compress stream chunks #97994
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 29 commits
April 5, 2021 21:50
…tor-integration-in-search-source
…tor-integration-in-search-source
ts improvements more inspector tests
more tests
…-msearch-from-searchsource
…-msearch-from-searchsource
…-msearch-from-searchsource
…-msearch-from-searchsource
…-msearch-from-searchsource
|
Waiting a little longer with merging this, to allow @mshustov try integrating |
|
@elasticmachine merge upstream |
|
@elasticmachine merge upstream |
|
@elasticmachine merge upstream |
|
merge conflict between base and head |
added 4 commits
May 30, 2021 13:09
…to bfetch/compress-stream
💛 Build succeeded, but was flaky
Test FailuresKibana Pipeline / general / Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/spaces/spaces_selection·ts.Spaces app Spaces Spaces Data displays separate data for each space in the default spaceStandard OutStack TraceKibana Pipeline / general / Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/spaces/spaces_selection·ts.Spaces app Spaces Spaces Data "after all" hook in "Spaces Data"Standard OutStack TraceMetrics [docs]Module Count
Public APIs missing comments
Page load bundle
Unknown metric groupsAPI count
References to deprecated APIs
History
To update your PR or re-run it, just comment with: cc @lizozom |
mshustov
reviewed
May 31, 2021
|
|
||
| // Send the payload to the server | ||
| xhr.send(body); | ||
| const stream = fromStreamingXhr(xhr, signal); |
There was a problem hiding this comment.
converting XMLHttpRequest --> Stream --> Observable overcomplecates the code. Let's merge the current PR and consider switching to DecompressionStream after bfetch migration from XMLHttpRequest to fetch API.
There was a problem hiding this comment.
Thanks for approving. I will go ahead and merge. 👍🏻
Any progress with DecompressionStream?
There was a problem hiding this comment.
Yes, I gave it a try, but as converting imperative XMLHttpRequest to Stream, with the following transformation to Observable it's PITA. That's why I suggested switching to DecompressionStream after migrating to fetch with built-in support of Streaming API.
lizozom
added
the
auto-backport
💔 Backport failed
To backport manually run: |
lizozom
added a commit
to lizozom/kibana
that referenced
this pull request
Jun 1, 2021
* Move inspector adapter integration into search source * docs and ts * Move other bucket to search source * test ts + delete unused tabilfy function * hierarchical param in aggconfig. ts improvements more inspector tests * fix jest * separate inspect more tests * jest * inspector * Error handling and more tests * put the fun in functional tests * delete client side legacy msearch code * ts * override to sync search in search source * delete more legacy code * ts * delete moarrrr * deflate bfetch chunks * update tests use only zlib * ts * extract getInflatedResponse * tests * Use fflate in attempt to reduce package size * use node streams, fflate and hex encoding. * DISABLE_SEARCH_COMPRESSION UI Settings Use base64 and async compression * i18n * Code review Use custom header for compression Promisify once * use custom headers * Update jest * fix tests * code review, baby! * integration * tests * limit * limit * limit Co-authored-by: Kibana Machine <[email protected]> # Conflicts: # packages/kbn-optimizer/limits.yml
|
Author of 3.5kB CommonJS build of unzlibSync"use strict";var r=Uint8Array,n=Uint16Array,a=Uint32Array,e=new r([0,0,0,0,0,0,0,0,1,1,1,1,2,2,2,2,3,3,3,3,4,4,4,4,5,5,5,5,0,0,0,0]),t=new r([0,0,0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10,11,11,12,12,13,13,0,0]),i=new r([16,17,18,0,8,7,9,6,10,5,11,4,12,3,13,2,14,1,15]),f=function(r,e){for(var t=new n(31),i=0;i<31;++i)t[i]=e+=1<<r[i-1];var f=new a(t[30]);for(i=1;i<30;++i)for(var o=t[i];o<t[i+1];++o)f[o]=o-t[i]<<5|i;return[t,f]},o=f(e,2),v=o[0],u=o[1];v[28]=258,u[258]=28;for(var l=f(t,0),c=l[0],s=(l[1],new n(32768)),w=0;w<32768;++w){var b=(43690&w)>>>1|(21845&w)<<1;b=(61680&(b=(52428&b)>>>2|(13107&b)<<2))>>>4|(3855&b)<<4,s[w]=((65280&b)>>>8|(255&b)<<8)>>>1}var d=function(r,a,e){for(var t=r.length,i=0,f=new n(a);i<t;++i)++f[r[i]-1];var o,v=new n(a);for(i=0;i<a;++i)v[i]=v[i-1]+f[i-1]<<1;if(e){o=new n(1<<a);var u=15-a;for(i=0;i<t;++i)if(r[i])for(var l=i<<4|r[i],c=a-r[i],w=v[r[i]-1]++<<c,b=w|(1<<c)-1;w<=b;++w)o[s[w]>>>u]=l}else for(o=new n(t),i=0;i<t;++i)r[i]&&(o[i]=s[v[r[i]-1]++]>>>15-r[i]);return o},h=new r(288);for(w=0;w<144;++w)h[w]=8;for(w=144;w<256;++w)h[w]=9;for(w=256;w<280;++w)h[w]=7;for(w=280;w<288;++w)h[w]=8;var p=new r(32);for(w=0;w<32;++w)p[w]=5;var y=d(h,9,1),g=d(p,5,1),k=function(r){for(var n=r[0],a=1;a<r.length;++a)r[a]>n&&(n=r[a]);return n},E=function(r,n,a){var e=n/8|0;return(r[e]|r[e+1]<<8)>>(7&n)&a},m=function(r,n){var a=n/8|0;return(r[a]|r[a+1]<<8|r[a+2]<<16)>>(7&n)},x=function(r){return(r/8|0)+(7&r&&1)},z=function(e,t,i){(null==t||t<0)&&(t=0),(null==i||i>e.length)&&(i=e.length);var f=new(e instanceof n?n:e instanceof a?a:r)(i-t);return f.set(e.subarray(t,i)),f},A=["unexpected EOF","invalid block type","invalid length/literal","invalid distance"],S=function(r,n,a){var e=new Error(n||A[r]);if(e.code=r,Error.captureStackTrace&&Error.captureStackTrace(e,S),!a)throw e;return e},U=function(n,a,f){var o=n.length;if(!o||f&&!f.l&&o<5)return a||new r(0);var u=!a||f,l=!f||f.i;f||(f={}),a||(a=new r(3*o));var s=function(n){var e=a.length;if(n>e){var t=new r(Math.max(2*e,n));t.set(a),a=t}},w=f.f||0,b=f.p||0,h=f.b||0,p=f.l,A=f.d,U=f.m,T=f.n,F=8*o;do{if(!p){f.f=w=E(n,b,1);var M=E(n,b+1,3);if(b+=3,!M){var O=n[(L=x(b)+4)-4]|n[L-3]<<8,j=L+O;if(j>o){l&&S(0);break}u&&s(h+O),a.set(n.subarray(L,j),h),f.b=h+=O,f.p=b=8*j;continue}if(1==M)p=y,A=g,U=9,T=5;else if(2==M){var q=E(n,b,31)+257,B=E(n,b+10,15)+4,C=q+E(n,b+5,31)+1;b+=14;for(var D=new r(C),G=new r(19),H=0;H<B;++H)G[i[H]]=E(n,b+3*H,7);b+=3*B;var I=k(G),J=(1<<I)-1,K=d(G,I,1);for(H=0;H<C;){var L,N=K[E(n,b,J)];if(b+=15&N,(L=N>>>4)<16)D[H++]=L;else{var P=0,Q=0;for(16==L?(Q=3+E(n,b,3),b+=2,P=D[H-1]):17==L?(Q=3+E(n,b,7),b+=3):18==L&&(Q=11+E(n,b,127),b+=7);Q--;)D[H++]=P}}var R=D.subarray(0,q),V=D.subarray(q);U=k(R),T=k(V),p=d(R,U,1),A=d(V,T,1)}else S(1);if(b>F){l&&S(0);break}}u&&s(h+131072);for(var W=(1<<U)-1,X=(1<<T)-1,Y=b;;Y=b){var Z=(P=p[m(n,b)&W])>>>4;if((b+=15&P)>F){l&&S(0);break}if(P||S(2),Z<256)a[h++]=Z;else{if(256==Z){Y=b,p=null;break}var $=Z-254;if(Z>264){var _=e[H=Z-257];$=E(n,b,(1<<_)-1)+v[H],b+=_}var rr=A[m(n,b)&X],nr=rr>>>4;rr||S(3),b+=15&rr;V=c[nr];if(nr>3){_=t[nr];V+=m(n,b)&(1<<_)-1,b+=_}if(b>F){l&&S(0);break}u&&s(h+131072);for(var ar=h+$;h<ar;h+=4)a[h]=a[h-V],a[h+1]=a[h+1-V],a[h+2]=a[h+2-V],a[h+3]=a[h+3-V];h=ar}}f.l=p,f.p=Y,f.b=h,p&&(w=1,f.m=U,f.d=A,f.n=T)}while(!w);return h==a.length?a:z(a,0,h)},T=function(r){(8!=(15&r[0])||r[0]>>>4>7||(r[0]<<8|r[1])%31)&&S(6,"invalid zlib data"),32&r[1]&&S(6,"invalid zlib data: preset dictionaries not supported")};function F(r,n){return U((T(r),r.subarray(2,-4)),n)}exports.unzlibSync=F3.2kB ESM build of unzlibSync"use strict";var r=Uint8Array,n=Uint16Array,a=Uint32Array,e=new r([0,0,0,0,0,0,0,0,1,1,1,1,2,2,2,2,3,3,3,3,4,4,4,4,5,5,5,5,0,0,0,0]),t=new r([0,0,0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10,11,11,12,12,13,13,0,0]),i=new r([16,17,18,0,8,7,9,6,10,5,11,4,12,3,13,2,14,1,15]),f=function(r,e){for(var t=new n(31),i=0;i<31;++i)t[i]=e+=1<<r[i-1];var f=new a(t[30]);for(i=1;i<30;++i)for(var o=t[i];o<t[i+1];++o)f[o]=o-t[i]<<5|i;return[t,f]},o=f(e,2),v=o[0],u=o[1];v[28]=258,u[258]=28;for(var l=f(t,0),c=l[0],s=(l[1],new n(32768)),w=0;w<32768;++w){var b=(43690&w)>>>1|(21845&w)<<1;b=(61680&(b=(52428&b)>>>2|(13107&b)<<2))>>>4|(3855&b)<<4,s[w]=((65280&b)>>>8|(255&b)<<8)>>>1}var d=function(r,a,e){for(var t=r.length,i=0,f=new n(a);i<t;++i)++f[r[i]-1];var o,v=new n(a);for(i=0;i<a;++i)v[i]=v[i-1]+f[i-1]<<1;if(e){o=new n(1<<a);var u=15-a;for(i=0;i<t;++i)if(r[i])for(var l=i<<4|r[i],c=a-r[i],w=v[r[i]-1]++<<c,b=w|(1<<c)-1;w<=b;++w)o[s[w]>>>u]=l}else for(o=new n(t),i=0;i<t;++i)r[i]&&(o[i]=s[v[r[i]-1]++]>>>15-r[i]);return o},h=new r(288);for(w=0;w<144;++w)h[w]=8;for(w=144;w<256;++w)h[w]=9;for(w=256;w<280;++w)h[w]=7;for(w=280;w<288;++w)h[w]=8;var p=new r(32);for(w=0;w<32;++w)p[w]=5;var y=d(h,9,1),g=d(p,5,1),k=function(r){for(var n=r[0],a=1;a<r.length;++a)r[a]>n&&(n=r[a]);return n},E=function(r,n,a){var e=n/8|0;return(r[e]|r[e+1]<<8)>>(7&n)&a},m=function(r,n){var a=n/8|0;return(r[a]|r[a+1]<<8|r[a+2]<<16)>>(7&n)},x=function(r){return(r/8|0)+(7&r&&1)},z=function(e,t,i){(null==t||t<0)&&(t=0),(null==i||i>e.length)&&(i=e.length);var f=new(e instanceof n?n:e instanceof a?a:r)(i-t);return f.set(e.subarray(t,i)),f},A=["unexpected EOF","invalid block type","invalid length/literal","invalid distance"],S=function(r,n,a){var e=new Error(n||A[r]);if(e.code=r,Error.captureStackTrace&&Error.captureStackTrace(e,S),!a)throw e;return e},U=function(n,a,f){var o=n.length;if(!o||f&&!f.l&&o<5)return a||new r(0);var u=!a||f,l=!f||f.i;f||(f={}),a||(a=new r(3*o));var s=function(n){var e=a.length;if(n>e){var t=new r(Math.max(2*e,n));t.set(a),a=t}},w=f.f||0,b=f.p||0,h=f.b||0,p=f.l,A=f.d,U=f.m,T=f.n,F=8*o;do{if(!p){f.f=w=E(n,b,1);var M=E(n,b+1,3);if(b+=3,!M){var O=n[(L=x(b)+4)-4]|n[L-3]<<8,j=L+O;if(j>o){l&&S(0);break}u&&s(h+O),a.set(n.subarray(L,j),h),f.b=h+=O,f.p=b=8*j;continue}if(1==M)p=y,A=g,U=9,T=5;else if(2==M){var q=E(n,b,31)+257,B=E(n,b+10,15)+4,C=q+E(n,b+5,31)+1;b+=14;for(var D=new r(C),G=new r(19),H=0;H<B;++H)G[i[H]]=E(n,b+3*H,7);b+=3*B;var I=k(G),J=(1<<I)-1,K=d(G,I,1);for(H=0;H<C;){var L,N=K[E(n,b,J)];if(b+=15&N,(L=N>>>4)<16)D[H++]=L;else{var P=0,Q=0;for(16==L?(Q=3+E(n,b,3),b+=2,P=D[H-1]):17==L?(Q=3+E(n,b,7),b+=3):18==L&&(Q=11+E(n,b,127),b+=7);Q--;)D[H++]=P}}var R=D.subarray(0,q),V=D.subarray(q);U=k(R),T=k(V),p=d(R,U,1),A=d(V,T,1)}else S(1);if(b>F){l&&S(0);break}}u&&s(h+131072);for(var W=(1<<U)-1,X=(1<<T)-1,Y=b;;Y=b){var Z=(P=p[m(n,b)&W])>>>4;if((b+=15&P)>F){l&&S(0);break}if(P||S(2),Z<256)a[h++]=Z;else{if(256==Z){Y=b,p=null;break}var $=Z-254;if(Z>264){var _=e[H=Z-257];$=E(n,b,(1<<_)-1)+v[H],b+=_}var rr=A[m(n,b)&X],nr=rr>>>4;rr||S(3),b+=15&rr;V=c[nr];if(nr>3){_=t[nr];V+=m(n,b)&(1<<_)-1,b+=_}if(b>F){l&&S(0);break}u&&s(h+131072);for(var ar=h+$;h<ar;h+=4)a[h]=a[h-V],a[h+1]=a[h+1-V],a[h+2]=a[h+2-V],a[h+3]=a[h+3-V];h=ar}}f.l=p,f.p=Y,f.b=h,p&&(w=1,f.m=U,f.d=A,f.n=T)}while(!w);return h==a.length?a:z(a,0,h)},T=function(r){(8!=(15&r[0])||r[0]>>>4>7||(r[0]<<8|r[1])%31)&&S(6,"invalid zlib data"),32&r[1]&&S(6,"invalid zlib data: preset dictionaries not supported")};function F(r,n){return U((T(r),r.subarray(2,-4)),n)};export{F as unzlibSync}Note that 3.9kB CommonJS build of unzlibSync and strFromU8"use strict";var r=Uint8Array,n=Uint16Array,e=Uint32Array,a=new r([0,0,0,0,0,0,0,0,1,1,1,1,2,2,2,2,3,3,3,3,4,4,4,4,5,5,5,5,0,0,0,0]),t=new r([0,0,0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10,11,11,12,12,13,13,0,0]),o=new r([16,17,18,0,8,7,9,6,10,5,11,4,12,3,13,2,14,1,15]),f=function(r,a){for(var t=new n(31),o=0;o<31;++o)t[o]=a+=1<<r[o-1];var f=new e(t[30]);for(o=1;o<30;++o)for(var i=t[o];i<t[o+1];++i)f[i]=i-t[o]<<5|o;return[t,f]},i=f(a,2),u=i[0],v=i[1];u[28]=258,v[258]=28;for(var l=f(t,0),c=l[0],d=(l[1],new n(32768)),s=0;s<32768;++s){var w=(43690&s)>>>1|(21845&s)<<1;w=(61680&(w=(52428&w)>>>2|(13107&w)<<2))>>>4|(3855&w)<<4,d[s]=((65280&w)>>>8|(255&w)<<8)>>>1}var h=function(r,e,a){for(var t=r.length,o=0,f=new n(e);o<t;++o)++f[r[o]-1];var i,u=new n(e);for(o=0;o<e;++o)u[o]=u[o-1]+f[o-1]<<1;if(a){i=new n(1<<e);var v=15-e;for(o=0;o<t;++o)if(r[o])for(var l=o<<4|r[o],c=e-r[o],s=u[r[o]-1]++<<c,w=s|(1<<c)-1;s<=w;++s)i[d[s]>>>v]=l}else for(i=new n(t),o=0;o<t;++o)r[o]&&(i[o]=d[u[r[o]-1]++]>>>15-r[o]);return i},b=new r(288);for(s=0;s<144;++s)b[s]=8;for(s=144;s<256;++s)b[s]=9;for(s=256;s<280;++s)b[s]=7;for(s=280;s<288;++s)b[s]=8;var g=new r(32);for(s=0;s<32;++s)g[s]=5;var p=h(b,9,1),y=h(g,5,1),C=function(r){for(var n=r[0],e=1;e<r.length;++e)r[e]>n&&(n=r[e]);return n},m=function(r,n,e){var a=n/8|0;return(r[a]|r[a+1]<<8)>>(7&n)&e},k=function(r,n){var e=n/8|0;return(r[e]|r[e+1]<<8|r[e+2]<<16)>>(7&n)},S=function(r){return(r/8|0)+(7&r&&1)},x=function(a,t,o){(null==t||t<0)&&(t=0),(null==o||o>a.length)&&(o=a.length);var f=new(a instanceof n?n:a instanceof e?e:r)(o-t);return f.set(a.subarray(t,o)),f},E=["unexpected EOF","invalid block type","invalid length/literal","invalid distance"],T=function(r,n,e){var a=new Error(n||E[r]);if(a.code=r,Error.captureStackTrace&&Error.captureStackTrace(a,T),!e)throw a;return a},U=function(n,e,f){var i=n.length;if(!i||f&&!f.l&&i<5)return e||new r(0);var v=!e||f,l=!f||f.i;f||(f={}),e||(e=new r(3*i));var d=function(n){var a=e.length;if(n>a){var t=new r(Math.max(2*a,n));t.set(e),e=t}},s=f.f||0,w=f.p||0,b=f.b||0,g=f.l,E=f.d,U=f.m,z=f.n,A=8*i;do{if(!g){f.f=s=m(n,w,1);var F=m(n,w+1,3);if(w+=3,!F){var M=n[(L=S(w)+4)-4]|n[L-3]<<8,O=L+M;if(O>i){l&&T(0);break}v&&d(b+M),e.set(n.subarray(L,O),b),f.b=b+=M,f.p=w=8*O;continue}if(1==F)g=p,E=y,U=9,z=5;else if(2==F){var j=m(n,w,31)+257,q=m(n,w+10,15)+4,B=j+m(n,w+5,31)+1;w+=14;for(var D=new r(B),G=new r(19),H=0;H<q;++H)G[o[H]]=m(n,w+3*H,7);w+=3*q;var I=C(G),J=(1<<I)-1,K=h(G,I,1);for(H=0;H<B;){var L,N=K[m(n,w,J)];if(w+=15&N,(L=N>>>4)<16)D[H++]=L;else{var P=0,Q=0;for(16==L?(Q=3+m(n,w,3),w+=2,P=D[H-1]):17==L?(Q=3+m(n,w,7),w+=3):18==L&&(Q=11+m(n,w,127),w+=7);Q--;)D[H++]=P}}var R=D.subarray(0,j),V=D.subarray(j);U=C(R),z=C(V),g=h(R,U,1),E=h(V,z,1)}else T(1);if(w>A){l&&T(0);break}}v&&d(b+131072);for(var W=(1<<U)-1,X=(1<<z)-1,Y=w;;Y=w){var Z=(P=g[k(n,w)&W])>>>4;if((w+=15&P)>A){l&&T(0);break}if(P||T(2),Z<256)e[b++]=Z;else{if(256==Z){Y=w,g=null;break}var $=Z-254;if(Z>264){var _=a[H=Z-257];$=m(n,w,(1<<_)-1)+u[H],w+=_}var rr=E[k(n,w)&X],nr=rr>>>4;rr||T(3),w+=15&rr;V=c[nr];if(nr>3){_=t[nr];V+=k(n,w)&(1<<_)-1,w+=_}if(w>A){l&&T(0);break}v&&d(b+131072);for(var er=b+$;b<er;b+=4)e[b]=e[b-V],e[b+1]=e[b+1-V],e[b+2]=e[b+2-V],e[b+3]=e[b+3-V];b=er}}f.l=g,f.p=Y,f.b=b,g&&(s=1,f.m=U,f.d=E,f.n=z)}while(!s);return b==e.length?e:x(e,0,b)},z=function(r){(8!=(15&r[0])||r[0]>>>4>7||(r[0]<<8|r[1])%31)&&T(6,"invalid zlib data"),32&r[1]&&T(6,"invalid zlib data: preset dictionaries not supported")};function A(r,n){return U((z(r),r.subarray(2,-4)),n)}var F=function(r){for(var n="",e=0;;){var a=r[e++],t=(a>127)+(a>223)+(a>239);if(e+t>r.length)return[n,x(r,e-1)];t?3==t?(a=((15&a)<<18|(63&r[e++])<<12|(63&r[e++])<<6|63&r[e++])-65536,n+=String.fromCharCode(55296|a>>10,56320|1023&a)):n+=1&t?String.fromCharCode((31&a)<<6|63&r[e++]):String.fromCharCode((15&a)<<12|(63&r[e++])<<6|63&r[e++]):n+=String.fromCharCode(a)}};var M=null!=typeof TextEncoder&&new TextEncoder;function O(r,n){if(n){for(var e="",a=0;a<r.length;a+=16384)e+=String.fromCharCode.apply(null,r.subarray(a,a+16384));return e}if(M)return M.decode(r);var t=F(r),o=t[0];return t[1].length&&T(8),o}exports.strFromU8=O;exports.unzlibSync=A3.9kB ESM build of unzlibSync and strFromU8"use strict";var r=Uint8Array,n=Uint16Array,e=Uint32Array,a=new r([0,0,0,0,0,0,0,0,1,1,1,1,2,2,2,2,3,3,3,3,4,4,4,4,5,5,5,5,0,0,0,0]),t=new r([0,0,0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10,11,11,12,12,13,13,0,0]),o=new r([16,17,18,0,8,7,9,6,10,5,11,4,12,3,13,2,14,1,15]),f=function(r,a){for(var t=new n(31),o=0;o<31;++o)t[o]=a+=1<<r[o-1];var f=new e(t[30]);for(o=1;o<30;++o)for(var i=t[o];i<t[o+1];++i)f[i]=i-t[o]<<5|o;return[t,f]},i=f(a,2),u=i[0],v=i[1];u[28]=258,v[258]=28;for(var l=f(t,0),c=l[0],d=(l[1],new n(32768)),s=0;s<32768;++s){var w=(43690&s)>>>1|(21845&s)<<1;w=(61680&(w=(52428&w)>>>2|(13107&w)<<2))>>>4|(3855&w)<<4,d[s]=((65280&w)>>>8|(255&w)<<8)>>>1}var h=function(r,e,a){for(var t=r.length,o=0,f=new n(e);o<t;++o)++f[r[o]-1];var i,u=new n(e);for(o=0;o<e;++o)u[o]=u[o-1]+f[o-1]<<1;if(a){i=new n(1<<e);var v=15-e;for(o=0;o<t;++o)if(r[o])for(var l=o<<4|r[o],c=e-r[o],s=u[r[o]-1]++<<c,w=s|(1<<c)-1;s<=w;++s)i[d[s]>>>v]=l}else for(i=new n(t),o=0;o<t;++o)r[o]&&(i[o]=d[u[r[o]-1]++]>>>15-r[o]);return i},b=new r(288);for(s=0;s<144;++s)b[s]=8;for(s=144;s<256;++s)b[s]=9;for(s=256;s<280;++s)b[s]=7;for(s=280;s<288;++s)b[s]=8;var g=new r(32);for(s=0;s<32;++s)g[s]=5;var p=h(b,9,1),y=h(g,5,1),C=function(r){for(var n=r[0],e=1;e<r.length;++e)r[e]>n&&(n=r[e]);return n},m=function(r,n,e){var a=n/8|0;return(r[a]|r[a+1]<<8)>>(7&n)&e},k=function(r,n){var e=n/8|0;return(r[e]|r[e+1]<<8|r[e+2]<<16)>>(7&n)},S=function(r){return(r/8|0)+(7&r&&1)},x=function(a,t,o){(null==t||t<0)&&(t=0),(null==o||o>a.length)&&(o=a.length);var f=new(a instanceof n?n:a instanceof e?e:r)(o-t);return f.set(a.subarray(t,o)),f},E=["unexpected EOF","invalid block type","invalid length/literal","invalid distance"],T=function(r,n,e){var a=new Error(n||E[r]);if(a.code=r,Error.captureStackTrace&&Error.captureStackTrace(a,T),!e)throw a;return a},U=function(n,e,f){var i=n.length;if(!i||f&&!f.l&&i<5)return e||new r(0);var v=!e||f,l=!f||f.i;f||(f={}),e||(e=new r(3*i));var d=function(n){var a=e.length;if(n>a){var t=new r(Math.max(2*a,n));t.set(e),e=t}},s=f.f||0,w=f.p||0,b=f.b||0,g=f.l,E=f.d,U=f.m,z=f.n,A=8*i;do{if(!g){f.f=s=m(n,w,1);var F=m(n,w+1,3);if(w+=3,!F){var M=n[(L=S(w)+4)-4]|n[L-3]<<8,O=L+M;if(O>i){l&&T(0);break}v&&d(b+M),e.set(n.subarray(L,O),b),f.b=b+=M,f.p=w=8*O;continue}if(1==F)g=p,E=y,U=9,z=5;else if(2==F){var j=m(n,w,31)+257,q=m(n,w+10,15)+4,B=j+m(n,w+5,31)+1;w+=14;for(var D=new r(B),G=new r(19),H=0;H<q;++H)G[o[H]]=m(n,w+3*H,7);w+=3*q;var I=C(G),J=(1<<I)-1,K=h(G,I,1);for(H=0;H<B;){var L,N=K[m(n,w,J)];if(w+=15&N,(L=N>>>4)<16)D[H++]=L;else{var P=0,Q=0;for(16==L?(Q=3+m(n,w,3),w+=2,P=D[H-1]):17==L?(Q=3+m(n,w,7),w+=3):18==L&&(Q=11+m(n,w,127),w+=7);Q--;)D[H++]=P}}var R=D.subarray(0,j),V=D.subarray(j);U=C(R),z=C(V),g=h(R,U,1),E=h(V,z,1)}else T(1);if(w>A){l&&T(0);break}}v&&d(b+131072);for(var W=(1<<U)-1,X=(1<<z)-1,Y=w;;Y=w){var Z=(P=g[k(n,w)&W])>>>4;if((w+=15&P)>A){l&&T(0);break}if(P||T(2),Z<256)e[b++]=Z;else{if(256==Z){Y=w,g=null;break}var $=Z-254;if(Z>264){var _=a[H=Z-257];$=m(n,w,(1<<_)-1)+u[H],w+=_}var rr=E[k(n,w)&X],nr=rr>>>4;rr||T(3),w+=15&rr;V=c[nr];if(nr>3){_=t[nr];V+=k(n,w)&(1<<_)-1,w+=_}if(w>A){l&&T(0);break}v&&d(b+131072);for(var er=b+$;b<er;b+=4)e[b]=e[b-V],e[b+1]=e[b+1-V],e[b+2]=e[b+2-V],e[b+3]=e[b+3-V];b=er}}f.l=g,f.p=Y,f.b=b,g&&(s=1,f.m=U,f.d=E,f.n=z)}while(!s);return b==e.length?e:x(e,0,b)},z=function(r){(8!=(15&r[0])||r[0]>>>4>7||(r[0]<<8|r[1])%31)&&T(6,"invalid zlib data"),32&r[1]&&T(6,"invalid zlib data: preset dictionaries not supported")};function A(r,n){return U((z(r),r.subarray(2,-4)),n)}var F=function(r){for(var n="",e=0;;){var a=r[e++],t=(a>127)+(a>223)+(a>239);if(e+t>r.length)return[n,x(r,e-1)];t?3==t?(a=((15&a)<<18|(63&r[e++])<<12|(63&r[e++])<<6|63&r[e++])-65536,n+=String.fromCharCode(55296|a>>10,56320|1023&a)):n+=1&t?String.fromCharCode((31&a)<<6|63&r[e++]):String.fromCharCode((15&a)<<12|(63&r[e++])<<6|63&r[e++]):n+=String.fromCharCode(a)}};var M=null!=typeof TextEncoder&&new TextEncoder;function O(r,n){if(n){for(var e="",a=0;a<r.length;a+=16384)e+=String.fromCharCode.apply(null,r.subarray(a,a+16384));return e}if(M)return M.decode(r);var t=F(r),o=t[0];return t[1].length&&T(8),o};export{O as strFromU8,A as unzlibSync}BTW, the reason tree shaking isn't working ATM is the // In the shared deps file
import { unzlibSync, strFromU8 } from 'fflate';
export const Fflate = { unzlibSync, strFromU8 };I've found |
lizozom
added a commit
that referenced
this pull request
Jun 1, 2021
* Move inspector adapter integration into search source * docs and ts * Move other bucket to search source * test ts + delete unused tabilfy function * hierarchical param in aggconfig. ts improvements more inspector tests * fix jest * separate inspect more tests * jest * inspector * Error handling and more tests * put the fun in functional tests * delete client side legacy msearch code * ts * override to sync search in search source * delete more legacy code * ts * delete moarrrr * deflate bfetch chunks * update tests use only zlib * ts * extract getInflatedResponse * tests * Use fflate in attempt to reduce package size * use node streams, fflate and hex encoding. * DISABLE_SEARCH_COMPRESSION UI Settings Use base64 and async compression * i18n * Code review Use custom header for compression Promisify once * use custom headers * Update jest * fix tests * code review, baby! * integration * tests * limit * limit * limit Co-authored-by: Kibana Machine <[email protected]> # Conflicts: # packages/kbn-optimizer/limits.yml
9 tasks
lizozom
added a commit
to lizozom/kibana
that referenced
this pull request
Jun 30, 2021
* Move inspector adapter integration into search source * docs and ts * Move other bucket to search source * test ts + delete unused tabilfy function * hierarchical param in aggconfig. ts improvements more inspector tests * fix jest * separate inspect more tests * jest * inspector * Error handling and more tests * put the fun in functional tests * delete client side legacy msearch code * ts * override to sync search in search source * delete more legacy code * ts * delete moarrrr * deflate bfetch chunks * update tests use only zlib * ts * extract getInflatedResponse * tests * Use fflate in attempt to reduce package size * use node streams, fflate and hex encoding. * DISABLE_SEARCH_COMPRESSION UI Settings Use base64 and async compression * i18n * Code review Use custom header for compression Promisify once * use custom headers * Update jest * fix tests * code review, baby! * integration * tests * limit * limit * limit Co-authored-by: Kibana Machine <[email protected]> # Conflicts: # packages/kbn-optimizer/limits.yml # test/api_integration/apis/search/bsearch.ts
lizozom
added a commit
that referenced
this pull request
Jun 30, 2021
* Move inspector adapter integration into search source * docs and ts * Move other bucket to search source * test ts + delete unused tabilfy function * hierarchical param in aggconfig. ts improvements more inspector tests * fix jest * separate inspect more tests * jest * inspector * Error handling and more tests * put the fun in functional tests * delete client side legacy msearch code * ts * override to sync search in search source * delete more legacy code * ts * delete moarrrr * deflate bfetch chunks * update tests use only zlib * ts * extract getInflatedResponse * tests * Use fflate in attempt to reduce package size * use node streams, fflate and hex encoding. * DISABLE_SEARCH_COMPRESSION UI Settings Use base64 and async compression * i18n * Code review Use custom header for compression Promisify once * use custom headers * Update jest * fix tests * code review, baby! * integration * tests * limit * limit * limit Co-authored-by: Kibana Machine <[email protected]> # Conflicts: # packages/kbn-optimizer/limits.yml # test/api_integration/apis/search/bsearch.ts
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Resolves #94998
This bug was introduced by switching to use
bfetchfor batching in 7.11.Since bfetch chunks are not compressed, we started transferring uncompressed data, whenever the
data.searchapi was used (low level search + search source).This PR encodes each chunk before sending it.
Testing
Compression is now applied by default to anywhere where
bfetchis used: Discover, Visualizations, Dashboard, Canvas, etc.If you wish to disable compression (for example for debugging), you can toggle the
DISABLE_BFETCH_COMPRESSIONUI setting.Toggling this setting on and off everything should work the same, but data transfer size should drastically decrease.
Release note
Fixes bug where
bfetchresponses were not compressed.Introduces new UI Setting
DISABLE_BFETCH_COMPRESSIONfor better debuggability.Checklist
Delete any items that are not applicable to this PR.
For maintainers