feat(middleware-compression): add middleware for request compression

[trivikr]

Issue

• Internal JS-4242
• Taking over feat(middleware-compression): add middleware for request compression #5618

Description

Adds middleware for request compression

Testing

Test compression/decompression of string in Node.js

import { toUint8Array } from "@smithy/util-utf8";
import { promisify } from "util";
import zlib from "zlib";

const gzipAsync = promisify(zlib.gzip);
const unzipAsync = promisify(zlib.unzip);

const body = "Hello World!";

const compressedBuffer = await gzipAsync(toUint8Array(body || ""));
const compressedUint8Array = toUint8Array(compressedBuffer);

const uncompressedBuffer = await unzipAsync(compressedUint8Array);
const uncompressedString = new TextDecoder().decode(uncompressedBuffer);

console.log(body === uncompressedString);

Test compression/decompression of stream in Node.js

import { createGzip, createUnzip } from "zlib";
import { Readable } from "stream";

const getGenerator = (chunks) =>
  async function* generator() {
    for (const chunk of chunks) {
      yield chunk;
    }
  };

const inputStream = Readable.from(getGenerator(["hello", "world"])());
// This will be the code in compressStream.
const compressedStream = inputStream.pipe(createGzip());

const decompressedStream = compressedStream.pipe(createUnzip());
decompressedStream.on("data", (chunk) => console.log(chunk.toString())); // prints helloworld

Test compression/decompression of string in browser

<script type="module">
  import { gzip } from "https://cdn.skypack.dev/[email protected]?min";

  const inputString = "Hello World!";
  const inputUint8Array = new TextEncoder().encode(inputString);

  const compressedUint8Array = await new Promise((resolve, reject) => {
    gzip(inputUint8Array, (err, compressedData) => {
      if (err) {
        reject(err);
      } else {
        resolve(compressedData);
      }
    });
  });

  const compressedStream = new ReadableStream({
    start(controller) {
      controller.enqueue(compressedUint8Array);
      controller.close();
    },
  });
  const decompressedStream = compressedStream.pipeThrough(
    new DecompressionStream("gzip")
  );

  const reader = decompressedStream.getReader();
  const chunks = [];
  while (true) {
    const { done, value } = await reader.read();
    if (done) {
      break;
    }
    chunks.push(...value);
  }

  const outputUint8Array = new Uint8Array(chunks);
  const outputString = new TextDecoder().decode(outputUint8Array);

  console.log(inputString === outputString);
</script>

Test compression/decompression of stream in browser

<script type="module">
  import { AsyncGzip } from "https://cdn.skypack.dev/[email protected]?min";

  const inputChunks = Array(3).fill("hello");
  const encoder = new TextEncoder();
  const inputUint8ArrayChunks = inputChunks.map((chunk) =>
    encoder.encode(chunk)
  );

  const inputStream = new ReadableStream({
    start(controller) {
      for (const inputUint8ArrayChunk of inputUint8ArrayChunks) {
        controller.enqueue(inputUint8ArrayChunk);
      }
      controller.close();
    },
  });

  let endCallback;
  const asyncGzip = new AsyncGzip();
  const CompressionStream = new TransformStream({
    start(controller) {
      asyncGzip.ondata = (err, data, final) => {
        console.log({ err, data, final });
        if (err) {
          controller.error(err);
        } else {
          controller.enqueue(data);
          if (final) {
            if (endCallback) endCallback();
            else controller.terminate();
          }
        }
      };
    },
    transform(chunk) {
      asyncGzip.push(chunk);
    },
    flush() {
      return new Promise((resolve) => {
        endCallback = resolve;
        asyncGzip.push(new Uint8Array(0), true);
      });
    },
  });

  const compressedStream = inputStream.pipeThrough(CompressionStream);

  const decompressedStream = compressedStream.pipeThrough(
    new DecompressionStream("gzip")
  );

  const decompressedStreamReader = decompressedStream.getReader();
  const chunks = [];
  while (true) {
    const { done, value } = await decompressedStreamReader.read();
    if (done) {
      break;
    }
    chunks.push(...value);
  }

  const outputUint8Array = new Uint8Array(chunks);
  const outputString = new TextDecoder().decode(outputUint8Array);

  console.log({ outputString });
</script>

Checklist

• If you wrote E2E tests, are they resilient to concurrent I/O?
• If adding new public functions, did you add the @public tag and enable doc generation on the package?

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.