feat: add wrapper for reading table data using Storage API

[alvarowolfx]

Add support for easily reading Tables using the BigQuery Storage API instead of the BigQuery API. This will provide increased performance and reduced memory usage for most use cases and will allow users to keep using the same interface as they used to use on our main library or fetch data directly via a new veneer on BigQuery Storage Read API

[alvarowolfx]

Some early results:

1. SELECT repository_url as url, repository_owner as owner, repository_forks as forks FROM `bigquery-public-data.samples.github_timeline` where repository_url is not null LIMIT 300000

• fetchGetQueryResultsI: 31.135s 🔴
• fetchStorageAPI: 20.033s ⬆️ 36% speedup

2. SELECT repository_url as url, repository_owner as owner, repository_forks as forks FROM `bigquery-public-data.samples.github_timeline` where repository_url is not null LIMIT 1000000

• fetchGetQueryResultsI: 1:32.622 (m:ss.mmm) 🔴
• fetchStorageAPI: 1:07.363 (m:ss.mmm) ⬆️ 27% faster

3. SELECT name, number, state from `bigquery-public-data.usa_names.usa_1910_current

• fetchGetQueryResults: 5:00.514 (m:ss.mmm) 🔴
• fetchStorageAPI: 3:20.987 (m:ss.mmm) ⬆️ 33% faster

[leahecole]

Got it. Re third bullet point, something like this test I am working on in gax - in this test in gax, I have two streams piped together (I'm using pipeline) - attemptStream is the stream that's making the request to the server, and userStream the consuming stream. They are being piped together

Some things to note

• Idk if you'd need the error checking I have here - I'm specifically writing a retry test so the .on('error') stuff may be irrelevant - I'm also firing errors using the showcase server at very specific points in the data stream (mainly after the 85th item)
• This test has listeners for data on both the attemptStream and userStream - each of those appends the data content to an array. At the end, the two arrays should be the same length. I think a test of yours could use pipeline to connect streams and check that the lengths agree at the end.
• If you experiment with this test and can't find a way to ever make it fail/be confident it's doing anything, don't sweat it. It may be more important on the retries case and when I originally wrote this comment I was very much in the weeds with that CBT stuff
• the pausing happens in the user stream, but not in the attemptStream. This forces data to build up in the buffer. You could also change that userStream.on('data') bit to only pause when results2 is a certain length, but I think pausing every time is more of a surefire way to force that buffer to fill.

let results: string[] = [];
let results2: string[] = [];

attemptStream.on('data', (data: {content: string}) => {
    results.push(data.content);
  });
  attemptStream.on('end', () => {
    assert.strictEqual(results.length, 100);
  });

  attemptStream.on('error', (e: GoogleError) => {
    assert.strictEqual(e.code, 13);
  });
  userStream.on('data', (data: {content: string}) => {
    results2.push(data.content);
    userStream.pause();
    setTimeout(() => {
      userStream.resume();
    }, 100);
  });
  userStream.on('end', () => {
    assert.strictEqual(results.length, 100);
    assert.strictEqual(results2.length, 100);
  });
  userStream.on('error', (e: GoogleError) => {
    // assert.strictEqual(results2.length, 85)
    assert.strictEqual(results.length, 85);

    assert.strictEqual(e.code, 14);
  });

[alvarowolfx]

Added some more comments, address some minor issues and pushed a new test using CTA and generate_array to test it with more data cc @shollyman @leahecole

[alvarowolfx]

Tried to use pipeline, but is meant to be used when we have a destination. In this case where, we are just applying a bunch of transforms and we don't know the destination beforehand.

The error that I got:

  TypeError [ERR_INVALID_ARG_TYPE]: The "streams[stream.length - 1]" property must be of type function. Received an instance of ArrowRecordBatchTransform

[alvarowolfx]

Errors are handled by the consumer of the stream and when used internally like here, we handle the errors.