Make query stream, similar to onSnapshot, allowing to download the stream and attach listener to changes.

[teebu]

Query stream as far as I understand only downloads the query request.
https://cloud.google.com/nodejs/docs/reference/firestore/1.3.x/Query#stream

PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.

Is your feature request related to a problem? Please describe.
The problem is if a collection is too big, I am getting timeout error like this because the onSnapshot is downloading a lot of data and not streaming it.

Issue discussed here, acupofjose/elasticstore#35
Reference to a similar problem: #1040

Error: Error 4: The datastore operation timed out, or the data was temporarily unavailable.
    at QueryWatch.onData (elasticstore\node_modules\@google-cloud\firestore\build\src\watch.js:350:34)
    at PassThrough.<anonymous> (elasticstore\node_modules\@google-cloud\firestore\build\src\watch.js:297:26)
    at PassThrough.emit (events.js:315:20)
    at addChunk (_stream_readable.js:302:12)
    at readableAddChunk (_stream_readable.js:278:9)
    at PassThrough.Readable.push (_stream_readable.js:217:10)
    at PassThrough.Transform.push (_stream_transform.js:152:32)
    at PassThrough.afterTransform (_stream_transform.js:96:10)
    at PassThrough._transform (_stream_passthrough.js:46:3)
    at PassThrough.Transform._read (_stream_transform.js:191:10)

Describe the solution you'd like
Firestore has a stream capability, https://cloud.google.com/nodejs/docs/reference/firestore/1.3.x/Query#stream but it only downloads the query, not constantly listens for changes. Is it possible to use the stream, in a similar way to onSnapshot does?

Describe alternatives you've considered
Use query stream as an initial source of data, then onSnapshot changes for only changed documents, using child_added and time > now query. proposed https://stackoverflow.com/questions/33885059/how-to-only-get-new-data-without-existing-data-from-a-firebase

[schmidt-sebastian]

@teebu Thanks for reporting this! Your stacktrace shows that you using the Watch API. The Watch API has a timeout of one hour, which should be plenty of time to download the contents of your query. One the contents have been downloaded, only diffs will be applied as long as the onSnapshot handler stays active.

If you see this timeout repeatedly, please file a customer support ticket, which will allow our backend team to look at exactly what was going on: https://cloud.google.com/support-hub

As a side note, the Query API restarts queries at the point the timed out (in recent version of the client) and you should be able to download almost arbitrarily large result sets.

[teebu]

The timeout happens close to 2 minutes after calling the query. It's not happening randomly, it happens every time.

And Error: Error 4: The datastore operation timed out, or the data was temporarily unavailable.

Using "firebase-admin": 8.12.1

I varied my my download speed in a few attempts, and each time the same timeout error occurred.

Updated to firebase-admin 9, still same error. below is a graph of 2 download attempts, of varied speeds.

It seems to me the duration is not the issue, but what happens after it finishes downloading the collection.

[schmidt-sebastian]

Can you file a support ticket and include your project ID and timestamps? Thanks!

[teebu]

I submitted a Ticket: Case 00073511
Timestamps would be July 17, 2020 2:30 - 2:50 for the tests above.