Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(firestore): Add support for Query Partitions #11635

Merged
merged 23 commits into from
Jun 15, 2021
Merged

feat(firestore): Add support for Query Partitions #11635

merged 23 commits into from
Jun 15, 2021

Conversation

quartzmo
Copy link
Member

@quartzmo quartzmo commented May 22, 2021 •
edited
Loading

@product-auto-label product-auto-label bot added the api: firestore Issues related to the Firestore API. label May 22, 2021
@quartzmo quartzmo self-assigned this May 22, 2021
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label May 22, 2021
@quartzmo quartzmo force-pushed the firestore-partition branch from ce6730a to 3895e3f Compare May 24, 2021 16:58
@quartzmo quartzmo force-pushed the firestore-partition branch from a8c3ee4 to a8388a2 Compare May 25, 2021 23:01
@quartzmo quartzmo marked this pull request as ready for review May 26, 2021 19:27
@quartzmo quartzmo requested a review from a team as a code owner May 26, 2021 19:27
@quartzmo quartzmo force-pushed the firestore-partition branch from f5f8520 to 759d4e4 Compare May 26, 2021 19:32
@quartzmo quartzmo force-pushed the firestore-partition branch from 759d4e4 to 4e61da5 Compare May 26, 2021 19:33
@quartzmo quartzmo added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label May 27, 2021
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label May 27, 2021
schmidt-sebastian
schmidt-sebastian reviewed May 27, 2021
View reviewed changes
Copy link

@schmidt-sebastian schmidt-sebastian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We just found out that the Partition RPC does not necessarily return the partitions in order, which can lead to overlapping results if we do not sort them ourselves. Can you add code to sort the results before building the QueryPartition? We also need to fix this in Node and Java.

@quartzmo quartzmo marked this pull request as draft May 27, 2021 15:38
@quartzmo
Copy link
Member Author

Can you add code to sort the results before building the QueryPartition?

Added in new class ResourcePath, based on parts of path.ts.

schmidt-sebastian
schmidt-sebastian reviewed Jun 1, 2021
View reviewed changes
Copy link

@schmidt-sebastian schmidt-sebastian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as my Ruby expertise goes, this looks good. I did not verify the tests but the implementation handles the cases I handled in Node and Java.

@quartzmo
Rename QueryPartition#create_query to #to_query
85c5ffa 
* Fix return type docs for QueryPartition.
quartzmo
quartzmo commented Jun 1, 2021
View reviewed changes
google-cloud-firestore/lib/google/cloud/firestore/query_partition.rb
##
# @private New QueryPartition from query and Cursor
def initialize query, start_at, end_before
@query = query
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be possible for a user to serialize the result of the QueryPartition API so that the partitions can be processed on other machines.

A reference to the original Google::Cloud::Firestore::Query instance is provided to each QueryPartition. This object holds a reference to the Google::Cloud::Firestore::Client instance, which is used in Query#get to run the query.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed offline with @dazuma, serialization will be provided via new methods:

data = my_query_partition.to_json
my_query_partition = QueryPartition.from_json data, client

@quartzmo quartzmo marked this pull request as ready for review June 1, 2021 21:19
dazuma
dazuma approved these changes Jun 1, 2021
View reviewed changes
Copy link
Member

@dazuma dazuma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's here LGTM. Ping me if/when you add serialization.

@quartzmo quartzmo added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 8, 2021
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 8, 2021
@quartzmo quartzmo merged commit 7360d09 into googleapis:master Jun 15, 2021
@quartzmo quartzmo deleted the firestore-partition branch June 15, 2021 20:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@schmidt-sebastian schmidt-sebastian schmidt-sebastian left review comments

@crwilcox crwilcox crwilcox left review comments

@dazuma dazuma dazuma approved these changes

Assignees

@quartzmo quartzmo

Labels
api: firestore Issues related to the Firestore API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@quartzmo @dazuma @crwilcox @schmidt-sebastian @yoshi-kokoro