Lazy loading of entities when running a query

[clardeur]

I just want to scroll all entities of a specific kind in a lazy way. I have tried to add a limit but as intended I have only the number of specified results and not the entire dataset.

StructuredQuery<Entity> query = Query.entityQueryBuilder()
    .kind("user")
    .limit(100)
    .build();

QueryResults<Entity> results = datastore.run(query);

How to do it ? Did you plan to add a fetchSize() parameter ?

Without limit() I have a DatastoreException: Server returned an error INTERNAL 500

[ajkannan]

Hi @clardeur, thanks for reporting this issue. Query results are already sent in batches, but it looks like the amount of data per batch is still causing you errors. gcloud-java doesn't have control over the batch size; that's decided by Google Cloud Datastore. Instead, gcloud-java could return you a cursor. The reason QueryResults.cursorAfter() isn't already exposed is that until Datastore v1beta3 is released, we cannot reliably populate that field.

As a work around for your issue, we can set cursorAfter. You should run a query with a limit, consume all the results of that query, and issue another query with the start cursor set to the end cursor of the previous query if the number of results is equal to the limit. For example:

int limit = 100;
StructuredQuery<Entity> query = Query.entityQueryBuilder()
    .kind("user")
    .limit(limit)
    .build();
while (true) {
  QueryResults<Entity> results = datastore.run(query);
  int resultCount = 0;
  while (results.hasNext()) {
    Entity result = results.next(); // consume all results
    // do something with the result
    resultCount++;
  }
  if (resultCount < limit) {
    break;
  }
  query = query.toBuilder().startCursor(results.cursorAfter()).build();
}

For GqlQuery, limit and start cursor can be specified as mentioned in the Cloud Datastore documentation.

[ajkannan]

This behavior stems from a Google Cloud Datastore issue, which can be tracked here.

I've edited my comment above to mention that instead of introducing a new cursor, we will set cursorAfter to work around this issue. I also took out the suggestion to use offset, since that could give you incorrect results (especially if entities are added/deleted/updated between query requests). I'm starting work on this issue now.

[clardeur]

@ajkannan Thanks for your quick reply and the workaround.

I understand that the batch size is a restriction from Google Cloud Datastore and you don't have the control on it but I think that as an end-user of the gcloud-java client, I don't want to add some boilerplate code to iterate over queries, even if it should be possible for keeping a fine-grained control.

Could you expose on the API a ScrollQuery with a querySize() parameter that would use the limit() and cursorAfter() internally ? What do you think about it ?

[mziccard]

@clardeur Thanks for the reply. I think you are right, what we do elsewhere in gcloud-java is to also provide an iterateAll method that returns an iterator over the whole query result avoiding you to explicitly deal with multiple requests. That way you should be able to do something like:

StructuredQuery<Entity> query = Query.entityQueryBuilder()
    .kind("user")
    .limit(100)
    .build();
QueryResults<Entity> results = datastore.run(query);
Iterator<Entity> iterator = results.iterateAll();
while (iterator.hasNext()) {
  // do something with the entity iterator.next()
}

How do you see this?
@ajkannan do you think this could be feasible?

[clardeur]

@mziccard the iterateAll() is what I need, like this I could build a Java 8 Stream on it. 👍

[aozarov]

@clardeur thank you for your input.

@mziccard, I think this case is different from the other gcloud-java list operations we have so far. Here are some differences:

1. limit != fetchSize - though they sound and feel similar (and indeed one could achieve the other's goal) they are not exactly the same. limit is an SQL construct that suggests no more results after that limit are necessary whereas fetchSize/chunkSize is a batching optimization.

2. Datastore supports a row/entity-level cursor (in concept, this does not work in v1beta2 but will work in the upcoming version) where as the other services only support page level cursor. Therefore the Datastore QueryResults is actually already an iterator that is equal to iterateAll, where "All" means all values constrained by the query, with an ability to stop at any point and use the cursor (once available) to resume.

3. The reason QueryResults is an iterator (rather than, lets say, a List) is because when a result is too big it is actually returned in chunks/batches, and gcloud-java gets new batches as needed. Currently, the decision of what is returned in one chunk/batch is the Datastore's and depends on both results size and fetch time. Unfortunately, the Datastore has a bug, Error when limit is not set for some queries google-cloud-datastore#85, where the service fails to detect that it should stop accumulating data when batch is too big for it to handle. This bug is known to be fixed in the upcoming version of the Datastore.

I do think a user-level batch size, which the server can use to further constrain its own chunking logic,
would be nice. I suggested it to the Datastore team but I am not sure if they will implement it.

Even without having a user-configurable batch size, I would expect this not to be an issue after Datastore fixes their bug. Datastore chunking will probably be sufficient in all cases (except maybe for clients that are very memory constraint).

@ajkannan's suggestion above is merely a workaround to avoid the Datastore bug until its fixed and without changing the API.

Indeed, gcloud-java could manipulate the query's limit (if given by the user or not) to make sure that we never ask for more than X results at a time (where X is is the user-configured batch size). This would provide a library level chunking. Even in this case, there is no need for an additional iterateAll; the current iterator could do that. However, using the current underlying Datastore API, we will not be able implement the same approach for GQL. I asked the Datastore team for advice about that as well.

For now, I suggest that we continue with #549, as it could be used as a workaround, does not effect the API and won't stop us from making any future changes if we choose too.

@clardeur Yes, once we move to Java 8 we will likely provide a Stream option instead as an alternative to plain Iterators. By then it is also possible that Datastore (as well as other services) will migrate to gRPC and provide streaming service calls.

[aozarov]

Laving it open at least until the underlying issue is fixed.

[clardeur]

I have tried the workaround and I get an error with the cursor.

java.lang.IllegalArgumentException: content is not a valid UTF-8
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
    at com.google.gcloud.datastore.Cursor.<init>(Cursor.java:47)
    at com.google.gcloud.datastore.QueryResultsImpl.cursorAfter(QueryResultsImpl.java:104)

[aozarov]

Thanks @clardeur

Yes, it seems that in this case the content of the cursor bytes is not a valid UTF-8.
We will remove this restriction and work with the Datastore team to make sure we could
still create a url-safe string value for it.

[clardeur]

@aozarov ok I hope that this problem will be solved soon by the Datastore team or with this workaround because since the migration to the gcloud-java library this is a blocking point for us.

Thanks for your envolvement.

[ajkannan]

Hi @clardeur, we added a fix for the issue (#578). Do you need us to push a new release version on Maven for this change?

[clardeur]

@ajkannan yes if it works, thanks.

[ajkannan]

Sure thing, I will make a release today. Before I do this I want to resolve one more issue related to Datastore queries (#585). Thanks for your patience!

[ajkannan]

Update: version 0.1.3 is now pushed and includes a fix for the IllegalArgumentException issue.

[clardeur]

@ajkannan tested and LGTM ;)

You can closed it if you want.

[ajkannan]

gcloud-java-datastore version 0.1.7 now uses v1beta3 which fixes the original problem. This work around should no longer be needed, so I'll close this issue.

[Chinmay-Deshpande]

Hi , I want to get the total count of entities in a kind in google cloud datastore. When the entities are 5 Million the following code doesnt return the count.

private static Datastore datastore;
datastore = DatastoreOptions.getDefaultInstance().getService();
Query query = Query.newKeyQueryBuilder().setKind(kind).build();
int count = Iterators.size(datastore.run(query)); //count has the entities count

Could you please share insights/alternative?