Error with gcloud-java for datastore access from compute engine: “LookupRequest overrides final method getUnknownFields”

[unitydynamics]

I started a brand new Java project (Java 8) and used maven to pull down my dependencies:

<dependency>
  <groupId>com.google.gcloud</groupId>
  <artifactId>gcloud-java</artifactId>
  <version>0.1.4</version>
</dependency>

I then wrote a very simple test call to write to the datastore... but there seems to be something structurally wrong with the library... here is the exception I ran into:

java.lang.VerifyError: class com.google.api.services.datastore.DatastoreV1$LookupRequest overrides final method getUnknownFields.

()Lcom/google/protob
uf/UnknownFieldSet;
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at com.google.gcloud.datastore.DatastoreOptions.normalize(DatastoreOptions.java:123)
        at com.google.gcloud.datastore.DatastoreOptions.access$400(DatastoreOptions.java:36)
        at com.google.gcloud.datastore.DatastoreOptions$Builder.build(DatastoreOptions.java:89)
        at com.google.gcloud.datastore.DatastoreOptions.defaultInstance(DatastoreOptions.java:175)

    ... my code here which called Datastore datastore = DatastoreOptions.defaultInstance().service();

I have plenty of jars but my environment is clean... I don't have any of the legacy google.appengine.* jars around. Just to make sure:

find . -name "*.jar" -exec grep -Hsli LookupRequest {} \;
returned only:

./google-api-services-datastore-protobuf-v1beta2-rev1-4.0.0.jar

Is the Google distribution broken? Is this a me thing? Any help appreciated.

[aozarov]

Looks like you are pulling a version of com.google.protobuf:protofuf-java which is not compatible with the underlying datastore-protobuf dependency google-api-services-datastore-protobuf-v1beta2-rev1-4.0.0.

Try running mvn dependency:tree -Dverbose -Dincludes=com.google.protobuf:protobuf-java to see
where you are pulling it from and exclude it from there.

A similar reported issue discovered that it was pulled from j2html and disabled it this way:

  <dependency>
     <groupId>com.j2html</groupId>
     <artifactId>j2html</artifactId>
     <version>0.7</version>
     <exclusions>
       <exclusion>
         <groupId>com.google.guava</groupId>
         <artifactId>guava</artifactId>
       </exclusion>
       <exclusion>
         <groupId>com.google.protobuf</groupId>
         <artifactId>protobuf-java</artifactId>
       </exclusion>
     </exclusions>
   </dependency>

/cc @eddavisson

[unitydynamics]

[WARNING] Using Maven 2 dependency tree to get verbose output, which may be inconsistent with actual Maven 3 resolution
[INFO] .... my jar......
[INFO] \- com.google.gcloud:gcloud-java:jar:0.1.4:compile
[INFO]    \- com.google.gcloud:gcloud-java-datastore:jar:0.1.4:compile
[INFO]       \- com.google.apis:google-api-services-datastore-protobuf:jar:v1beta2-rev1-4.0.0:compile
[INFO]          +- com.google.protobuf:protobuf-java:jar:3.0.0-beta-1:compile
[INFO]          \- com.google.http-client:google-http-client-protobuf:jar:1.15.0-rc:compile
[INFO]             \- (com.google.protobuf:protobuf-java:jar:2.4.1:compile - omitted for conflict with 3.0.0-beta-1)

Thanks for pointing me in the right direction.... I should have noticed that old version of http-client-protobuf was there. I moved that to 1.21 and deleted protobuf-java-2.4.1.jar and I'm off to the races now. Thank you @aozarov

I immediately tested an now I'm getting "not authorized" type of error...

Feb 29, 2016 1:08:06 PM com.google.api.services.datastore.client.BaseDatastoreFactory makeClient
WARNING: Not using any credentials
Feb 29, 2016 1:08:07 PM com.google.api.services.datastore.client.BaseDatastoreFactory makeClient
WARNING: Not using any credentials
com.google.gcloud.datastore.DatastoreException
        at com.google.gcloud.spi.DefaultDatastoreRpc.translate(DefaultDatastoreRpc.java:110)
        at com.google.gcloud.spi.DefaultDatastoreRpc.lookup(DefaultDatastoreRpc.java:147)
        at com.google.gcloud.datastore.DatastoreOptions.normalize(DatastoreOptions.java:128)
        at com.google.gcloud.datastore.DatastoreOptions.access$400(DatastoreOptions.java:36)
        at com.google.gcloud.datastore.DatastoreOptions$Builder.build(DatastoreOptions.java:89)
        at com.google.gcloud.datastore.DatastoreOptions.defaultInstance(DatastoreOptions.java:175)
        at com.industryopenings.seeker.driver.SiteLogger.saveEntry(SiteLogger.java:122)
        at com.industryopenings.seeker.driver.SiteLogger.log(SiteLogger.java:235)
        at com.industryopenings.seeker.driver.SiteLogger.log(SiteLogger.java:188)
        at com.industryopenings.seeker.driver.SiteLogger.info(SiteLogger.java:273)
        at com.industryopenings.seeker.driver.Driver.main(Driver.java:61)
Caused by: com.google.api.services.datastore.client.DatastoreException
        at com.google.api.services.datastore.client.RemoteRpc.makeException(RemoteRpc.java:115)
        at com.google.api.services.datastore.client.RemoteRpc.call(RemoteRpc.java:81)
        at com.google.api.services.datastore.client.BaseDatastoreFactory$RemoteRpc.call(BaseDatastoreFactory.java:41)
        at com.google.api.services.datastore.client.Datastore.lookup(Datastore.java:93)
        at com.google.gcloud.spi.DefaultDatastoreRpc.lookup(DefaultDatastoreRpc.java:145)
        ... 9 more
Caused by: com.google.api.client.http.HttpResponseException: 401 Unauthorized
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1061)
        at com.google.api.services.datastore.client.RemoteRpc.call(RemoteRpc.java:78)
        ... 12 more

If I can bug you just one more time... what's the magic I need to add to authenticate? Perhaps I read the docs too quickly... I thought it was handled automatically. My compute engine instances is in the same project... but inspection of the environment variables doesn't seem to yield anything that would tip off gcloud-java what my project ID is.

I'm almost there!

Oh and I built my custom image from the standard image: debian-8-jessie-v20160219
... and here is the permissions section on my instance. I haven't tried with "Allow All" option yet... maybe I'll test that.

[mziccard]

@unitydynamics glad you solved your original issue.

For the 401 Unauthorized error try also adding the "User Info" scope (the first permission in your screenshot) when you create the instance.

[unitydynamics]

You nailed it @mziccard . I've written my first entity to the datastore with gcloud-java! Thanks.

I noticed the required permissions are indeed documented...: https://github.com/GoogleCloudPlatform/gcloud-common/blob/master/authentication/readme.md#authentication

but... I'm probably not the last wandering spirit to trip over this... gentle recommendation to clean up the scopes.

[unitydynamics]

On more small comment... Stderr is getting littered with plenty of:
Feb 29, 2016 3:14:07 PM com.google.api.services.datastore.client.BaseDatastoreFactory makeClient
WARNING: Not using any credentials

[aozarov]

@unitydynamics You can safely ignore this WARNING. It will be gone in the next version of the datastore
(v1beta3). Also, the need to use "User Info" scope is going to be removed by then.