Manifold Graphql from Apache Beam using a Cloud Runner "Unable to convert url (manifoldclass://XXXXXXXXXX/) to file"

[gae123]

I am not sure if you guys are familiar with Apache Beam but basically there are two ways to run your a pipeline one locally and one in the cloud with multiple distributed workers etc. The environment is Java 8

I was able to to get the local case to work (DirectRunner), I basically need to do some GQL calls during data ingestion, and as I mentioned in another thread it works nicely.

However, when I try to run the same pipeline remotely it errors with a huge stack trace that I attach below. I assume it has something to do with the way the "fat jar" is created and what it contains. All that was working fine before but now it fails.

I would appreciate any suggestions....

[rsmckinney]

@gae123 Hi. From the stack trace this appears to be a general limitation with Apache Beam regarding URLs from a URLClassLoader. As part of Manifold's runtime library, it adds a URL of the form manifoldclass://<id> to the app class loader to facilitate dynamic loading of classes. It looks like Apache Beam expects only file:// URLs. There was a similar issue discovered with the Spring framework that was fixed a year or so ago.

In any case I'll have a look at the Apache Beam source and see what I can do to remedy this.

[gae123]

Thanks @rsmckinney since the project I work with is under extreme schedule pressure, I had to backpedal really quickly and use another way to do my GQL calls but I would love to come back to your approach when this issue is resolved. I suggest you work with the Beam guys to have it resolved and add an example in your samples.

[rsmckinney]

No problem, thanks for reporting this!

[rsmckinney]

Filed issue with Beam: https://issues.apache.org/jira/browse/BEAM-8875

[rsmckinney]

Note, #181 will make this issue obsolete; you will be able to use Manifold without the dynamic runtime, which is at the heart of this issue.

[rsmckinney]

With release 2020.1.17 you can use manifold-graphql as a compile-only dependency, which eliminates the runtime URL behavior. See the manifold-graphql setup info.