Frequently Asked Questions

Table of Contents

• My app uses the async client libraries, but it never seems to get any results?
• I'm getting a NoSuchMethodError or NoClassDefFoundError
• Creating shaded Jars to avoid dependency conflicts
• How to manage dependencies of my application running on Azure Data Bricks?

My app uses the async client libraries, but it never seems to get any results?

The Java client library asynchronous methods returns reactive types - Flux<T> and Mono<T>, or types extending from these.

These reactive types are referred to as "cold". That is, all underlying work will not be activated without a subscription. When an application invokes an async method, it will return without receiving any result. Only once a subscription is added will work be initiated to receive results.

The simplest way to create a subscription is to call subscribe() on the reactive type, but sometimes even after subscription you might not receive result. Consider the following example:

Mono<Response<Key>> result = asyncClient.getKey();
result.subscribe(response -> {
    System.out.println(response.value());
});

While the subscribe() call causes execution of the underlying work that makes an API call to retrieve the key, sometimes the lambda provided to print out the response value does not get called.

The reason for this is obvious in hindsight: reactive types are asynchronous. The rest of the application continues to operate as per usual, and this includes having the application exit before the results are returned!

Knowing this, the easiest means to block execution and be sure you get the results before your application continues execution (and / or exits) is to use the various block calls available, for example:

Mono<Response<Key>> result = asyncClient.getKey();
result.subscribe(response -> {
    System.out.println(response.value());
}).block();

---

I'm getting a NoSuchMethodError or NoClassDefFoundError

Explanation of the Cause

This exception is commonly the result of dependency conflicts. When two libraries depend on different versions of the same package, Maven will simply pick the first version that it sees and ignore the others ("First" is determined by which version is closer to the root project, and if they are the same distance, then the one declared first in the pom will be chosen) . If the two versions are sufficiently incompatible, it is possible that they offer a different set of types and methods. The library depending on the version that wasn't picked may then be unable to find the types or methods it depends on at runtime, resulting in these errors.

For example, suppose package A depends on version 1.0 of package C and package B depends on version 2.0 of version C. Version 1 contains type Foo, but in version 2 this type was removed and replaced with type Bar. If my application specifies a dependency on package A and then package B, only version 1 of package C will be included in my build. When my application's code requires package B to do something using type Bar, the JVM will throw this error.

Resolution

• Step 1: Validate the Cause of the Error In the same directory as the pom for your project, run the command mvn dependency:tree. This will output a recursive list of all your dependencies and their dependencies. In this list, search for the package that contains the type or method that is throwing the error. There will likely be multiple entries of different versions. Be aware that not all packages follow semantic versioning strictly. Just because both entries specify the same major version of a given package, that is not sufficient to assume they are compatible.

• Step 2: Resolve the dependency conflict In your project's pom, include a dependency entry that specifies the version of the shared dependency you want both packages to use. This will force Maven to use the specified version that should work for both. Because only one version can be included at a time, you must pick a single version that works for both packages. In the event that this is not possible, it may be necessary to upgrade one of your direct dependencies. If this still does not resolve the problem, your best course of action is to post an issue on the repo on each of the projects and request that they upgrade their dependencies.

---

Creating shaded Jars to avoid dependency conflicts

We have documented how you can create a shaded JAR file that includes all dependencies. This should allow for dependency conflicts to be mitigated.

---

How to manage dependencies of my application running on Azure Data Bricks?

Azure Data Bricks provides an Apache Spark environment for your application. For example: Azure Data Bricks Runtime version 6.1 comes with Apache Spark 2.4.4 pre-installed. When running applications in ADB, ADB injects Apache Spark and it’s dependencies into application's class-path. Given runtime is providing Spark, normally application uses <provided> scope for Spark dependency.

Side effect of dependency injection

A side effect of ADB injecting Spark and its dependencies into the app class-path is, they take precedence over the direct and transitive dependencies that the application want to use. This is true for many hosting service that offer environment similar to ADB. When the application dependencies conflict with the ADB injected dependencies, errors such as NoSuchMethodError/NoClassDefFoundError can occur.

Example: Azure Storage SDK v12.0.0 has a dependency on Jackson version 2.10.0. If an application uses this version of storage SDK in ADB, it conflicts with the Jackson version 2.6.7 that DBRv6.1 place in the class-path. Jackson v2.10.0 is not back compact with v2.6.7, hence in this setup the application is likely to fail with NoSuchMethodError/NoClassDefFoundError linkage error.

How do I use these libraries in Azure Data Bricks?

Follow the steps below to upload these libraries into Azure Data Bricks:

1. Follow the guidance above to create a shaded Jar.

2. From azure portal navigate to your Azure Data bricks portal (e.g. West US ADB portal)

3. Create a cluster based on Runtime: 6.1 (Scala 2.11, Spark 2.4.4)

4. Once cluster is ready, create a Job:

   a. Provide name for job

   b. Choose task as "Set JAR", this will bring up option to upload the JAR, point it to the shaded JAR

   c. Enter Main class name that you provided for mainClass entry of shaded plugin

What if application still fail with conflict?

Jackson is the known conflict at the time of writing this. If you run into more conflicts (NoSuchMethodError/NoClassDefFoundError exceptions) then follow the sample relocation approach outlined above for other conflicting dependencies.