Frequently Asked Questions

Table of Contents

• My app uses the async client libraries, but it never seems to get any results?
• I'm getting a NoSuchMethodError or NoClassDefFoundError
• How to manage dependencies of my application running on Azure Data Bricks?

My app uses the async client libraries, but it never seems to get any results?

The Java client library asynchronous methods returns reactive types - Flux<T> and Mono<T>, or types extending from these.

These reactive types are referred to as "cold". That is, all underlying work will not be activated without a subscription. When an application invokes an async method, it will return without receiving any result. Only once a subscription is added will work be initiated to receive results.

The simplest way to create a subscription is to call subscribe() on the reactive type, but sometimes even after subscription you might not receive result. Consider the following example:

Mono<Response<Key>> result = asyncClient.getKey();
result.subscribe(response -> {
    System.out.println(response.value());
});

While the subscribe() call causes execution of the underlying work that makes an API call to retrieve the key, sometimes the lambda provided to print out the response value does not get called.

The reason for this is obvious in hindsight: reactive types are asynchronous. The rest of the application continues to operate as per usual, and this includes having the application exit before the results are returned!

Knowing this, the easiest means to block execution and be sure you get the results before your application continues execution (and / or exits) is to use the various block calls available, for example:

Mono<Response<Key>> result = asyncClient.getKey();
result.subscribe(response -> {
    System.out.println(response.value());
}).block();

I'm getting a NoSuchMethodError or NoClassDefFoundError

Explanation of the Cause

This exception is commonly the result of dependency conflicts. When two libraries depend on different versions of the same package, Maven will simply pick the first version that it sees and ignore the others ("First" is determined by which version is closer to the root project, and if they are the same distance, then the one declared first in the pom will be chosen) . If the two versions are sufficiently incompatible, it is possible that they offer a different set of types and methods. The library depending on the version that wasn't picked may then be unable to find the types or methods it depends on at runtime, resulting in these errors.

For example, suppose package A depends on version 1.0 of package C and package B depends on version 2.0 of version C. Version 1 contains type Foo, but in version 2 this type was removed and replaced with type Bar. If my application specifies a dependency on package A and then package B, only version 1 of package C will be included in my build. When my application's code requires package B to do something using type Bar, the JVM will throw this error.

Resolution

• Step 1: Validate the Cause of the Error In the same directory as the pom for your project, run the command mvn dependency:tree. This will output a recursive list of all your dependencies and their dependencies. In this list, search for the package that contains the type or method that is throwing the error. There will likely be multiple entries of different versions. Be aware that not all packages follow semantic versioning strictly. Just because both entries specify the same major version of a given package, that is not sufficient to assume they are compatible.

• Step 2: Resolve the dependency conflict In your project's pom, include a dependency entry that specifies the version of the shared dependency you want both packages to use. This will force Maven to use the specified version that should work for both. Because only one version can be included at a time, you must pick a single version that works for both packages. In the event that this is not possible, it may be necessary to upgrade one of your direct dependencies. If this still does not resolve the problem, your best course of action is to post an issue on the repo on each of the projects and request that they upgrade their dependencies.

How to manage dependencies of my application running on Azure Data Bricks?

Azure Data Bricks (ADB)

The Azure Data Bricks provides Apache Spark environment for your application. For example: Azure Data Bricks Runtime version 6.1 comes with Apache Spark 2.4.4 pre-installed. When running applications in ADB, ADB injects Apache Spark and it’s dependencies into application's class-path. Given runtime is providing Spark, normally application uses <provided> scope for Spark dependency.

Side effect of dependency injection

A side effect of ADB injecting Spark and it’s dependencies into the app class-path is, they take precedence over the direct and transitive dependencies that the application want to use. This is true for many hosting service that offer environment similar to ADB. When the application dependencies conflict with the ADB injected dependencies, errors such as NoSuchMethodError/NoClassDefFoundError can occur.

Example: Azure Storage SDK v12.0.0 has a dependency on Jackson version 2.10.0. If an application uses this version of storage SDK in ADB, it conflicts with the Jackson version 2.6.7 that DBRv6.1 place in the class-path. Jackson v2.10.0 is not back compact with v2.6.7, hence in this setup the application is likely to fail with NoSuchMethodError/NoClassDefFoundError linkage error.

Workaround for conflict: Create relocated Uber version of the application JAR (aka Shaded JAR)

The workaround is to shade your application using apache shade plugin. Using this plugin, you will create a self contained version of your application, which is a single Uber JAR that contains the application package and all of its dependency packages (both immediate and transitive dependencies). Using the same plugin, you will relocate the conflicting dependency packages in this Uber JAR to prevent their path names from conflicting with those that ADB bring into the class-path. The resulting JAR after applying relocation to Uber JAR is called Shaded JAR.

Creating Shaded JAR, example

To create Uber version of your application and to relocate conflicting Jackson dependency in the Uber JAR, add following shaded plugin entry under <build><plugins> section of application POM.

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<executions>
    <execution>
    <phase>package</phase>
    <goals>
        <goal>shade</goal>
    </goals>
    <configuration>
        <transformers>
        <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
            <mainClass><!-- YOUR_APPLICATION_MAIN_CLASS --></mainClass>
        </transformer>
        </transformers>
        <finalName>${project.arifactId}-${project.version}-shaded</finalName>
        <filters>
        <filter>
            <artifact>*:*</artifact>
            <excludes>
            <exclude>META-INF/maven/**</exclude>
            <exclude>META-INF/*.SF</exclude>
            <exclude>META-INF/*.DSA</exclude>
            <exclude>META-INF/*.RSA</exclude>
            </excludes>
        </filter>
        </filters>
        <relocations>
        <relocation>
            <pattern>com</pattern>
            <shadedPattern>com.microsoft.shaded.com.fasterxml.jackson</shadedPattern>
            <includes>
            <include>com.fasterxml.jackson.**</include>
            </includes>
        </relocation>
        </relocations>
    </configuration>
    </execution>
</executions>
</plugin>

then run:

mvn package

This will generate a shaded version of the application in target directory. For example: if your application POM has following maven co-ordinates defined:

<artifactId>adb-app</artifactId>
<version>1.0-SNAPSHOT</version>

then the shaded jar will be named as:

adb-app-1.0-SNAPSHOT-shaded.jar

This name is derived from the finalName entry in shaded plugin ${project.arifactId}-${project.version}-shaded.

The Jackson relocation configuration defined in the plugin will result in renaming the com.fasterxml.jackson package in the Uber JAR to com.microsoft.shaded.com.fasterxml.jackson and update all references to the classes from the original package.

Uploading Shaded JAR to ADB

1. From azure portal navigate to your Azure Data bricks portal (e.g. West US ADB portal)

2. Create a cluster based on Runtime: 6.1 (Scala 2.11, Spark 2.4.4)

3. Once cluster is ready, create a Job:

   a. Provide name for job

   b. Choose task as "Set JAR", this will bring up option to upload the JAR, point it to the shaded JAR

   c. Enter Main class name that you provided for mainClass entry of shaded plugin

What if application still fail with conflict?

Jackson is the known conflict at the time of writing this. If you run into more conflicts (NoSuchMethodError/NoClassDefFoundError exceptions) then follow the sample relocation approach outlined above for other conflicting dependencies.