[SPARK-14222][BUILD] Remove jackson-module-scala dependency

[JoshRosen]

This patch removes our jackson-module-scala dependency in order to reduce the number of dependencies that we'll have to upgrade when adding experimental Scala 2.12 support. I think that our current use of this library is fairly minimal and removing the dependency seems like less work than having to help test and publish it for every Scala 2.12 milestone plus the final 2.12 release (see FasterXML/jackson-module-scala#245).

/cc @vanzin @andrewor14

[JoshRosen]

/cc @aarondav and @ahirreddy, do you know of any pitfalls that I watch out for in removing this dependency?

[JoshRosen]

By the way, I'm almost certain that this patch is going to fail its first round of tests provided that our coverage is good enough; just wanted to get an early draft out for feedback to find out if there are any catastrophic blockers.

[cowtowncoder]

I don't know enough about Spark internals to know if this makes sense, but there is a smaller alternative databinding, Jackson jr:

https://github.com/FasterXML/jackson-jr

which does use Jackson's streaming parser+generator (jackson-core), but not databinding. This makes it about 80% smaller to use. There is also uber-jar variant with shaded jackson-core, which would remove any conflicts with various jackson versions, with size of 350kB (jackson-jr-objects which does not bundle jackson-core is about 75kB).
Jackson-jr does support both "untyped" access (Maps, Lists, wrappers) and very basic Bean-binding, and there is even simple optional tree model (jackson-jr-stree, 24kB); and performance-wise is comparable to full Jackson databinding for most use cases.

So using this might allow removal of jackson-databind, and this in turn could help with some of version compatibility problems as many spark jobs use Jackson for their own needs.
But as I said, I don't know how heavy usage is and whether this is a practical possibility.

[JoshRosen]

Besides a few direct uses of Jackson, I think that Spark also uses it indirectly through json4s-jackson so switching to jackson-jr might be a little tricky right now. Thanks for pointing out that library, though: it's going to be a huge help if we do wind up having to shade / hide the full version of Jackson from our classpath.

[cowtowncoder]

@JoshRosen ah. Yes, good point on usage via json4s.

[vanzin]

I think one place that would break without the scala module, and where we don't have unit tests right now, is rebuilding the SQL UI in the history server. The events are read from the log file and processed with jackson, and since they're case classes, that probably won't work without the scala module.

[JoshRosen]

This was added in #10061 in order to allow the new SparkListenerSQLExecutionStart and SparkListenerSQLExecutionEnd events to be written to the event log using Jackson. I'll see if there's an existing unit test for roundtrip serialization of these events.

[JoshRosen]

/cc @carsonwang

[SparkQA]

Test build #55145 has finished for PR 12213 at commit 4104b1a.

• This patch fails from timeout after a configured wait of 250m.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Okay, so it turns out that removing this is going to be kind of tricky right now because we'd take on the burden of ensuring proper serialization of Scala Option types and, from experience, that's going to become a bit messy.

Therefore, I'm going to fall back on an alternative proposal: I'll take on the burden of getting jackson-module-scala to run on 2.12 and then will work to move from a lot of imperative json4s code to Jackson databind (https://issues.apache.org/jira/browse/SPARK-12141) so that we can eventually remove json4s and consolidate on Jackson (https://issues.apache.org/jira/browse/SPARK-14439). Once we've done that, if it later turns out that we need to remove jackson-module-scala then we can just absorb / inline whatever portion of it we actually use.

[cowtowncoder]

@JoshRosen probably makes sense, but just in case it needs to be revisited Jackson's core databind has much more support for "referential types" (which includes Option/Optional) as of 2.7, and supporting it should be much simpler with basic (de)serializer like one used for AtomicReference. I can help more if/when this is needed; for now I just mention it as forward-reference. I will actually need to work with @nbauernfeind (maintainer of Jackson Scala module) to ensure Scala module 2.7 is fully up-to-date with this part of type handling; earlier it required much more explicit work by module.