Workflow independent model loading

[tovbinm]

Related issues
Following up on #269 - a sequence of PRs on allowing model loading without workflows...

Describe the proposed solution

1.

This PR finally allows loading models without recreating the workflow by simply calling:

val model = OpWorkflowModel.load(path)

Importantly the old syntax (i.e. workflow.loadModel(path)), still works as before.

With this change we are going to be serializing all transformers and feature generator instances, then reconstructing them when loading the model without requiring the original workflow.

1. Introduce a new reader/writer interface with a default reflection based implementation
2. Update stage reader & writer to use it
3. Add custom serializer for feature generator stages
4. Update model reader & writer to use new stage read / write technique

Of course updated all the tests with the changes.

Migration Notes
All lamdba transformers and extract functions cannot neither be inline nor have external dependencies on variables. Functions are now required to be specified in objects (Scala 2.11) and as concrete classes for Scala 2.12 and up. Example:

// the following syntax wont work anymore and error at runtime:
val description = FeatureBuilder.Text[Passenger].extract(_.getDescription.toText).asPredictor
val lowerCase = textFeature.map[Text](t => t.value.map(_.toLowerCase).toText)

// replace with concrete function classes as follows:
class DescriptionExtract extends Function1[Passenger, Text] with Serializable { def apply(p: Passenger): Text = p.getDescription.toText }
class LowerCaseText extends Function1[Text, Text] with Serializable { def apply(t: Text): Text = t.value.map(_.toLowerCase).toText }

val description = FeatureBuilder.Text[Passenger].extract(new DescriptionExtract).asPredictor
val lowerCase = textFeature.map[Text](new LowerCaseText)

2.
Additionally one can now provide a custom reader/writer for each transformer stage if necessary as follows:

// implement a custom reader/writer for your stage
class MyStageReaderWriter extends OpPipelineStageReaderWriter[MyStage] {
    def read(stageClass: Class[MyStage], json: JValue): Try[MyStage] = ???
    def write(stage: MyStage): Try[JValue] = ???
}

// add annotation with ReaderWriter to your transformer with the custom reader/writer implementation class MyStageReaderWriter
@ReaderWriter(classOf[MyStageReaderWriter])
class MyStage(...) extends UnaryTransformer[Text, Text] { ... } 

Here is a full example for our own TextTokenizer stage.

[codecov]

Codecov Report

Merging #274 into master will increase coverage by 0.05%.
The diff coverage is 83.72%.

@@            Coverage Diff             @@
##           master     #274      +/-   ##
==========================================
+ Coverage   86.51%   86.56%   +0.05%     
==========================================
  Files         329      335       +6     
  Lines       10617    10748     +131     
  Branches      334      567     +233     
==========================================
+ Hits         9185     9304     +119     
- Misses       1432     1444      +12

Impacted Files	Coverage Δ
...c/main/scala/com/salesforce/op/stages/HasOut.scala	100% <ø> (ø)	⬆️
.../scala/com/salesforce/op/test/FeatureAsserts.scala	100% <ø> (ø)	⬆️
...ages/base/sequence/BinarySequenceTransformer.scala	100% <ø> (ø)	⬆️
...la/com/salesforce/op/stages/OpPipelineStages.scala	63.88% <ø> (+1.38%)	⬆️
...rce/op/stages/impl/feature/FilterTransformer.scala	0% <0%> (ø)
...ce/op/stages/impl/feature/ReplaceTransformer.scala	0% <0%> (ø)
...rce/op/stages/impl/feature/ExistsTransformer.scala	0% <0%> (ø)
.../salesforce/op/utils/text/LuceneTextAnalyzer.scala	98.46% <100%> (+0.18%)	⬆️
...s/impl/feature/EmailToPickListMapTransformer.scala	100% <100%> (ø)
...cala/com/salesforce/op/OpWorkflowModelWriter.scala	100% <100%> (ø)	⬆️
... and 34 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 428eb4c...296fe8a. Read the comment docs.

[codecov]

Codecov Report

Merging #274 into master will decrease coverage by 29.33%.
The diff coverage is 58.33%.

@@             Coverage Diff             @@
##           master     #274       +/-   ##
===========================================
- Coverage   86.35%   57.02%   -29.34%     
===========================================
  Files         319      319               
  Lines       10431    10454       +23     
  Branches      345      550      +205     
===========================================
- Hits         9008     5961     -3047     
- Misses       1423     4493     +3070

Impacted Files	Coverage Δ
...rce/op/stages/OpPipelineStageReadWriteShared.scala	100% <ø> (ø)	⬆️
...rce/op/stages/impl/feature/ScalerTransformer.scala	92.59% <ø> (-3.71%)	⬇️
...la/com/salesforce/op/stages/OpPipelineStages.scala	63.51% <ø> (ø)	⬆️
...ages/base/sequence/BinarySequenceTransformer.scala	100% <ø> (ø)	⬆️
...com/salesforce/op/features/types/FeatureType.scala	95.95% <100%> (+0.04%)	⬆️
...a/com/salesforce/op/test/OpPipelineStageSpec.scala	98.03% <100%> (+0.03%)	⬆️
...lesforce/op/utils/reflection/ReflectionUtils.scala	88.23% <11.11%> (-9.14%)	⬇️
...m/salesforce/op/stages/OpPipelineStageReader.scala	64.86% <57.14%> (-0.77%)	⬇️
...m/salesforce/op/stages/OpPipelineStageWriter.scala	70.96% <90%> (+12.63%)	⬆️
...cala/com/salesforce/op/OpWorkflowModelWriter.scala	0% <0%> (-100%)	⬇️
... and 110 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7bcd1e4...4bb8d15. Read the comment docs.

[tovbinm]

Can anyone please review this PR?

[Jauntbox]

A few comments / questions:

1. Right now both model loading methods exist, correct? (eg. we can still load with a workflow with lambdas as we always have?)
2. Do you think we should we explicitly deprecate the old way, or leave it around forever?
3. Should we also update the helloworld examples since they use lambdas? Or do you think we should leave them as is since they're not saving/loading models?
4. There are also many examples where we use UnaryLambdaTransformers in tests, but don't save models (eg. BadFeatureZooTest). Do you think we should change all of those too, or just the ones that result in saved models?

If you think we should do either of the last two items, I'm fine with them being in separate PRs since this one is already enormous.

[tovbinm]

Kudos to @wsuchy @Jauntbox for all the help!

[tovbinm]

@Jauntbox

1. Correct.
2. I think we should deprecate workflow.loadModel(path) call eventually just to keep things simple.
3. Yes, we will update them before the next release.
4. Some of the test changes were not necessary, though I did it anyways.

[salesforce-cla]

Thanks for the contribution! Before we can merge this, we need @wsuchy to sign the Salesforce.com Contributor License Agreement.