Provide PySpark and SparkR wrapper packages

[naidubharadwaj9]

Is there a way I can use this library for importing a PMML model into Spark using pyspark or scala or sparkR?

[vruusmann]

importing a PMML model into Spark using pyspark or scala or sparkR?

JPMML-Evaluator-Spark is a Java library, which means that you can call it directly from Scala code.

If you want to call it from PySpark or SparkR/Sparklyr, then you should take time to develop simple Python/R wrappers for it. Otherwise, simply use their Java gateways for communication (eg. in PySpark, you can access it via sc._jvm).

[SachinTen11]

So, we cannot use this package in Zeppelin or Jupyter notebooks? since JPMML is a java package and it cannot be used with a interpreted language? I am new to spark, please correct me if my understanding is incorrect.

[vruusmann]

So, we cannot use this package in Zeppelin or Jupyter notebooks?

You can use this package in whatever Apache Spark ML environment you like. However, if you are using non-Java environments such as Python or R, then it would be recommended to create an appropriate Python or R wrapper API on top of the existing Java API to improve end user experience.

I haven't had time to build those Python or R wrapper APIs yet. But it's a fairly short task, maybe one or two days of development work.