Executing dotnet spark application via Spark REST API

[willim32533]

Is anybody aware of whether (and how, if possible) it is possible to execute a dotnet spark application using the native spark REST api? For example, I've tried posting the following to the REST api (http://localhost:6066/v1/submissions/create) and the UI indicates that the driver is submitted, but nothing ever happens - it stays in the driver submitted state indefinitely. The same application runs as expected via spark-submit.

--data {
 "appResource": "dotnet /opt/app/LoadBronze.dll",
  "sparkProperties": {
    "spark.executor.memory": "1g",
    "spark.master": "spark://spark:7077",
    "spark.driver.memory": "8g",
    "spark.driver.cores": "2",
    "spark.eventLog.enabled": "true",
    "spark.eventLog.dir" : "file:/usr/bin/spark/hadoop/logs",
    "spark.app.name": "LoadBronze",
    "spark.submit.deployMode": "client",
    "spark.jars": "/opt/bitnami/spark/jars/microsoft-spark-3-1_2.12-2.0.0.jar",
    "spark.driver.supervise": "true"
  },
  "clientSparkVersion": "3.1.2",
  "mainClass": "org.apache.spark.deploy.dotnet.DotnetRunner",
  "environmentVariables": {
    "SPARK_ENV_LOADED": "1"
  },
  "action": "CreateSubmissionRequest",
  "appArgs": [
    "filename.ext"
  ]
}

The same application executed using spark-submit runs:

/opt/bitnami/spark/bin/spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner \
   --conf "spark.eventLog.enabled=true" \
   --conf "spark.eventLog.dir=file:/usr/bin/spark/hadoop/logs" \
   --master spark://spark:7077 \
   /opt/bitnami/spark/jars/microsoft-spark-3-1_2.12-2.0.0.jar \
   dotnet /opt/app/LoadBronze.dll "filename.ext"

[willim32533]

Finally figured out a solution by running spark-submit via the CLI and specifying --master spark://spark:6066 and then comparing the driver launch commands that were executed on the worker node.

In case this helps anybody else, here is the body of an example post command (using Postman) for the native Spark REST API to trigger a dotnet spark application.

Spark REST API endpoint: http://[localhost or dns name or ip address]:6066/v1/submissions/create

{ "action": "CreateSubmissionRequest", "appArgs": [ "dotnet", "/path/to/your/compiled_dotnet_app.dll", "app arg 1", "app arg 2", "etc..." ], "appResource": "file:/opt/bitnami/spark/jars/microsoft-spark-3-2_2.12-2.1.0.jar", "clientSparkVersion": "3.2.1", "environmentVariables": { "SPARK_ENV_LOADED": "1" }, "mainClass": "org.apache.spark.deploy.dotnet.DotnetRunner", "sparkProperties": { "spark.driver.supervise": "false", "spark.app.name": "org.apache.spark.deploy.dotnet.DotnetRunner", "spark.submit.deployMode": "cluster", "spark.master": "spark://spark:7077", "spark.jars":"file:/opt/bitnami/spark/jars/microsoft-spark-3-2_2.12-2.1.0.jar", "spark.submit.pyFiles":"" } }

HTH