[SPARK-6869][PySpark] Add pyspark archives path to PYTHONPATH

[Sephiroth-Lin]

From SPARK-1920 and SPARK-1520 we know PySpark on Yarn can not work when the assembly jar are package by JDK 1.7+, so ship pyspark archives to executors by Yarn.
Usage:

1. Do nothing, then will zip pyspark auto, and ship pyspark.zip, py4j.zip to executors;
2. Set PYSPARK_ARCHIVES_PATH

1. ship archives to executor, then set PYSPARK_ARCHIVES_PATH=/you/path/pyspark.zip,/you/path/py4j.zip
2. don't ship archives, use locally, then set PYSPARK_ARCHIVES_PATH=local:///you/path/pyspark.zip,local:///you/path/py4j.zip

[SparkQA]

Test build #30111 has finished for PR 5478 at commit 413fa25.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[andrewor14]

@Sephiroth-Lin the point of running PySpark on YARN is that the user does not have to install Spark on the slave machines. Instead, we package the python files in the assembly jar, which is automatically shipped by YARN to all containers.

This change assumes that the python files will already be present on the slave machines, since PYTHONPATH reads from the local file system. I don't believe this is a deployment requirement that we want to enforce, especially since the user must now ensure all Spark python files are consistent across all the machines (as they must do so in standalone mode).

I would recommend that we close this issue since this isn't a feature we want to support.

[sryza]

IIUC, the motivation for this change is that the assembly jar distribution mechanism doesn't work for some Java versions.

I agree with Andrew that, if at all possible, we should avoid deployment models that expect PySpark or anything to be on every node. Even if we advise against it, it increases the number of places one needs to check when debugging why something does or does not appear on the executor PYTHONPATH.

Are there workarounds for the Java versions issue that don't require python to be installed on the NodeManagers?

[Sephiroth-Lin]

@andrewor14 @sryza Yes, to assume that the python files will already be present on the slave machines is not very reasonable. But if user want to use PySpark, then they must compile the Spark in JDK1.6, but I think now most user are use JDK1.7+. Maybe a good solution is package the PySpark in another jar and automatically shipped by YARN to all containers. And add this jar to PYTHONPATH with asseambly jar.

[sryza]

That alternative solution makes sense to me. If it's not going to be added to the classpath, it might make more sense to use a zip than a jar.

[andrewor14]

That sounds good. We can ship it through --py-files. @Sephiroth-Lin Would you mind updating the PR and the JIRA to reflect this change in intention?

[SparkQA]

Test build #30407 has finished for PR 5478 at commit 51ebb26.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[Sephiroth-Lin]

@andrewor14 @sryza Done, thanks.

[WangTaoTheTonic]

Hey guys, after discussion with @Sephiroth-Lin offline, we have some questions:

• If we try to ship the zip file to executer backend with --py-files:
  • User must use --py-file to point to the xxx.zip file we provide;
  • We must change format of directories(use xxx.zip to replace python dir) in our release.
  • Even the zip file is not very big, shipping it everytime still cost time.
• Inspired by the former solution, I thought we might add some config like spark.exectuor.extraClassPath in PySpark side, which points to the files needed by user's application so that user could use them without shipping.

[SparkQA]

Test build #30415 has finished for PR 5478 at commit 052e288.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #30414 has finished for PR 5478 at commit 309679a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[Sephiroth-Lin]

@andrewor14 @sryza @WangTaoTheTonic As I have test again, if we install Spark on each node, then we can set spark.executorEnv.PYTHONPATH=${SPARK_HOME}/python:${SPARK_HOME}/python/lib/py4j-0.8.2.1-src.zip to pass PYTHONPATH to executor. So this PR is another solution to run PySpark on yan if we don't install Spark on each node.

[WangTaoTheTonic]

Yeah, spark.executorEnv.PYTHONPATH is PySpark's spark.executor.extraClassPath in some way.

[SparkQA]

Test build #30417 has finished for PR 5478 at commit 3a0ec77.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[lianhuiwang]

yes, i think in SparkSubmit we can automatically add PYSPARK_ARCHIVES_PATH to dist files. and then in Client and ExecutorRunnable can set PYTHONPATH according to PYSPARK_ARCHIVES_PATH if it exist. and if user set PYTHONPATH, PYSPARK_ARCHIVES_PATH is unused. i have run successfully on yarn client and cluster.

[sryza]

Regarding the performance issue, this can be solved with the YARN distributed cache in the same way it works for the Spark assembly jar. If the file is placed on HDFS in a public location, it will be cached on the nodes as a public YARN local resource so it doesn't need to be downloaded each time an app is submitted.

[vanzin]

if we install Spark on each node, then we can set spark.executorEnv.PYTHONPATH

That's a workaround, but not really in the spirit of how Spark-on-YARN is expected to work. For example, you'd have to have all nodes have $SPARK_HOME be the same for that to work.

If this must be distributed as a separate file, then Sandy's solution is the way to go. The pyspark zip needs to be treated the same way the spark assembly is: if no configuration, find it locally and upload it to nodes using the distributed cache; add a config option so that users can store that file on HDFS or even use local: URIs in case they want to manually distribute the file.

[andrewor14]

Also, how big is the actual zip? I would imagine that it's at least one or two orders of magnitude smaller than the assembly jar, so it shouldn't be expensive especially if we cache it. As others have pointed out, the whole point of Spark on YARN is that the user doesn't need to install it on every node, and doing this through spark.executorEnv.PYTHONPATH defeats this purpose.

[lianhuiwang]

@sryza we can export PYSPARK_ARCHIVES_PATH=local://xx/pyspark.zip;local://xx/py4j.zip in spark-env.sh and we also can export PYSPARK_ARCHIVES_PATH=hdfs://xx/pyspaark.zip. then in SparkSubmit we can automatically add PYSPARK_ARCHIVES_PATH to yarn's dist files. and then Spark-on-Yarn can put dist files to YARN distributed cache. these work is same as Spark assembly jar.
@andrewor14 in my test, the pyspark.zip is 378KB, i think it is very small than the assembly jar.so it can put it to dist files automatically.

[SparkQA]

Test build #30462 has finished for PR 5478 at commit 547fd95.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #30478 has finished for PR 5478 at commit c63f31f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[andrewor14]

@Sephiroth-Lin I looked at the latest changes and it doesn't seem to reflect what I had in mind. There is still a manual step in which the user still needs to package the zip themselves and put it in a certain environment variable or config.

My proposal was to automatically zip up the python files in SparkSubmit and add it to the --py-files. If you do that then I believe all the PYTHONPATH will just be automatically set up as well. You can take a look at Java's ZipEntry or this post as a reference on how to do this.

[lianhuiwang]

@Sephiroth-Lin i think later i will submit my PR based on this PR and then please help me review it. thanks.

[Sephiroth-Lin]

@lianhuiwang OK.

[andrewor14]

@Sephiroth-Lin would you mind closing this PR then?

[Sephiroth-Lin]

@andrewor14 Sorry, these days I am busy, now I have update the code. ^-^

[SparkQA]

Test build #30747 has finished for PR 5478 at commit d012cde.

• This patch passes all tests.
• This patch does not merge cleanly.
• This patch adds no public classes.
• This patch adds the following new dependencies:
  • commons-math3-3.1.1.jar
  • snappy-java-1.1.1.6.jar
• This patch removes the following dependencies:
  • commons-math3-3.4.1.jar
  • snappy-java-1.1.1.7.jar

[SparkQA]

Test build #30748 has finished for PR 5478 at commit 5d9bcb6.

• This patch passes all tests.
• This patch does not merge cleanly.
• This patch adds no public classes.
• This patch adds the following new dependencies:
  • commons-math3-3.1.1.jar
  • snappy-java-1.1.1.6.jar
• This patch removes the following dependencies:
  • commons-math3-3.4.1.jar
  • snappy-java-1.1.1.7.jar

[Sephiroth-Lin]

@andrewor14 @sryza how about your opinions? thanks. @lianhuiwang please help me review this, thanks.

[tgravescs]

so is this competing directly with #5580?

[Sephiroth-Lin]

@tgravescs yes