Skip to content

Commit

Permalink
Merge pull request apache#1 from tangzhankun/zhankun-HDL
Browse files Browse the repository at this point in the history 
merge TensorFlow-YARN from zhankun's branch
  • Loading branch information
tangzhankun authored Feb 7, 2017
2 parents 4d5a247 + bcb5598 commit 74e48e7
Show file tree
Hide file tree
Showing 82 changed files with 3,844 additions and 3 deletions.
3 changes: 3 additions & 0 deletions hadoop-deeplearning-project/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Hadoop Deep Learning Project
======================
##[YARN-TensorFlow](YARN-TensorFlow/hadoop-yarn-applications-tensorflow/README.md)
32 changes: 32 additions & 0 deletions ...plearning-project/YARN-TensorFlow/hadoop-yarn-applications-tensorflow/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
TensorFlow on YARN
======================
TensorFlow on YARN is a YARN application to enable an easy way for end user to run TensorFlow scripts.

Note that current project is a prototype with limitation and is still under development

## Features
- [x] Launch a TensorFlow cluster with specified number of worker and PS server
- [x] Replace python layer with java bridge layer to start server
- [x] Generate ClusterSpec dynamically
- [x] RPC support for client to get ClusterSpec from AM
- [x] Signal handling for graceful shutdown
- [ ] Package TensorFlow runtime as a resource that can be distributed easily
- [ ] Fault tolerance
- [ ] Code refine and more tests

## Set up and run
1. Git clone ..
2. Compile [tensorflow-bridge](../tensorflow-bridge/README.md) and put libbridge.so to a place be aware to YARN application. For instance, JVM lib directory.
3. Compile TensorFlow on YARN

```sh
cd <path_to_hadoop-yarn-application-tensorflow>
mvn clean package -DskipTests
```
4. Run your Tensorflow script. Let's assume a "job.py"

```sh
./bin/yarn-tf -job job.py -numberworkers 4 -numberps 1 -jar <path_to_tensorflow-on-yarn-with-dependency_jar>
```

Note that at present, the "job.py" should parse worker and PS server from parameters "ps" and "wk" populated by TensorFlow on YARN client in the form of comma seperated values.
46 changes: 46 additions & 0 deletions hadoop-deeplearning-project/YARN-TensorFlow/hadoop-yarn-applications-tensorflow/bin/yarn-tf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
#!/bin/bash

JOB=""
WORKERS=0
PSES=0
JAR=""
while true
do
case "$1" in
-job)
JOB="$2"
echo "job script: $JOB"
shift
;;
-numberworkers)
WORKERS="$2"
echo "worker num: $WORKERS"
shift
;;
-numberps)
PSES="$2"
echo "ps num: $PSES"
shift
;;
-jar)
JAR="$2"
echo "jar path: $JAR"
shift
;;
*)
shift
break
;;
esac
shift
done

CLIENT_MAIN_CLASS="org.apache.hadoop.yarn.applications.tensorflow.Client"

yarn jar $JAR $CLIENT_MAIN_CLASS \
--jar $JAR \
--tf_client $JOB \
--num_worker $WORKERS \
--num_ps $PSES \
--container_memory 4096

211 changes: 211 additions & 0 deletions hadoop-deeplearning-project/YARN-TensorFlow/hadoop-yarn-applications-tensorflow/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>YARN-TensorFlow</artifactId>
<groupId>org.apache.hadoop</groupId>
<version>3.0.0-alpha2-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>hadoop-yarn-applications-tensorflow</artifactId>
<version>3.0.0-alpha2-SNAPSHOT</version>
<name>TensorFlow on YARN</name>

<repositories>
<repository>
<id>bintray</id>
<name>Bintray Repository</name>
<url>http://dl.bintray.com/fvunicorn/maven</url>
</repository>
</repositories>

<properties>
<!-- jackson versions -->
<jackson.version>1.9.13</jackson.version>
</properties>

<dependencies>
<!-- 'mvn dependency:analyze' fails to detect use of this dependency -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<scope>provided</scope>
</dependency>
<!-- 'mvn dependency:analyze' fails to detect use of this dependency -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-client</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.tensorflow</groupId>
<artifactId>java-bridge</artifactId>
<version>0.1.0</version>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-server-timelineservice</artifactId>
<type>test-jar</type>
<scope>test</scope>
</dependency>
<!-- 'mvn dependency:analyze' fails to detect use of this dependency -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<type>test-jar</type>
<scope>test</scope>
</dependency>
<!-- 'mvn dependency:analyze' fails to detect use of this dependency -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-server-nodemanager</artifactId>
<scope>test</scope>
</dependency>
<!-- 'mvn dependency:analyze' fails to detect use of this dependency -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-server-resourcemanager</artifactId>
<scope>test</scope>
</dependency>
<!-- 'mvn dependency:analyze' fails to detect use of this dependency -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-server-tests</artifactId>
<type>test-jar</type>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-server-timeline-pluginstorage</artifactId>
<type>test-jar</type>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-common</artifactId>
<type>test-jar</type>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<scope>test</scope>
<type>test-jar</type>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>jar</goal>
</goals>
<!-- strictly speaking, the unit test is really a regression test. It
needs the main jar to be available to be able to run. -->
<phase>test-compile</phase>
</execution>
</executions>
<configuration>
<archive>
<manifest>
<mainClass>org.apache.hadoop.yarn.applications.tensorflow.Client</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<environmentVariables>
<JAVA_HOME>${java.home}</JAVA_HOME>
</environmentVariables>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-maven-plugins</artifactId>
<executions>
<execution>
<id>compile-protoc</id>
<goals>
<goal>protoc</goal>
</goals>
<configuration>
<protocVersion>${protobuf.version}</protocVersion>
<protocCommand>${protoc.path}</protocCommand>
<imports>
<param>${basedir}/../../../hadoop-common-project/hadoop-common/src/main/proto</param>
<param>${basedir}/../../../hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto</param>
<param>${basedir}/src/main/proto</param>
</imports>
<source>
<directory>${basedir}/src/main/proto</directory>
<includes>
<include>yarn_tensorflow_cluster_protos.proto</include>
<include>TensorflowCluster.proto</include>
</includes>
</source>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>package-yarn</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>


</project>
Loading

0 comments on commit 74e48e7

Please sign in to comment.