-
Notifications
You must be signed in to change notification settings - Fork 6
Home
The Heterogeneity-incorporating Workflow ApplicationMaster for YARN (Hi-WAY) provides the means to execute arbitrary scientific workflows on top of Apache's Hadoop 2.2.0 (YARN). In this context, scientific workflows are directed acyclic graphs (DAGs), in which nodes are executables accessible from the command line (e.g. tar, cat, or any other executable in the PATH of the worker nodes), and edges represent data dependencies between these executables.
Hi-WAY currently supports the workflow languages Pegasus DAX and Cuneiform as well as the workflow schedulers static round robin, HEFT, greedy queue and C3PO. Hi-WAY uses Hadoop's distributed file system HDFS to store the workflow's input, output and intermediate data. The ApplicationMaster has been tested for up to 320 concurrent tasks and is fault-tolerant in that it is able to restart failed tasks.
The installation instructions can be found here.