Yuheng Chen [email protected]
Combining Spark with elegant Ruby closure on JVM.
Download Spark 1.6 distribution.
Note that if you use rvm or similiar tools, use
rvm use system
to avoid loading installed gems.
Download JRuby 9.1.0.0
snapshot
to ./libs, then build the jruby spark:
rake package`
The release package is available in pkg.
Run JRuby Spark REPL shell:
export SPARK_HOME=$YOUR_SPARK_HOME
./bin/jruby-spark-repl.sh
By default, jruby-spark-repl.sh use spark-submit to submit the REPL job
locally, similiar to pyspark.
Run the word count example:
$sc.text_file('SOME_TXT_FILE.txt').flat_map{|x| x.downcase.split(/[^\w']+/)}
.map_to_pair{|x| tuple(x, 1)}.reduce_by_key(:+).collect_as_map
Note how well Ruby fits into Spark framework:
- Block and lambda
- symbol to
proc - use powerful utilities in Ruby's standard library