BNDL is a library for map-reduce based distributed processing akin to Apache Spark but is implemented in python (with a bit of cython).
BNDL can be installed through pip:
pip install bndl
The main commands to use BNDL are bndl-compute-shell
to open an interactive shell hooked up to
BNDL workers and bndl-compute-workers
to start workers seperately (e.g. throughout the
cluster).
Obtain a compute context in python scripts by importing ctx
from bndl.compute.run
:
from bndl.compute import ctx print(ctx.range(1000).map(str).map(len).stats())