A MapReduce server using web browsers as workers, written in Go.
This is a proof of concept.
This is an implementation of a MapReduce master server, able to utilize the free computing capabilities of web browsers as MapReduce workers.
Communication is done via WebSockets, which allow consistent full-duplex comminucation between master and workers.
Worker computation is done via WebWorkers, which allow background processing to take place, while the main UI thread is left uninterrupted.
The server is written in Go and is highly concurrent, able to make full use of all available CPUs and memory. Every worker connection is managed by a cheap goroutine. This makes the master able to handle hundreds of thousands of connections at any given moment on a modern machine.
Resources used in the implementation:
- Google Research: MapReduce: Simplified Data Processing on Large Clusters
- WebSocket RFC
- WebRTC RFC
- Google/MR4C
- Gorrila/Websocket
- SockJS
- Gopher images are courtesy of Renee French.
The things left "unticked" are potential improvements, as a future project.
- Master-Worker protocol
- Protocol unit tests
- Master algorithm and job management
- Algorithm unit tests
- Algorithm benchmark tests
- Websocket worker management using SockJS
- Unit tests
- Websocket integration tests
- Master
- Master unit tests
- Master benchmark tests
- WebRTC - Necessary in order to fully implement MapReduce
- HTTP Server
- Server unit tests
- Website pages (HTML, CSS).
- Client (Javascript)
- Client unit tests
- Master-Worker integration tests
- Extras
- Travis
- Coveralls
- Install Go: You should probably either use [gvm][gvm] or install version >= 1.4 manually.
- Clone this repo into:
$GOPATH/src/github.com/oryband/go-web-mapreduce.
# download dependencies
$ go get -t -u -v ./...
# execute
$ go run server/server.go server/api.go server/views.go# run all tests including sub-packages, and output coverage.
$ go test -v -cover ./...