Skip to content

MichalPitr/map_reduce

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

29 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Map Reduce

Usage

Run master:

first build and push docker image. Mapper and Reducer jobs rely on pulling this image from container registry. Make sure to update the script to push to your own registry.

sh build_image.sh

then,

go run main.go --mode master --image <image> --input-dir /mnt/nfs/input/ --nfs-path /mnt/nfs/

For debugging, you can run mapper and reducer locally:

go run main.go --mode=mapper --input-dir /mnt/nfs/input --output-dir /mnt/nfs/job-test/ --file-range book-0-40
go run main.go --mode=reducer --reducer-id <id> --job-id <job-id> --nfs-path <nfs-mount-folder>

About

Building Map Reduce from scratch

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published