presentation.org

Using Apache Mesos with Clojure

Think of something catchy

Background

What’s Mesos?

apache.mesos.org

Originally Written by Benjamin Hindman

Cluster resouce manager

Uses CGroups and Containers to isolate tasks

Used by Twitter, Airbnb, Groupon, Apple, among a few

How does it work?

Based Around Frameworks

Implement a Scheduler and Executors

Connect to Mesos with Drivers

Mesos track cluster usage and offers resources to Schedulers

clj-mesos

dgrnbrg/clj-mesos

Production Ready

Slightly out of date with mainline Mesos

Some Other Tools for Development

Docker and Fig

or

Vagrant

Let’s Build a Framework

Scheduler

Manages resources of your framework

Lauches Tasks

Interact through Callbacks

(clj-mesos.scheduler/scheduler
  (disconnected [driver])
  (error [driver message])
  (executorLost [driver executor-id slave-id status])
  (frameworkMessage [driver executor-id])
  (offerRescinded [driver offer-id])
  (registered [driver framework-id master-info])
  (reregistered [driver master-info])
  (resourceOffers [driver offers])
  (slaveLost [driver slave-id])
  (statusUpdate [driver status]))

Executor

Manages Executor of Tasks

Can be ‘Corse-Grained’ or ‘Fine-Grained’

Like the Scheduler interact through Callbacks

(clj-mesos.executor/executor
 (disconnected [driver])
 (error [driver message])
 (frameworkMessage [driver data])
 (killTask [driver task-id])
 (launchTask [driver task-info])
 (registered [driver executor-info framework-info slave-info])
 (reregistered [driver slave-info])
 (shutdown [driver]))

Hello Mesos

Has a Scheduler, but no Executor

Just Prints “Hello Mesos”

(clj-mesos.scheduler/launch-tasks driver
                                  [{:task-id 1
                                    :name "Task"
                                    :slave-id (:slave-id offer)
                                    :resources {:cpus 1.0
                                                :mem 128.0}
                                    :executor {}
                                    :command {}}])

Hello Mesos [Slight Return]

Has a Scheduler and a Executor

Demonstrates a status update

(clj-mesos.executor/send-status-update driver
                                       {:task-id (:task-id task-info)
                                        :state :task-running})

RENDLER!

A Framework that does something

FrameworkMessages

Allows Executors and Schedulers to Communicate

Only makes a best attempt at delivering the message

(clj-mesos.executor/send-framework-message driver
                                           (.getBytes "A Message!"))

Other Topics

High Availabilty Mode

Crash Recovery

Mesos State

What else is there?

Use existing distributed applications

Storm

Jenkins

Cassandra

Spark

Hadoop

Manage Traditional Applications

Marathon

Chronos

Similar to Mesos

Deis

Flynn

Fleet/CoreOS

Google Omega