Tasks

Bootstrapping a Clojure application or building a Clojure project for deployment or distribution is a complex process. If we want to be able to fully leverage the richness of the platform we must be able to integrate all of the various components and technologies that comprise a modern application or library into a package that can be easily consumed. It is important to do as much of this work as possible at build time, as this frees consumers from having to know or care how to perform this work correctly.

This demands a highly flexible, composable architecture. Ideally, the build process consists of self-contained, modular components combined in different ways to handle the specific requirements of the project at hand.

Note: You can see the boot tasks available on your system by typing boot --help. The output will also include custom tasks and tasks pulled in, in build.boot, from external dependencies. Typically boot tasks are documented in the source code in the form of API documentation (docstrings) which can be shown via boot taskname --help. Additionally all built-in tasks are listed here.

Overview

Boot adopts a model based on another architecture that has proven to be flexible, powerful, and modular in demanding real-world use: Ring middleware.

The diagram above shows how a web request (the orange arrow) is processed by layers of middleware (the concentric circles). The request is handled by the view (the black circle in the center), where it becomes the response, which then percolates back through the middleware and is sent off to the client.

What's so great about this setup?

• Middleware can be composed uniformly: any two middleware can be composed using normal function composition to form new middleware that can then be further composed, etc.

• Handlers are decoupled from each other: the only communication between handlers occurs via immutable request/response maps. A handler neither knows nor cares what the next handler in the stack does or how it operates.

Boot Task Model

In boot tasks provide the middleware functionality.

• Tasks are middleware factories – functions that return middleware. They may take configuration arguments and create state that is closed over by the middleware they return.

• Middleware return handlers – middleware expect a handler when called and return a handler. Middleware are composed using comp.

• Handlers process a fileset – immutable fileset objects serve the role of Ring's request and response maps. Handlers expect a fileset when called and return a fileset (the result of applying the next handler, usually).

Boot process flow:

The diagram above illustrates how a pipeline of task middleware processes a build. This could happen as a single command line invocation, as a series of incremental builds using the watch task, or even from the REPL via the boot macro.

From top to bottom:

1. Project directories – The blue folders represent the various directories containing build files that comprise the project. These are divided into two categories:

   • The target directory is write-only. Boot creates final artifacts here, but it never reads from any of the files in this directory.

   • The resource, source, and asset directories are read-only. Boot will not modify these directories or the files they contain in any way.

2. Directory watch service – Uses the native filesystem events API to monitor the project directories. Project files are copied into temporary directories managed by boot whenever they are modified. This effectively decouples the project files from the class path. Decoupling project files and classpath allows tasks to remove and add files to the classpath without modifying the user's project files.

   Note that this is not the same as the watch task. This is a background thread used internally by boot. No tasks are run when these files change.

3. Initial fileset object – When a build is started an initial fileset object is created by boot and populated with project files from the temporary directories (only those files that are part of the project; no files created by previous builds are included). This initial fileset is passed to the outermost middleware in the build pipeline for processing.

4. Middleware pre-processing – Each layer of middleware is passed a fileset object for processing. New files are created in anonymous temporary directories that are local to the task and managed by boot. These files can be added to the fileset to obtain a new immutable fileset object, which can then be committed to disk and passed to the next handler in the pipeline.

   In the diagram, the handlers execute left to right, terminating with the boot handler.

5. Final artifacts emitted – The boot handler copies artifacts from the fileset into the target directory, removing any stale artifacts that may be present there from a previous build. The fileset is then returned back up the stack.

6. Middleware post-processing – After the boot handler executes, the stack unwinds. As the fileset percolates back up through the middleware layers, each handler has an opportunity to perform side effects, if applicable.

   Tasks don't modify the fileset during the post-processing phase.

The Role of Tasks in Boot

What roles are appropriate for tasks to perform in boot? Just about any process or operation can be provided as a task. The functionality of Leiningen tasks, subtasks, plugins, middleware, profiles, injections, etc. can all be implemented in boot as modular, composable tasks.

The rest of this document will demonstrate how these various patterns translate to the world of tasks, and some general guidelines are presented as a basic set of best-practices to follow.

Task Anatomy

All tasks have the following anatomical structure:

(deftask foop                                   ; [1]
  "Task docstring."                             ; [2]
  [...]                                         ; [3]
  (let [...]                                    ; [4]
    (fn middleware [next-handler]               ; [5]
      (fn handler [fileset]                     ; [6]
        ...                                     ; [7]
        (let [fileset' (... fileset)            ; [8]
              fileset' (commit! fileset')       ; [9]
              result   (next-handler fileset')] ; [10]
          ...                                   ; [11]
          result)))))                           ; [12]

1. The deftask macro – defines a named task.
2. Docstring – describes the purpose of the task.
3. Task arguments – these are described in the Task-Options-DSL page.
4. Local state – local bindings are closed over by task middleware.
5. Middleware – middleware take the next handler function as arguments.
6. Handler – handlers take immutable fileset objects as arguments.
7. Pre-processing – build tasks will do most of their work here.
8. Fileset operations – files can be added or removed to obtain a new fileset.
9. Commit to disk – the underlying filesystem is synced to the fileset object.
10. Call next handler – the next handler continues the build process.
11. Post-processing – tasks can perform only side effects here.
12. Return – the next handler's result is passed back up the stack.

The Null Task

A decent starting point in the exploration of tasks is the null task–a task that just passes through to the next task without doing anything itself.

You could write one:

(deftask null-task
  "Does nothing."
  []
  (fn null-middleware [next-handler]
    (fn null-handler [fileset]
      (next-handler fileset))))

Conveniently enough, there already is such a thing: (null-task) evaluates to a function that is equivalent to clojure.core/identity, so the following is equivalent:

(deftask null-task
  "Does nothing."
  []
  clojure.core/identity)

Pre and Post Tasks

Most tasks do their work either before or after calling the next task in the stack. These are called pre and post tasks. Boot provides macros that make creating these kinds of tasks easier by eliminating some boilerplate.

They also inject assertions that ensure that fileset objects are correctly passed down the stack and that the fileset is not manipulated as it percolates back up through the middleware.

Pre Tasks

Pre tasks have the following structure:

(deftask pre1
  "A pre task."
  [...]
  (let [...]
    (fn [next-handler]
      (fn [fileset]
        ... ; work is done here
        (next-handler (commit! (... fileset)))))))

The with-pre-wrap macro equivalent is a bit simpler:

(deftask pre1
  "A pre task."
  [...]
  (let [...]
    (with-pre-wrap fileset
      ... ; work is done here
      (commit! (... fileset)))))

The macro binds the fileset passed to the hander and evaluates the body, and passes the result to the next handler. The value returned by the next handler is then returned back up the stack.

Post Tasks

Post tasks generally look like this:

(deftask post1
  "A post task."
  [...]
  (let [...]
    (fn [next-handler]
      (fn [fileset]
        (let [result (next-handler fileset)]
          ... ; side effects are performed here
          result)))))

The with-post-wrap macro eliminates some boilerplate:

(deftask post1
  "A post task."
  [...]
  (let [...]
    (with-post-wrap fileset
      ... ; side effects are performed here
      )))

The macro passes the incoming fileset object to the next handler, binds the result, evaluates the body for side effects, and returns the bound result back up the stack.

Pre-Post Decomposition

Tasks that work in both pre and post phases may sometimes be decomposed into separate pre and post tasks:

(deftask pre-post1
  "A pre-post task."
  [...]
  (let [...]
    (fn [next-handler]
      (fn [fileset]
        ... ; work is done here
        (let [fileset' (... fileset)
              fileset' (commit! fileset')
              result   (next-handler fileset')]
          ... ; side effects are performed here
          result)))))

could be rewritten as:

(deftask pre-post1
  "A pre-post task."
  [...]
  (let [...]
    (comp
      (with-pre-wrap fileset
        ... ; work is done here
        (commit! (... fileset)))
      (with-post-wrap fileset
        ... ; side effects are performed here
        ))))

This decomposition can sometimes result in cleaner code.

Control Tasks

Some tasks serve to control the build process by varying how or when they call the next task in the stack. One could imagine a repeat task that calls the next handler repeatedly:

(deftask repeat
  "Run repeatedly every 5s."
  []
  (fn [next-handler]
    (fn [fileset]
      (while true
        (Thread/sleep 5000)
        (reset-build!)         ; [1]
        (try (-> fileset
                 reset-fileset ; [2]
                 commit!
                 next-handler)
             (catch Throwable error (.printStackTrace error)))))))

The annotated lines above reset the build state between iterations. Normally, this is handled automatically by boot. Control tasks, however, need to perform these operations explicitly because unlike normal tasks they do not return control back up the stack.

1. reset-build! – restores boot's internal warning counters etc. to initial states in preparation for the next build iteration.

2. reset-fileset – returns a new fileset reflecting changes to the user's project directories. This is needed because user files may change between iterations. Files in the fileset created by tasks that ran earlier in the pipeline are kept.