add RecipesBase and RecipesPipeline docs

docs/make.jl

@@ -55,15 +55,16 @@ unique!(galleries_assets)
 
 ##################
 # `UnitfulRecipes`
-unitfulrecipes_src = joinpath(@__DIR__, "src/unitfulrecipes")
-notebooks = joinpath(unitfulrecipes_src, "notebooks")
+src_unitfulrecipes = "src/UnitfulRecipes"
+unitfulrecipes = joinpath(@__DIR__, src_unitfulrecipes)
+notebooks = joinpath(unitfulrecipes, "notebooks")
 
 execute = true  # set to true for executing notebooks and documenter!
 nb = false      # set to true to generate the notebooks
-for (root, _, files) in walkdir(unitfulrecipes_src), file in files
+for (root, _, files) in walkdir(unitfulrecipes), file in files
     last(splitext(file)) == ".jl" || continue
     ipath = joinpath(root, file)
-    opath = replace(ipath, "src/unitfulrecipes" => "src/generated") |> splitdir |> first
+    opath = replace(ipath, src_unitfulrecipes => "src/generated") |> splitdir |> first
     Literate.markdown(ipath, opath, documenter = execute)
     nb && Literate.notebook(ipath, notebooks, execute = execute)
 end
@@ -85,7 +86,37 @@ const PAGES = Any[
         "Subplot Attributes" => "generated/attributes_subplot.md",
         "Axis Attributes" => "generated/attributes_axis.md",
         "Layouts" => "layouts.md",
-        "Recipes" => "recipes.md",
+        "Recipes" => [
+            "Overview" => "recipes.md",
+            "RecipesBase" => [
+                "Home" => "RecipesBase/index.md",
+                "Recipes Syntax" => "RecipesBase/syntax.md",
+                "Recipes Types" => "RecipesBase/types.md",
+                "Internals" => "RecipesBase/internals.md",
+                "Public API" => "RecipesBase/api.md",
+            ],
+            "RecipesPipeline" => [
+                "Home" => "RecipesPipeline/index.md",
+                "Developer manual" => [
+                    "Public API" => "RecipesPipeline/api.md",
+                    "Recipes" => "RecipesPipeline/recipes.md",
+                ],
+                "Reference" => "RecipesPipeline/reference.md",
+                # "Source code" => joinpath.("generated", [
+                #     "RecipesPipeline.md",
+                #     "api.md",
+                #     "user_recipe.md",
+                #     "plot_recipe.md",
+                #     "type_recipe.md",
+                #     "series_recipe.md",
+                #     "group.md",
+                #     "recipes.md",
+                #     "series.md",
+                #     "group.md",
+                #     "utils.md"
+                # ])
+            ],
+        ],
         "Colors" => "colors.md",
         "ColorSchemes" => "generated/colorschemes.md",
         "Animations" => "animations.md",
@@ -98,12 +129,12 @@ const PAGES = Any[
     "Ecosystem" => [
         "StatsPlots" => "generated/statsplots.md",
         "GraphRecipes" => [
-            "Introduction" => "graphrecipes/introduction.md",
-            "Examples" => "graphrecipes/examples.md",
+            "Introduction" => "GraphRecipes/introduction.md",
+            "Examples" => "GraphRecipes/examples.md",
             "Attributes" => "generated/graph_attributes.md",
         ],
         "UnitfulRecipes" => [
-            "Introduction" => "unitfulrecipes/unitfulrecipes.md",
+            "Introduction" => "UnitfulRecipes/unitfulrecipes.md",
             "Examples" => [
                 "Simple" => "generated/unitfulrecipes_examples.md",
                 "Plots" => "generated/unitfulrecipes_plots.md",

docs/src/RecipesBase/api.md

@@ -0,0 +1,3 @@
+```@autodocs
+Modules = [RecipesBase]
+```

docs/src/RecipesBase/index.md

@@ -0,0 +1,15 @@
+# RecipesBase.jl
+
+**Author: Thomas Breloff (@tbreloff)**
+
+RecipesBase is a lightweight Package without dependencies that allows to define custom visualizations with the [`@recipe`](@ref) macro.
+
+Package developers and users can define recipes to tell [Plots.jl](https://github.com/JuliaPlots/Plots.jl) how to plot custom types without depending on it.
+Furthermore, recipes can be used for complex visualizations and new series types.
+Plots, for example, uses recipes internally to define histograms or bar plots.
+[StatsPlots.jl](https://github.com/JuliaPlots/StatsPlots.jl) and [GraphRecipes.jl](https://github.com/JuliaPlots/GraphRecipes.jl) extend Plots functionality for statistical plotting and visualization of graphs.
+
+RecipesBase exports the [`@recipe`](@ref) macro which provides a nice syntax for defining plot recipes.
+Under the hood [`@recipe`](@ref) defines a new method for `RecipesBase.apply_recipe` which is called recursively in Plots at different stages of the argument processing pipeline.
+This way other packages can communicate with Plots, i.e. define custom plotting recipes, only depending on RecipesBase.
+Furthermore, the convenience macros [`@series`](@ref), [`@userplot`](@ref) and [`@shorthands`](@ref) are exported by RecipesBase.

docs/src/RecipesBase/internals.md

@@ -0,0 +1,149 @@
+## RecipesBase
+
+The [`@recipe`](@ref) macro defines a new method for `RecipesBase.apply_recipe`.
+```julia
+@recipe function f(args...; kwargs...)
+```
+defines
+```julia
+RecipesBase.apply_recipe(plotattributes, args...; kwargs...)
+```
+returning a `Vector{RecipeData}` where `RecipeData` holds the `plotattributes` Dict and the arguments returned in [`@recipe`](@ref) or in [`@series`](@ref).
+```julia
+struct RecipeData
+    plotattributes::AbstractDict{Symbol,Any}
+    args::Tuple
+end
+```
+This function sets and overwrites entries in `plotattributes` and possibly adds new series.
+- `attr --> val` translates to `haskey(plotattributes, :attr) || plotattributes[:attr] = val`
+- `attr := val` sets `plotattributes[:attr] = val`.
+- [`@series`](@ref) allows to add new series within [`@recipe`](@ref). It copies `plotattributes` from [`@recipe`](@ref), applies the replacements defined in its code block and returns corresponding new `RecipeData` object.
+    !!! info
+        [`@series`](@ref) have to be defined as a code block with `begin` and `end` statements.
+        ```julia
+        @series begin
+            ...
+        end
+        ```
+
+So `RecipesBase.apply_recipe(plotattributes, args...; kwargs...)` returns a `Vector{RecipeData}`.
+Plots can then recursively apply it again on the `plotattributes` and `args` of the elements of this vector, dispatching on a different signature.
+
+
+## Plots
+
+The standard plotting commands
+```julia
+plot(args...; plotattributes...)
+plot!(args...; plotattributes...)
+```
+and shorthands like `scatter` or `bar` call the core internal plotting function `Plots._plot!`.
+```julia
+Plots._plot!(plt::Plot, plotattributes::AbstractDict{Symbol, Any}, args::Tuple)
+```
+
+In the following we will go through the major steps of the preprocessing pipeline implemented in `Plots._plot!`.
+
+#### Preprocess `plotattributes`
+Before `Plots._plot!` is called and after each recipe is applied, `preprocessArgs!`  preprocesses the `plotattributes` Dict.
+It replaces aliases, expands magic arguments, and converts some attribute types.
+- `lc = nothing` is replaced by `linecolor = RGBA(0, 0, 0, 0)`.
+- `marker = (:red, :circle, 8)` expands to `markercolor = :red`, `markershape = :circle` and `markersize = 8`.
+
+#### Process User Recipes
+
+In the first step, `_process_userrecipe` is called.
+
+```julia
+kw_list = _process_userrecipes(plt, plotattributes, args)
+```
+It converts the user-provided `plotattributes` to a vector of `RecipeData`.
+It recursively applies `RecipesBase.apply_recipe` on the fields of the first element of the `RecipeData` vector and prepends the resulting `RecipeData` vector to it.
+If the `args` of an element are empty, it extracts `plotattributes` and adds it to a Vector of Dicts `kw_list`.
+When all `RecipeData` elements are fully processed, `kw_list` is returned.
+
+#### Process Type Recipes
+
+After user recipes are processed, at some point in the recursion above args is of the form `(y, )`, `(x, y)` or `(x, y, z)`.
+Plots defines recipes for these signatures.
+The two argument version, for example, looks like this.
+
+```julia
+@recipe function f(x, y)
+    did_replace = false
+    newx = _apply_type_recipe(plotattributes, x)
+    x === newx || (did_replace = true)
+    newy = _apply_type_recipe(plotattributes, y)
+    y === newy || (did_replace = true)
+    if did_replace
+        newx, newy
+    else
+        SliceIt, x, y, nothing
+    end
+end
+```
+
+It recursively calls `_apply_type_recipe` on each argument until none of the arguments is replaced.
+`_apply_type_recipe` applies the type recipe with the corresponding signature and for vectors it tries to apply the recipe element-wise.
+When no argument is changed by `_apply_type_recipe`, the fallback `SliceIt` recipe is applied, which adds the data to `plotattributes` and returns `RecipeData` with empty args.
+
+#### Process Plot Recipes
+
+At this stage all arguments have been processed to something Plots supports.
+In `_plot!` we have a `Vector{Dict}` `kw_list` with an entry for each series and already populated `:x`, `:y` and `:z` keys.
+Now `_process_plotrecipe` is called until all plot recipes are processed.
+
+```julia
+still_to_process = kw_list
+kw_list = KW[]
+while !isempty(still_to_process)
+    next_kw = popfirst!(still_to_process)
+    _process_plotrecipe(plt, next_kw, kw_list, still_to_process)
+end
+```
+
+If no series type is set in the Dict, `_process_plotrecipe` pushes it to `kw_list` and returns.
+Otherwise it tries to call `RecipesBase.apply_recipe` with the plot recipe signature.
+If there is a method for this signature and the seriestype has changed by applying the recipe, the new `plotattributes` are appended to `still_to_process`.
+If there is no method for the current plot recipe signature, we append the current Dict to `kw_list` and rely on series recipe processing.
+
+After all plot recipes have been applied, the plot and subplots are set-up.
+```julia
+_plot_setup(plt, plotattributes, kw_list)
+_subplot_setup(plt, plotattributes, kw_list)
+```
+
+#### Process Series Recipes
+
+We are almost finished.
+Now the series defaults are populated and `_process_seriesrecipe` is called for each series .
+
+```julia
+for kw in kw_list
+    # merge defaults
+    series_attr = Attr(kw, _series_defaults)
+    _process_seriesrecipe(plt, series_attr)
+end
+```
+
+If the series type is natively supported by the backend, we finalize processing and pass the series along to the backend.
+Otherwise, the series recipe for the current series type is applied and `_process_seriesrecipe` is called again for the `plotattributes` in each returned `RecipeData` object.
+Here we have to check again that the series type changed.
+Due to this recursive processing, complex series types can be built up by simple blocks.
+For example if we add an `@show st` in `_process_seriesrecipe` and plot a histogram, we go through the following series types:
+
+```julia
+plot(histogram(randn(1000)))
+```
+```julia
+st = :histogram
+st = :barhist
+st = :barbins
+st = :bar
+st = :shape
+```
+```@example
+using Plots # hide
+plot(histogram(randn(1000))) #hide
+```

docs/src/RecipesBase/syntax.md

@@ -0,0 +1,121 @@
+```@setup syntax
+using Plots, Random
+Random.seed!(100)
+default(legend = :topleft, markerstrokecolor = :auto, markersize = 6)
+```
+
+# Recipes Syntax
+
+The syntax in the [`@recipe`](@ref) macro is best explained using an example.
+Suppose, we have a custom type storing the results of a simulation `x` and `y` and a measure `ε` for the maximum error in `y`.
+
+```@example syntax
+struct Result
+    x::Vector{Float64}
+    y::Vector{Float64}
+    ε::Vector{Float64}
+end
+```
+
+If we want to plot the `x` and `y` values of such a result with an error band given by `ε`, we could run something like
+```@example syntax
+res = Result(1:10, cumsum(rand(10)), cumsum(rand(10)) / 5)
+
+using Plots
+
+# plot the error band as invisible line with fillrange
+plot(
+    res.x,
+    res.y .+ res.ε,
+    xlabel = "x",
+    ylabel = "y",
+    fill = (res.y .- res.ε, :lightgray, 0.5),
+    linecolor = nothing,
+    primary = false, # no legend entry
+)
+
+# add the data to the plots
+plot!(res.x, res.y, marker = :diamond)
+```
+
+Instead of typing this plot command over and over for different results we can define a **user recipe** to tell Plots what to do with input of the type `Result`.
+Here is an example for such a user recipe with the additional feature to highlight datapoints with a maximal error above a certain threshold `ε_max`.
+
+```@example syntax
+@recipe function f(r::Result; ε_max = 0.5)
+    # set a default value for an attribute with `-->`
+    xlabel --> "x"
+    yguide --> "y"
+    markershape --> :diamond
+    # add a series for an error band
+    @series begin
+        # force an argument with `:=`
+        seriestype := :path
+        # ignore series in legend and color cycling
+        primary := false
+        linecolor := nothing
+        fillcolor := :lightgray
+        fillalpha := 0.5
+        fillrange := r.y .- r.ε
+        # ensure no markers are shown for the error band
+        markershape := :none
+        # return series data
+        r.x, r.y .+ r.ε
+    end
+    # get the seriescolor passed by the user
+    c = get(plotattributes, :seriescolor, :auto)
+    # highlight big errors, otherwise use the user-defined color
+    markercolor := ifelse.(r.ε .> ε_max, :red, c)
+    # return data
+    r.x, r.y
+end
+```
+
+Let's walk through this recipe step by step.
+First, the function signature in the recipe definition determines the recipe type, in this case a user recipe.
+The function name `f` in is irrelevant and can be replaced by any other function name.
+[`@recipe`](@ref) does not use it.
+In the recipe body we can set default values for [Plots attributes](http://docs.juliaplots.org/latest/attributes/).
+```
+attr --> val
+```
+This will set `attr` to `val` unless it is specified otherwise by the user in the plot command.
+```
+plot(args...; kw..., attr = otherval)
+```
+Similarly we can force an attribute value with `:=`.
+```
+attr := val
+```
+This overwrites whatever the user passed to `plot` for `attr` and sets it to `val`.
+!!! tip
+    It is strongly recommended to avoid using attribute aliases in recipes as this might lead to unexpected behavior in some cases.
+    In the recipe above `xlabel` is used as aliases for `xguide`.
+    When the recipe is used Plots will show a warning and hint to the default attribute name.
+    They can also be found in the attribute tables under http://docs.juliaplots.org/latest/attributes/.
+
+We use the [`@series`](@ref) macro to add a new series for the error band to the plot.
+Within an [`@series`](@ref) block we can use the same syntax as above to force or set default values for attributes.
+
+In [`@recipe`](@ref) we have access to `plotattributes`. This is an `AbstractDict` storing the attributes that have been already processed at the current stage in the Plots pipeline.
+For user recipes, which are called early in the pipeline, this mostly contains the keyword arguments provided by the user in the `plot` command.
+In our example we want to highlight data points with an error above a certain threshold by changing the marker color.
+For all other data points we set the marker color to whatever is the default or has been provided as keyword argument.
+We can do this by getting the `seriescolor` from `plotattributes` and defaulting to `auto` if it has not been specified by the user.
+
+Finally, in both, [`@recipe`](@ref)s and [`@series`](@ref) blocks we return the data we wish to pass on to Plots (or the next recipe).
+
+!!! compat
+    With RecipesBase 1.0 the `return` statement is allowed in [`@recipe`](@ref) and [`@series`](@ref).
+
+With the recipe above we can now plot `Result`s with just
+
+```@example syntax
+plot(res)
+```
+
+or
+
+```@example syntax
+scatter(res, ε_max = 0.7, color = :green, marker = :star)
+```