Proposal: split ColorSchemes.jl into two separate packages

[cormullion]

How about splitting the ColorSchemes.jl package into two separate packages?

1: ColorSchemes.jl - providing just the color schemes and the accessor functions

2: a new package ColorSchemeTools.jl - providing the tools for extracting color schemes from images, saving to images, conversion, sorting. etc.

I originally made ColorSchemes.jl simply to group sets of related colors together, avoiding dependencies on plotting packages (since I rarely plot). Ironically it now relies on Images.jl and Clustering.jl, which make the package a bit more dependency-heavy than the basic functionality might suggest.

After the split it should be possible for anyone to more easily add their preferred functionality to either of the ColorSchemes.jl or ColorSchemeTools.jl packages, such as libraries, lists, catalogues and categories, traits (whatever those are), interfaces, or anything else that people need, in order to steer the packages in the directions that people want to go. ... I'd prefer to at least keep the current basic usage unchanged, though:

using ColorSchemes
get(ColorSchemes.leonardo, 0.75)

since that's my primary use... :)

(I think this proposal could be either a temporary alternative to bike-shedding a brand new package design, or a useful thing to do anyway, even if a new package is ultimately the best way forward.)

@mkborregaard @chakravala @tpapp

[mkborregaard]

cc also @SimonDanisch
That's worth considering. Would

using Colorschemes.colorschemes
leo = colorgradient(:leonardo)
get(leo, 0.75) # or leo[0.75]

also work as an interface?
The submodule would solve the issue of 1) licenses and attribution, 2) namespacing (so many "rainbow" and "earth" out there). The colorgradient object would make it easier to pass around and broadcasting. And this would be easy to align with PlotUtils way of doing things.
Just to check, you also mean to drop ImageMagic, FileIO and QuartzImageIO into ColorSchemeTools, right?

[cormullion]

"You also mean to drop ImageMagic, FileIO and QuartzImageIO into ColorSchemeTools, right?"

Yes, assuming you wouldn't need them to merely access an array of RGB triplets by name.

"also work as an interface?"

As an additional interface, or the only one?

[chakravala]

As an additional interface, or the only one?

It's probably a good idea to have both a trait based system, with generators for particular classes of color maps, and also a unified interface which makes all of the pre-definitions available from one function.

[mkborregaard]

I guess I meant alternative but probably not :-)
OK, so if we make a colorgradient an object defined at it's most minimal as

struct ColorScheme
   x::Vector{RGB{Float64}} # or RGB{UInt8} is possibly good enough
end

instead of just a Vector as it currently is, but still leave them as const objects in module global scope, we could define get on that, and with exported submodules you could still get

using ColorSchemes
get(colorschemes.leonardo, 0.75) # get(colorbrewer.RdYlBu) etc

which would be close enough, and still support the syntax I suggest?

[cormullion]

"close enough"

yes, i could live with that... :)

[mkborregaard]

See the discussion here for some discussion of leaving schemes as objects in module global scope in terms of static compilation JuliaPlots/PlotUtils.jl#52 (IIUC it should be OK).

[mkborregaard]

In terms of splitting this - @peterkovesi 's excellent PerceptualColorMaps package actually also has 1) a very good range of color maps, and 2) advanced utilities depending on Images. Maybe it would even be worth considering consolidating that into each of these two new suggested packages?

[mkborregaard]

But the long and short of it - I think this is a good idea. @cormullion how much functionality would you suggest having here? It could simply be a repository of colorschemes - or it could have some added functionality that can be implemented dependency-free. E.g. in PlotUtils there is functionality for log-transformed scales, scales that correspond to certain values (not just 0-1), reversing colorscales, getindex acessing, searching across libraries for a given colorscheme name etc. I'd be happy to argue to the JuliaPlots org to move any functionality here - but it depends what the vision is?

[cormullion]

Since I'm a simple soul that likes simple solutions, I like the idea of a place where a bunch of these simple static colorschemes are stored 'as is' in a catalogue, with the more complex functionality stored elsewhere, accessible (say in ColorScheme(s)Tools?) but not 'in the way'.

As I was trying to convert some of the Matplotlib ones it occurred to me that perhaps some of these schemes were designed to be built and used 'on the fly' or 'live', which, if it's possible in Python, presumably even more possible in Julia. However, eventually you end up with complexity at the point of end use, which is what I don't like in software, where people are usually busy trying to solve their own problems, not navigating a maze made by the developers.

So perhaps the separate catalogue and toolbox paradigm is what I'm suggesting. :)

[chakravala]

Here's an idea: you could make a deps/build.jl script in the lighter pre-defined color maps catalogue package. This way, the build script can load the ColorTools package, which has generators needed for some of the presets, while the dependecy is not needed for the loading of the package. For example, some of the gradient maps might all be generated from the same ColorTool method, so in the build script for the catalogue package all you need is to specify the input sequence to generate those presets.

Since the presets were generated on the build script, you don't need to load the ColorTools or whatever package needed to generate the presets, since the presets are now written and loaded from the catalogue within the catalogue package (generated on Pkg.build() only once). Leave heavy machinery in shed.

That allows a unified interface for both generating the maps and storing them as fast presets. The generator ColorTools is the heavy package, while the catalogue package is light, and partially generated using the heavier package on the build script (but not loaded as dependency).

[peterkovesi]

Apologies for coming in a bit late here

Splitting ColorSchemes.jl into Schemes and Tools would seem a good idea

With regard to PerceptualColourMaps.jl a similar split into separate Schemes and Tools components had been suggested to me some time ago by @tpapp. I agree very much with this but have been consumed by other projects to do anything about it

The package could do with a bit of a rework

1. Its design is a bit of an embarrassment to me, it was some research code that got away... All the colour map spline definitions are embedded within the code. These should be defined in one or more separate files that are read in to generate the maps. Then people could more readily define and experiment with their own colour maps.
2. It is dependancy heavy. The diagnostic plots used for design work and the Tools component of the package are mostly to blame here.

I think the main thing for me to do is set things up so that PerceptualColourMap schemes can be readily output as a data file for incorporation in ColorSchemes.jl/data. This then achieves the split between Schemes and Tools (for me) and PerceptualColourMaps.jl can remain primarily as a tool for designing schemes rather than supplying them directly for plotting

With regard to any potential consolidation into ColorSchemes.jl in general I would be very happy for this to occur. However here I think I would prefer that the core components of PerceptualColourMaps.jl that are related to the design of maps remain as an independent package as it represents a piece of research work that I would like people to be able to reference as an independent item. Components that are not core to the research could certainly be consolidated if that was useful.

Anyway, I am open to any suggestions for PerceptualColourMaps people might have

[cormullion]

I went ahead and split the two packages up. You can see the manual of the new prototype (v3.0) of ColorSchemes.jl here, but you can't install either of them just yet, because they're still in METADATA purgatory. (My fault, I find this aspect of developing Julia packages quite difficult...)

ColorSchemes.jl v3.0 isn't compatible with v2.0... 😱

You can use them to decide whether a completely new design is a good idea or whether you can extend and/or re-use any of the code, using submodules or whatever, to build a more complete system.

I'm happy to help, to the extent of my abilities! :)