Specification for 2.0 CPU compression API

doc/feature/cppPlugins.md

@@ -0,0 +1,175 @@
+C++ Plugin API for CPU compressors
+============
+
+This document specifies a modern API for the in-memory C plugin API. It
+will not replace the CPU-GPU transfer plugins.
+
+## Requirements
+
+* Support all features used by current CPU compression plugins
+    * Support 1D and 2D data layouts
+    * Drop alpha during compression of 2D data with alpha
+    * Lossy compression for 2D data
+    * Support for common image pixel layouts
+    * Support for different 1D data (byte, float, double, ...)
+    * Multiple compressors per plugin DSO
+* Information provided for each compressor:
+    * Compression speed (relative to RLE compressor)
+    * Compression ratio (relative to RLE compressor)
+    * 1D or 2D compressor
+    * Implemention drops alpha (for 2D)
+* Compressors may produce multiple output chunks
+* Compressor engines are instantiated to allow reuse of internal state
+* Easy to use API for plugin implementers
+
+## Dependency Changes
+
+* New: Lunchbox plugin API for loading plugins
+
+## API
+
+Change signature of Lunchbox::Plugin::HandlesFunc to:
+    typedef boost::function< bool ( InitDataT& ) > HandlesFunc;
+
+    const Plugins& Lunchbox::PluginFactory<..>::getPlugins();
+
+    enum class PluginToken
+    {
+        // 1D data
+        byte,
+        unsigned,
+        float,
+
+        // 2D data
+        rgba8,
+        rgb10a2,
+        ...
+    };
+
+    class PluginInfo
+    {
+    public:
+        std::string name;
+        PluginToken token;
+        bool alpha;
+        float quality; //!< Normalized 0..1 compression quality for 2D
+        float ratio; //!< Normalized 0..1 size after compression
+        float speed; //!< Relative speed compared to RLE compressor
+    };
+    typedef std::vector< PluginInfo > PluginInfos;
+
+    class PluginInitData
+    {
+    public:
+        // Input
+        PluginToken token;
+        float minQuality;
+        bool alpha;
+
+        // Output
+        PluginInfos infos;
+    }
+
+    class CompressorPlugin : public Lunchbox::Plugin< Compressor, PluginInfo >
+    {
+    public:
+        std::unique_ptr< Compressor > create( const PluginInfo& info );
+    };
+
+    class Compressor
+    {
+    public:
+        typedef lunchbox::Bufferb Result;
+        typedef std::vector< Result > Results;
+
+        // size[1] == 1 for 1D data
+        virtual const Results& compress( const void* const data,
+                                         const size_t size[2] ) = 0;
+        virtual const Result& decompress( const Results& input,
+                                          const size_t size[2] ) = 0;
+        void clear();
+
+        static void registerEngine( const PluginInfo& info,
+                                    const Constructor& ctor );
+    protected:
+        Results compressed;
+        Result uncompressed;
+
+    private:
+        static bool _handles( PluginInfo& info );
+    };
+
+
+## Examples
+
+
+
+    #include "zstd/lib/zstd.h"
+
+    namespace pression
+    {
+    namespace plugin
+    {
+    namespace
+    {
+    static PluginInfo _getInfo()
+    {
+    }
+
+    static bool _register()
+    {
+        Compressor::registerEngine(
+            { "pression::CompressorZSTD", pression::PluginToken::byte,
+              false, .47f, .25f },  CompressorZSTD::newInstance );
+        return true;
+    }
+
+    static const bool LB_UNUSED _initialized = _register();
+    }
+
+    const Results& CompressorZSTD::compress( const void* const data,
+                                             const size_t inSize[2] )
+    {
+        if( _results.empty( ))
+            _results.push_back( new pression::plugin::Compressor::Result );
+        size_t size = ZSTD_compressBound( nPixels );
+        _results[0]->reserve( size );
+
+        size = ZSTD_compress( _results[0]->getData(), size, inData,
+                              inSize[0] * inSize[1], 2 );
+        _results[0]->setSize( size );
+        return results;
+    }
+
+    const Result& CompressorZSTD::decompress( const Results& input,
+                                              const size_t size[2] );
+    {
+        if( input.empty( ))
+            return;
+
+        _result.resize( size[0] * size[1] );
+        ZSTD_decompress( _result.getData(), _result.getSize(),
+                         input[0].getData(), input[0].getSize( ));
+    }
+    }
+    }
+
+## Issues
+
+### Issue 1: How do we select compression engines?
+
+_Resolution: Make InitDataT& parameter given to handles() mutable_
+
+In the current Lunchbox plugin API, the plugin decides if it can handle
+a given request. Based on the return value of handles(), the first
+plugin available is typically instantiated.
+
+Making the templated InitDataT& given to the static handles() function
+mutable will allow our implementation to both return true/false if the
+plugin is compatible, and to fill in the plugin information into the
+given init data. Based on this information, Pression can select all
+plugins for a given request, and then select the best of the compatible
+ones. The lunchbox::PluginFactory needs to be introspectable.
+
+A given plugin DSO needs to fill in one information structure for each
+contained engine.