Introduce scan "recipes"

[pombredanne]

We have some minimal feature today to drive how scans are done: you can ignore some paths with --ignore which is OK but rather weak.

Instead of just adding more ways to ignore things based on other attributes (such as file types) we should have a new way to "drive" what is scanned and which scans are used.

We could for instance skip certain file types entirely. Or decide to perform only a package scan on a given subtree and no detailed license and copyright scan, etc.

This would be a new concept of scan recipes that can be stored in YAML files and passed as CLI options.
The general idea would be to have a bunch of conditions on attributes such as combos of path patterns (e.g. similar to the ignore), file types (e.g. ignore binaries), file names, file classification #426 , etc. and for a given condition we would list the scans that should be used (or the fact no scans should be done).

Recipes would typically be used at the front end of the scan, before scanning proper, though we could also envision similar recipes on the tail end of the scan such as filters ( e.g. --only-findings, etc)

@elmay61 this is based on your suggestion
@sschuberth ping as this likely will be of interest to you
@jpopelka ping: this is the way we can implement nicely skipping scanning of binaries as you do it in fabric8
@chinyeungli ping: you have quite a bit of experience in this area and your input is highly valued
@jdaguil ping: eventually this would be something where ABCMgr can be a helper to design and create the recipes
@yashdsaraf ping: this is something we would want to implement as plugins

[sschuberth]

Isn't this more or less what we call "policy" in #214 and what we call "user preferences and scan configuration" in #520? In fact, the latter ticket already is a collection that points at other tickets. Personally, I prefer the term "configuration" over "recipe", and thus would recommend to close this ticket in favor of continuing the discussion at #520.

[pombredanne]

@sschuberth you have a good point. Policies are a tad separate but eventually a config too. I will merge this back in #520

[pombredanne]

Note that we have implemented the bulk of these recipes as pipelines in scancode.io
See https://scancodeio.readthedocs.io/en/latest/scanpipe-concepts.html#pipelines for details.
Of note, there is no YAML involved. These are instead plain Python scripts
This is an example of a sophisticated recipe for Docker image scanning https://github.com/nexB/scancode.io/blob/main/scanpipe/pipelines/docker.py

There is still a need to have a better way to handle ignores and scanning multiple paths at once.
@mnonnenmacher @MarcelBochtler ping

[pombredanne]

These recipes are implemented as ScanCode.io pipelines alright now. Policies are created in DejaCode and usable locally in ScanCode. Closing!