Add migration for data sources.

[blkt]

Summary

This migration adds support for three additional tables, data_sources and data_sources_functions containing the definition of a single data source in a many-to-one relationship, and a rule_type_data_sources table representing the many-to-many relationship between rule types and data sources.

Change Type

Mark the type of change your PR introduces:

• Bug fix (resolves an issue without affecting existing features)
• Feature (adds new functionality without breaking changes)
• Breaking change (may impact existing functionalities or require documentation updates)
• Documentation (updates or additions to documentation)
• Refactoring or test improvements (no bug fixes or new functionality)

Testing

Manually tested that migrate up and migrate down work as expected.

Review Checklist:

• Reviewed my own code for quality and clarity.
• Added comments to complex or tricky code sections.
• Updated any affected documentation.
• Included tests that validate the fix or feature.
• Checked that related changes are merged.

[JAORMX]

gotta run make gen

[coveralls]

coverage: 54.624% (-0.007%) from 54.631%
when pulling f8bdbdd on feat/migration-data-sources
into 3cc4ae8 on main.

[JAORMX]

LGTM. Thanks!

[blkt]

@JAORMX I forgot to add timestamps, it should be good to go now.

[evankanderson]

A few nits about case insensitivity in names, as we don't want people to be able to do tricky things with datafetch and dataFetch doing different things.

[evankanderson]

I think I missed this in the design -- how do functions and data sources interact?

[blkt]

The idea is that a data source is a "container" of functions that also defines a namespace.
As an example, this might be a shortened version of Trusty API v2 data source

version: v1
type: data-source
name: trusty
context:
  project: foo
rest:
 summary:
    endpoint: "https://api.trustypkg.dev/v2/summary?package_name={package_name}
    method: GET
    args:
      - package_name:
        schema:
          type: string
 pkg:
    endpoint: "https://api.trustypkg.dev/v2/pkg?package_name={package_name}"
    method: GET
    args:
      package_name:
        schema:
          type: string

This YAML defines data source trusty having "functions" summary and pkg. Eventually, they would be available to Rego engine as minder.datasource.trusty.summary(...) and minder.datasource.trusty.pkg(...). Some field names might be different, and some lists could be maps and vice versa, but that's about it.

Getting back to the comment, we thought that saving all that informatino in a single row of table data_sources was too coarse, and decided to add a separate table containing functions. I'm not sure I answered your question.

[evankanderson]

Thanks, the last time I'd reviewed the doc, multiple endpoints for the same data source weren't in it.

The tables make sense to me.