[super] Continuous Data Integration workflows around Data Packages

[rufuspollock]

For a long time we've discussed "continuous data integration" and doing this with Data Packages. This is a discussion space for these ideas.

What do we mean by "Continuous Data Integration"?

Continuous Integration with code means automated running of tests (and sometimes deployments) whenever new code is pushed.

For data, this would mean automatically validating (testing) data and metadata on each new contribution and running any deployment tests.

Automatically validating a Data Package data against its schema on each commit (assuming the data was stored in git)

Sub parts

• Demo example - https://github.com/frictionlessdata/ex-continuous-data-integration
• Proper Library - Factor this out into a mini library e.g. datapackage-check-py example-continuous-data-integration#2
• Section in Tutorial - Tutorial: Validating Data #240

Initial Implementation

We could start out piggy-backing on existing code CI services such as Travis: instead of running code test we run data validate tests as a code test.

By hand version

• create a demo repo - probably clone a core data package dataset
• implement a tiny test script in scripts called datatest.py or similar. Use goodtables to test the data
• turn on travis ci and run the test script
  • bonus points for somehow showing pretty output from goodtables

Standardize:

• make a simple library: goodtables-ci or similar ;-) - Factor this out into a mini library e.g. datapackage-check-py example-continuous-data-integration#2

[rufuspollock]

Working here: https://github.com/frictionlessdata/ex-continuous-data-integration

[rufuspollock]

Working pretty well.

[danfowler]

This issue was moved to #48