New command to apply Compliance objects to dataset

[igorbrigadir]

There should be a command to actually apply the compliance objects to a dataset. Currently you can grab a list of IDs, get the compliance job results, but there's no command that will do the actual filtering.

I'm thinking of a command like:

twarc2 compliance apply dataset.json compliance.json result.json

That will take a dataset, compliance results, and output a clean, compliant dataset. (Either full objects or IDs)

[SamHames]

Big thumbs up from me. I'd be happy to collaborate on that one, if for no other reason than to have a consistent way to do this kind of filtering (ie, do we remove referenced tweets that were part of the deleted tweets but still present in includes?)

[igorbrigadir]

Yeah sure thing! I haven't started this at all yet