migrate processors from common assets

[jsnoble]

• date_guard
• count_unique
• copy_metadata_field
• filter
• filter_by_required_fields
• filter_by_unknown_fields
• sample_exact
• sample_random
• json_parse
• remove_empty_fields
• set_field_conditional

[godber]

Here are some suggestions for renaming drop_doc. Given that the existing algorithm is probabilistic, we probably ought to convey that to the user. So I suggest we begin implementing a number of sample_FOO processors. Where the existing dropdoc processor is sample_random with the property probability, and we describe probability as the chance between 1 and 100 of the record being included in the final result set.

Edit: I think we should not include the shuffle option in this processor.

[godber]

Then we can implement sample_exact, which would have a percentage property, which we would describe as the percentage of records to be kept.

We would add an index to each record, shuffle the records, slice the result up to the percentage, then return that set, sorted by the indices.

The difference between sample_random and sample_exact is that random won't return a fixed percentage and sample_exact will.

Edit: renamed sample_shuffle to sample_exact.

[godber]

Maybe:

• keep_probability, probability_keep
• keep_percent, percent_keep

[godber]

@jsnoble I talked with Kimbro about the naming of the sample_random, and that seems OK. He suggested we name sample_shuffle to sample_exact to better match the users point of view rather than being implementation oriented.

We can file an issue to implement sample_exact in the future.

[godber]

You didn't add anything in the docs/operations folder.

[ciorg]

LGTM