[DOCS] Add new topic about data deduplication

[dedemorton]

Closes #13739

[dedemorton]

@ycombinator I can't the add_id or fingerprint processors to work in my config (using Metricbeat 7.6 BC1). I see the following error:

2020-01-16T15:36:33.232-0800	ERROR	instance/beat.go:933	Exiting: error initializing 
processors: the processor action add_id does not exist. Valid actions: add_labels, copy_fields, 
decompress_gzip_field, rename, community_id, registered_domain, add_observer_metadata, 
decode_base64_field, drop_event, drop_fields, add_cloud_metadata, add_host_metadata, 
add_locale, add_kubernetes_metadata, add_fields, add_tags, decode_json_fields, truncate_fields, 
dissect, extract_array, include_fields, add_process_metadata, convert, dns, add_docker_metadata
Exiting: error initializing processors: the processor action add_id does not exist. Valid actions: 
add_labels, copy_fields, decompress_gzip_field, rename, community_id, registered_domain, 
add_observer_metadata, decode_base64_field, drop_event, drop_fields, add_cloud_metadata, 
add_host_metadata, add_locale, add_kubernetes_metadata, add_fields, add_tags, 
decode_json_fields, truncate_fields, dissect, extract_array, include_fields, add_process_metadata, 
convert, dns, add_docker_metadata

[ycombinator]

@dedemorton Thanks for bringing this up. I am able to reproduce this issue locally and am working on a fix now. There's a chance that the fix may not go into 7.6.0 (but we might be able to get it into 7.6.1).

[ycombinator]

@dedemorton The PR to fix the issue is up: #15624.

[dedemorton]

@ycombinator This is ready for review. I wish I had more time to add realistic examples, but this at least gets the basic content in place with simple examples. It would be really nice to add an extended example that shows all the bits (log file, filebeat configs, LS configs) and tells a story. Since the processors were broken until recently, I didn't have time to come up with something better. Maybe we can add the blog later?

[urso]

@dedemorton Elasticsearch input also defaults to @metadata._id. We are switching to @metadata._id and add missing support for the document_id setting in the decode_json_fields processor: #15859

[dedemorton]

@ycombinator I've addressed your feedback and updated the field name to _id.

I think I should also add an example of decode_json_fields now that you can use it to set the id. WDYT?

[ycombinator]

I think I should also add an example of decode_json_fields now that you can use it to set the id. WDYT?

Agreed.

Otherwise, LGTM.

[dedemorton]

@ycombinator Please take a look at the latest changes. I've added a section for the decode_json_fields processor, but could not test it yet because the document_id setting does not seem to be valid in the latest build candidate (BC4).

[ycombinator]

LGTM.