add ECS support and target option

[kaisecheng]

add target option

Fixed: #19

[kares]

🍑 I do like the proposed changes

one potential suggestion whether it makes sense to use event.original in ECS mode, that would represent the JSON "line" ... might be far fetched given that the original data is a multiline string that gets split-ed.

[kares]

I think we want to test with setting the field to v8:

Suggested change

	ecs_compatibility_matrix(:disabled, :v1, :v8 => :v1) do |ecs_select|
	ecs_compatibility_matrix(:disabled, :v1, :v8) do |ecs_select|

[kares]

would it make sense to event.set('[event][original]', bulk.get("message").dup.freeze) in ECS mode?

[kaisecheng]

There are four options here. Considering the following input

      { "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } }
      { "field1" : "value1" }
      { "delete" : { "_index" : "test", "_type" : "type1", "_id" : "2" } }
      { "create" : { "_index" : "test", "_type" : "type1", "_id" : "3" } }
      { "field1" : "value3" }
      { "update" : {"_id" : "1", "_type" : "type1", "_index" : "index1"} }
      { "doc" : {"field2" : "value2"} }

1. Store the whole input as event.original.
   The input generates four events. Each event saves the complete raw data.
   It could generate too much extra bandwidth

2. Store { "field1" : "value1" } as event.original.
   This is the line when target_event_factory is called. Metadata is not included. The information is not complete.
   This is the result of event.set('[event][original]', bulk.get("message").dup.freeze)
   The information is not particularly interesting because the same stuff transforms into an event.

3. tailor-make a new line to concatenate metadata { "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } } and data { "field1" : "value1" }
   This will be a full event that requires concatenating the decoded string

4. do not store event.original

I am opened to save it in event.original if you think it is useful

[kares]

oh right, it's tricky with the multi-line and the managed state ... was undecided on this making sense.
Reading your comments I am leaning towards 4. do not store event.original for now and ship as is...

[kares]

🍒