[Agent] Exposed stream.type, stream.dataset and stream.namespace to every events.

[ph]

Some modules define like Suricata a single input where mixed events are generated: metrics, logs or alerts. The identification of the data is often done in the ingest pipeline, where the type is generated from fields or values from the data.

The way the agent operates it assumes unique type of data would be generated from input. This is normally true, but there is an exception like the Suricata system. To allow the maximun flexibility we should add fields to the event to allow the target index to be generated inside an ingest pipeline.

To do so we want to add the following fields to each event.

• stream.dataset
• stream.type
• stream.namespace

The values for stream.type and stream.namespace are inherited from the datasource and the input definition.

So see the following examples for generation rules:

datasources: 
   - id: nginx-x1 
     namespace: prod 
     inputs: 
       - type: logs 
         streams: 
           - id: {id} 
             enabled?: true # default to true 
             dataset: nginx.acccess 
             paths: /var/log/nginx/access.log 

Will generate an event with the following values:

fields:
 type: logs
 namespace: prod
 dataset: nginx.access

[elasticmachine]

Pinging @elastic/ingest (Project:fleet)

[ph]

@ruflin I see two issues here:

1. For adding fields on the event do we want to use the add_fields processors or fields defined in the input? I suppose we want to formalize one way to do it and maybe it's a good time to do so?

2. Are we allowing a user to define the type, namespace at the stream level? If so we need to change the index generation code.

[ph]

Maybe this could be done on the Fleet side? I worry adding magic fields or processors especially since they could impact other user-defined processors, because ordering is important and having values modified without you knowing is not nice?

[ruflin]

I think fields should be deprecated in favor or add_fields. The problem about doing it on the Fleet side is if someone sets up the agent manually, the fields will not be there. My thinking here is that adding this fields should be done by each Beat and not rely on the agent. This keeps the agent simple and adding it to the existing modules should be pretty straight forward. This might also allow an easier migration path from Beats to Agent in the future.

Namespace: Lets not allow the user to modify the namespace at the stream level to keep things simple.

There is an other reason having this fields in each even is important. Lets assume we support LS output in the future an all events are sent through LS. I expect LS to use these fields to make the right decision on where to send the data. (@jsvd FYI)

[ph]

I think fields should be deprecated in favor or add_fields. The problem about doing it on the Fleet side is if someone sets up the agent manually, the fields will not be there. My thinking here is that adding this fields should be done by each Beat and not rely on the agent. This keeps the agent simple and adding it to the existing modules should be pretty straight forward. This might also allow an easier migration path from Beats to Agent in the future.

To have that logic in beats this would mean that we add type, namespace and dataset as configuration field in the input and the input magically use add_fields?

[ruflin]

Yes. If add_fields is used or hardcoded does not matter in the end.

[elasticmachine]

Pinging @elastic/ingest-management (Team:ingest-management)

[ph]

@michalpristas @ruflin I think this has felt into a crack. Michal can you take a look?