[Fleet] The log level feature in UI is showing logs for a given libbeat line and we don't see it associated correctly with the log level

[EricDavisX]

We see a specific log line that seems to only be shown when no-specific log level is selected in the UI. It should be categorized and shown as being under one of the ones listed right? Unless we have a Kibana side bug that the log level isn't in the listing (that would make sense actually, maybe its a Kibana bug).

This is as tested with an 8.0 self managed stack snapshot setup and 8.0 latest Agent on Linux Debian. I am testing on our demo server: kibana.endpoint.elastic.dev/app/fleet#/fleet/agents/

see video for concern:
what-level-is-this-log-line-at.mov.zip

PH notes it should be at Info level, per libbeat code here:
https://github.com/elastic/beats/blob/master/libbeat/monitoring/report/log/log.go#L145

I took another video showing off that there are logs, but that when selected NONE of the available levels show any logs
no-log-levels-matching.zip

screenshots:

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[nchaulet]

so looks like logs logged by the agent are logged with the field log.level and logs logged by metricbeat or filebeat are logged with the field level
Should we change that so we can use the same field for both?

[ph]

It should be log.level this is the ECS field that need to be used see reference in https://www.elastic.co/guide/en/ecs/current/ecs-log.html#field-log-level

I will reassign it to @michalpristas, didn't we turn off the ecs log format for the running process?

As a side node @ferullo is your endpoint logging ECS compliant?

[ferullo]

Endpoint logs are ECS compliant. If you find they aren't that's a bug in Endpoint.

[blakerouse]

I think what you are seeing here is that because all beats are started at trace level, that means that every event is logged to the file with the log.level: debug. Then its picked up again by the monitoring filebeat, so every event is in the log twice.

#19179 documents the real issue, this seems to be a side-effect.

[ph]

Ok, I though we change the default of beats to info this is not the case?

[blakerouse]

Maybe we did? If so should close the #19179 issue.

@michalpristas ^ can you confirm? Thought beats were still running at a higher level.

[michalpristas]

yep we changed it to info some time ago when making log level reloadable

[michalpristas]

@ph we did turn of ECS because there was a version reported twice with different ecs versions. one coming from beat another one from library #21616

[ph]

@michalpristas Maybe the fix was the incorrect one and we should just align the version?

[michalpristas]

even if versions are aligned field is reported twice, which is not nice. i will drop one of the fields added by lib and turn on ecs back

[michalpristas]

@EricDavisX this got closed with the PR please reopen if you see this behavior again, i checked events and they have correct log level field now

[dikshachauhan-qasource]

Hi @EricDavisX

While performing exploratory testing on 8.0.0 kibana cloud build today, we have observed that no-specific level logs get generated at logs UI on specific search.

Pre-condition:

1. Agent should be deplayed to 8.0.0 Kibana cloud environment. Build details are as follows:

BUILD 38728
COMMIT f2d961df6a5f78807dc2ef265aaba3c96d691881
Artifact link: https://snapshots.elastic.co/8.0.0-5823f363/downloads/beats/elastic-agent/elastic-agent-8.0.0-SNAPSHOT-windows-x86_64.zip

Steps to reproduce:

1. Login to Kibana cloud environment.
2. Go to Agents tab. Observe, logs are generated successfully.
3. Click on Agent logging level and select option 'error'.
4. Apply changes. And observe confirmation messgae is displayed.
5. Refresh the browser. Observe, changes are updated on agent details page at logging level field.
6. Navigate back to Logs UI. Select 'elastic-agent.filebeat' filter at 'Dataset'
7. Observe, no-specific level logs are displayed.

As per our understanding, whenever we change agent logging level, no-specific level logs are generated at backend.

Screenshot:

Hence, we are reopening this issue.

[EricDavisX]

I don't know if the last research @dikshachauhan-qasource posted is a separate issue, I will need to review and play with the feature more. I am seeing the specific original concern is still happening, so I am glad it is re-opened. I will push some convo to slack to see if we can hash out what remains to be done. @ph @nchaulet @michalpristas

[michalpristas]

i need to take a look but these sshould be info level so with this settings they should not be there.
also checked commit used in agent deba31 this commit is 3 commits before fix got in.

[EricDavisX]

I'm testing in on 8.0 latest a few days later and I am not seeing the original issue, so I would like to close this out, it looks like the latest changes helped fix this up. @dikshachauhan-qasource can you confirm if you are still seeing the problem you reported, and if so, let us open a new ticket please with all specifics and a clean history? And then we can close this out.

[dikshachauhan-qasource]

Hi @EricDavisX

Thanks for the feedback.

We have noticed same behavior on 8.0 latest Kibana cloud build. Hence, reported new defect #23145 for same.

Please let us know if anything is required from our side.

[EricDavisX]

ok, thanks. closing this out then