[Logs UI] Original log message doesn't show for events coming from cisco.asa and netflow.log

[afgomez]

This comes from a user report in our discuss.

From the user:

I've setup the Cisco ASA module in filebeat and all the ASA logs are coming OK to my ELK server on port UDP/514. I can see the ASA lohgs in Discover and SIEM Netwotk tab.

But when I go to Kibana --> Observavility --> Logs I see a lot of error messages:

We need to determine if filebeat is populating the right field in the schema or if the logs app is not processing the entry correctly.

[weltenwort]

There is not message field - mainly because there's no canonical string representation.

Related:

• [Logs UI] Pre-ECS netflow log messages are not displayed correctly #47557
• Filebeat modules: Always send message by default beats#14708

[afgomez]

I see. I'm trying to think what would make sense for a user. Let me know what you think:

For the netflow.log dataset, since there's no canonical representation (because there's not really a "log") I think it's better to filter them out when fetching log entries instead of showing something confusing. This will have two steps:

• Change the query to filter them out.
• Document in the netflow filebeat module that the entries will not appear in the logs UI (explaining why), but the data will still be available in Discover and Visualize.

Filtering those out will have implications in the Logs ML integration. In case we move forward with this idea we will need to decide what to do there.

For the cisco.asa dataset, the original log seems to be available in the event.original field. I think we can rebuild the message from that field for 7.10, until elastic/beats#14708 gets merged.

The potential downsides I see is that we are treating these two datasets as special snowflakes, but we do that already with nginx, kibana, etc.

Separately from this, if in filebeat 8.0.0 the default will be to ship the original log message, we need to decide if we will deprecate ourselves the reconstruction of the messages based on the individual fields. I personally think it makes sense to do so, documenting to the user that if they decide to not send the original message, the log will only appear in discover/visualize/etc.

Thoughts?

[weltenwort]

Change the query to filter them out.

That sounds easier than it is - mostly due to the message reconstruction heuristics applied for other modules. I'd love to have a conversation about the future of that aspect overall.

Filtering those out will have implications in the Logs ML integration. In case we move forward with this idea we will need to decide what to do there.

The log rate analysis job currently would include it, because it only counts documents and doesn't have a technical dependency on the message. The log categorization job already filters out documents that don't have a message field, because it's the basis of the categorization.

Document in the netflow filebeat module that the entries will not appear in the logs UI

Yes, in general it would be great if each module's documentation indicated what the indexed message will be.

[stephanbinder]

Are there any updates on this bug? We need to have the asa logs correctly showing up in Kibana (case was allready opened and led to this bug).

[afgomez]

@stephanbinder I don't think there has been any progress, but let me check that for you.

[drexciyaforever]

Hey team, any chance we've made progress on this? Wondering if the added support for event.original in Elastic Agent might be a solution!

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

[gbamparop]

@weltenwort I assume that in Logs Explorer we would display the JSON doc in the content column, do you think there's still work to be done from our side?

[weltenwort]

Ideally this would be solved via curated columns provided by these specific integrations, but the "content" column already mitigates much of this.

[gbamparop]

Closing as it's covered by the summary / document column in Discover and the content section in the log details flyout.