pfSense & syslog

[arisentow]

This is sort of an update to a previous post now that I've done some more digging...

I know that this has been addressed in other posts, and I've tried some of the recommendations/troubleshooting steps, however I am unable to get my pfSense logs parsed in kibana or SOC.

I've done:
1.) Configured remote logging in pfSense... BSD formatted, sending logs to my management interface on port 514
2.) Toggled 'Raw Logs/Show raw filter logs' in pfSense just to test if that was the issue
3.) Ran 'so-allow/syslog device' for my pfSense system and confirmed that it took with 'so-firewall includedhosts syslog'
4.) Confirmed that the logs are being received via Kibana
5.) Confirmed that 'filterlog' is present in '/opt/so/conf/elasticsearch/ingest/' and in the Kibana Stack Management -> Ingest Pipelines

Again, the logs are being received, however they are not being parsed and are being put into the syslog 'event.dataset' versus firewall

Are there any configurations that I've missed or additional troubleshooting steps I can take?

[dougburks]

What version of pfSense are you using?

What version of Security Onion are you using?

Have you checked Elasticsearch logs for any potential clues about parsing issues?

[dougburks]

If you select the SYSLOG dashboard, do you have logs there?

What do you get if you change the query to event.dataset:syslog | groupby event.module?

[arisentow]

Doug,

I do have logs in the dashboard and via that query... to include pfSense logs, just not parsed and they are in the syslog dataset. Some screen shots without the actual message not to reveal IP addresses.

'soc_source' is :so-syslog-2022.10.28

[arisentow]

See my reply to Ragnar below and let me know if you have any thoughts there.

[dougburks]

The third screenshot shows that the log is a dhcpd log so even though it may have come from the DHCP server on your pfSense firewall, it's not actually a firewall log. Do you have any examples of actual firewall logs? They should show filterlog in the message field.

Just to confirm, this is a production firewall that is processing traffic and you have the firewall rules configured to log traffic?

I don't see a response to my question above, so I'll ask again: What do you get if you change the query to event.dataset:syslog | groupby event.module?

[arisentow]

So, I feel like a dumb***. I grabbed a DHCP log (it was early, I was heading out the door for work). Regardless, I rebooted my pfSense as a last resort and now it is working. I threw in some firewall rules across different VLANS to test and am now seeing the events on the firewall dashboard.

Thank you for your time Doug. Feel free to close.

[RagnarSTS]

I know it has been quite a while, but I remember posting when pfsense 2.6 was still in beta about the syslog and the parsers. I don't think the firewall parser ever got updated. When pfsense sends the logs, in order to have them recognized properly they have to be formatted "BSD (RFC 3164, default)" ... and it might take an hour for the new index to be built after the system sees it for the first time.

[arisentow]

Thanks Ragnar. I do have pfSense configured to send the logs formatted as "BSD (RFC 3164, default" right now. I'm wondering if the issue stems from when pfSense would have first started sending logs they would have been in the RFC 5424 format because I had not switched to 3164 prior to shipping the logs when I first configured it. I had it configured as RFC 5424 initially because I had been sending my logs to a Gray Log server in that format. That is the only thing I can think of, otherwise I'm at a loss. Maybe Elasticsearch didn't catch on to the change in format and keeps placing them in the syslog dataset?