pfsense - pfblockerng logs to pfelk - working solution

[robeweber]

Hi,
First let me tell you how happy I am to have found this repo and product.
Nice work and so detailed. I am impressed with your tutorials and guides. Realy great work and I know how much work that is.

I have for a long time wanted to get logs from pfblockerng into a dashboard. I have used pfsense dashboard project from Victor Robellini for a while now. He is using telegraf to write log entries to influx and then showing them in a rather huge grafana dashboard.

His solution does not work for the IP Block logs for a few years now. So I had to look further.

I found a discussion thread that is now closed which talks about integrating pfblockerng. The discussion stops with "started a new project". However I can not find this project. So I thought, I would dive into it and share back here. Not sure if this is the right discussion category. Let me know and I move it to wherever it is needed.

I had success with my dev skills and wanted to share this for anyone still looking for a solution.

Yesterday, I have started from scratch. Installing an Ubuntu 22.04.2 LTS and running the install script from your git.
My pfSense version is 2.7.0-RELEASE (amd64)
pfSense running on Protecli.com vp2410

I have the syslog-ng plugin and made two new Source/Log/Destination entries for the two log files I was interested in and sending them to two different ports: 5055 and 5056

for the ip_block.log:

{
  wildcard-file(
    base-dir("/var/log/pfblockerng")
    filename-pattern("ip_block.log")
    recursive(yes)
    follow-freq(1)
    program-override("pfblocker-ip")
    flags(no-parse)
  );
};

{
  source(pfblocker-ip);
  destination(pfblocker-ip);
};

{
   tcp("YOURPFELKHOST"
   port(5055)
   failover( servers("YOURPFELKIP") )
   );
};

and for the dnsbl.log:

{
  wildcard-file(
    base-dir("/var/log/pfblockerng")
    filename-pattern("dnsbl.log")
    recursive(yes)
    follow-freq(1)
    program-override("pfblocker-dnsbl")
    flags(no-parse)
  );
};

{
  source(pfblocker-dnsbl);
  destination(pfblocker-dnsbl);
};

{
   tcp("YOURPFELKHOST"
   port(5056)
   failover( servers("YOURPFELKIP") )
   );
};

---

The rest happens in the PFELK setup:
On Pfelk I have added a new input pfelk file: 02-input-pfblockerng.pfelk

input {
  ## pfblocker-ip ##
  syslog {
    id => "pfelk-pfblocker-ip-0001"
    type => "firewall"
    port => 5055
    syslog_field => "message"
    ecs_compatibility => v1
    grok_pattern => "<%{POSINT:[log][syslog][priority]}>%{GREEDYDATA:pfelk}"
    tags => ["pfelk","pfblockerip"]
  }
  syslog {
    id => "pfelk-pfblocker-dnsbl-0001"
    type => "firewall"
    port => 5056
    syslog_field => "message"
    ecs_compatibility => v1
    grok_pattern => "<%{POSINT:[log][syslog][priority]}>%{GREEDYDATA:pfelk}"
    tags => ["pfelk","pfblockerdnsbl"]
  }
}
# 
filter {
  grok {
    patterns_dir => [ "/etc/pfelk/patterns" ]
    match => [ "pfelk", "%{PFBLOCKERNG}" ]
  }
#### RFC 5424 Date/Time Format ####
  date {
    match => [ "[event][created]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ]
    target => "[event][created]"
  }
}

And in the pattern pfelk.grok I have added the following lines:

# pfblockerng
PFBLOCKERNG (%{PFBLOCKER_IP}|%{PFBLOCKER_DNSBL})
PFBLOCKER_IP %{SYSLOGTIMESTAMP:[event][created]}\s%{SYSLOGHOST:[log][syslog][hostname]}\s%{PROG:[log][syslog][appname]}\s%{MONTH}\s%{MONTHDAY}\s%{HOUR}:%{MINUTE}:%{INT},%{GREEDYDATA:filter_message}\\n
PFBLOCKER_DNSBL %{SYSLOGTIMESTAMP:[event][created]}\s%{SYSLOGHOST:[log][syslog][hostname]}\s%{PROG:[log][syslog][appname]}\s%{NOTSPACE:[observer][name]},%{MONTH}\s%{MONTHDAY}\s%{HOUR}:%{MINUTE}:%{INT},%{GREEDYDATA:filter_message}\\n
IPBLOCK %{NUMBER:[event][id]},%{WORD:[observer][interface][id]},%{WORD:[observer][interface][name]},%{WORD:[event][action]},%{INT:[observer][communication][ipv]},%{INT:[observer][communication][protocol]},%{DATA:[network][transport]},%{IPV4:[source][ip]},%{IPV4:[destination][ip]},%{INT:[source][port]},%{INT:[destination][port]},%{WORD:[network][direction]},%{WORD:[source][geo][country_iso_code]},%{WORD:[event][dataset]},(?<[source][cidr]>%{INT}.%{INT}.%{INT}.%{INT}/%{INT}|%{IPV4}),%{WORD:[event][category]},%{URIHOST:[source][domain]},%{HOSTNAME:[host][hostname]},%{WORD:[source][asn]},%{GREEDYDATA:[event][duplicationindicator]}
DNSBL %{IPORHOST:[destination][address]},%{IP:[client][ip]},%{DATA:[client][useragent]},%{WORD:[event][module]},%{WORD:[event][category]},%{IPORHOST:[dns][question][name]},%{DATA:[event][dataset]},%{GREEDYDATA:[event][duplicationindicator]}

in the 05-apps.pfelk configuration, I have added the following two new sections:

  ### pfblockerng - ip_block ###
  if "pfblockerip" in [tags] {
    mutate {
      add_tag => "pfblockerng"
      add_tag => "ipblock"
      remove_tag => "pfblockerip"
    }
    grok {
      patterns_dir => [ "/etc/pfelk/patterns" ]
      match => [ "filter_message", "%{IPBLOCK}" ]
    }
  }
  ### pfblockerng - dnsbl ###
  if "pfblockerdnsbl" in [tags] {
    mutate {
      add_tag => "pfblockerng"
      add_tag => "dnsbl"
      remove_tag => "pfblockerdnsbl"
    }
    grok {
      patterns_dir => [ "/etc/pfelk/patterns" ]
      match => [ "filter_message", "%{DNSBL}" ]
    }
  }

I am using the 45-enhanced_private.pfelk config and had to amend line 10 from:
if "dhcp" in [tags] or "unbound" in [tags] or "squid" in [tags] {

to:
if "dhcp" in [tags] or "unbound" in [tags] or "squid" in [tags] or "pfblockerng" in [tags] {

Finally, in 50-outputs.pfelk I have added two new datastreams just before the "catch all else" else -> blabla "unknown" bucket clause:

  } else if [log][syslog][appname] == "pfblocker-ip" {
    mutate { add_field => { "[data_stream][namespace]" => "ipblock" } }
  } else if [log][syslog][appname] == "pfblocker-dnsbl" {
    mutate { add_field => { "[data_stream][namespace]" => "dnsbl" } }

This now creates two new data streams in elastic search that you can create data views on and eventually dashboards for visualization.

I thought I would share this here and start a support or thinktank discussion for changes or improvements.

What do you think guys?

I have now uploaded all files as I am using them at this moment.
https://github.com/robeweber/pfelk/tree/main
Only download and overwrite them in your setup if you have not changed anything else yourselves in them and only if you follow all the steps of (#486) and #487

[robeweber]

I forgot to mention:
Yes, I eventually want to make an index template for these data streams, so that I can also use a data retention policy.
And yes, I don't have a dashboard yet.
And yes, some of the fields are not correctly mapped to the intended fields from the original author, but they show in the discover view and that counts already.

And please, before you change any pfelk settings and files, make a backup copy of the original working state so you can revert back to it in case you miss a { or ( or any other typo. Believe me, I have learned that the hard way.

[robeweber]

To Implement:

One thing that https://github.com/VictorRobellini/pfSense-Dashboard has, that I absolutely also want in my kibana dashboard:
Output from his "telegraf_pfifgw.php" script. Just have not figured out how to do this yet.

[robeweber]

updated the GROK pattern to accommodate the \n at the end of every message and changed the second timestamp, which does not match any other available pattern. This because the first item after the timestamp appears to be microseconds but is in fact the first value of the log, the event.id.
_grokparsefailure, _dateparsefailure is still in the tags, but the Grok Debugger does not give any warnings or issues. No idea why.

[robeweber]

While tinkering with the php script originally designed by VictorRobellini, I saw that the GROK patterns above needed tweaking. I have updated the post to be the patterns as they run now.
I have successfully integrated the Interface and Gateway PHP script to run on pfSense as a cronjob every minute. This script is not the original script anylonger as the influx integration does not work with logstash very well. I will post a separate show-and-tell discussion with the actual code how to get regular updates in pfELK of the interfaces and gateways from the pfSense.

[robeweber]

I have created a dashboard for the pfBlockerNG streams ipblock and dnsbl.
If you have exactly implemented what I wrote above (incl naming the destination in syslog-ng the exact same way) it should work for you as well: pfBlockerNG Kibana Dashboard

[flightnut]

I was unable to import this as I was getting a server error but so far the other changes are working.

[a3ilson]

@robeweber
Thank you!!!

• I should look at the discussion tab more often (don't get notifications - no excuse). However, thank you for providing instructions on how to replicate and configure! I can reference your repo, incorporate and/or add the instructions to the wiki - let me know how you'd prefer to link your pfBlockerNG configuration/dashboard etc.

Thanks again for enhancing this project!!!

[a3ilson]

@robeweber
Feel free to submit a PR and/or I can update your pfBlockerNG enhancements.

[a3ilson]

#247, #201, and #267

[Stizzy-98]

I was just thinking about this the other day after wanting to visualize those logs thank you for writing this up and taking the time to create this!