Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some Juniper IPFIX flows are not parsed correctly #180

Open
cvicente opened this issue Jul 15, 2019 · 4 comments
Open

Some Juniper IPFIX flows are not parsed correctly #180

cvicente opened this issue Jul 15, 2019 · 4 comments
Assignees

Comments

@cvicente
Copy link

  • Logstash version: 6.5.1
  • Codec Version: 4.2.1
  • Operating System: Ubuntu 14.04
  • Exporter device: Juniper MX480 running JUNOS 17.3R3-S3.3
  • Config File (if you have sensitive info, please remove it):

The configs are rather long. These are relevant sections:

input {
    udp {
        host => "192.0.2.1"
        port => 9995
	workers => 1
        codec => netflow {
            versions => [10]
            target => "ipfix"
        }
        type => ipfix
    	add_field => {
            "site" => "xyz"
	}
    }
}

    # Multiply by sampling interval to calculate the total bytes and packets the flow represents
    ruby {
	code => "event.set('[ipfix][octetDeltaCount]', event.get('[ipfix][octetDeltaCount]') * 1000)"
    }
    ruby {
	code => "event.set('[ipfix][packetDeltaCount]', event.get('[ipfix][packetDeltaCount]') * 1000)"
    }

    # add a bits field
    ruby {
	code => "event.set('[ipfix][bits]', event.get('[ipfix][octetDeltaCount]') * 8)"
    }

    # add fields to correctly calculate packets/sec, bits/sec
    ruby {
	code => "event.set('[ipfix][bits_1_60]', event.get('[ipfix][bits]') / 60)"
    }
    ruby {
	code => "event.set('[ipfix][packets_1_60]', event.get('[ipfix][packetDeltaCount]') / 60)"
    }
  • Sample Data:

About 0.4% of flows saved to a file appear incomplete and look like the following:

{"host":"192.0.2.2","@version":"1","tags":["IPv6","_rubyexception"],"hostname":"router.example.net","site":"xyz","token":"xxx","ipfix":{"flowIdleTimeout":60,"systemInitTimeMilliseconds":1552313129000,"exporterIPv4Address":"192.0.2.2","version":10,"exporterIPv6Address":"::","samplingInterval":1000,"flowActiveTimeout":60,"exportProtocolVersion":10,"exportedMessageTotalCount":3111510,"exportingProcessId":2,"exportTransportProtocol":17,"exportedFlowRecordTotalCount":4830582},"type":"ipfix","@timestamp":"2019-07-11T18:25:03.000Z"}
@s1sfa
Copy link

s1sfa commented Jul 18, 2019

I've just ignored these. This looks like the periodic update that the router will send with stats about the netflow setup, like samplingInterval:1000. It would be interesting if the plugin could somehow use this to map sampling interval so we wouldn't need to add config like:
code => "event.set('[ipfix][octetDeltaCount]', event.get('[ipfix][octetDeltaCount]') * 1000)"

@cvicente
Copy link
Author

I should add that I noticed this while troubleshooting my installs. I am observing a significant mismatch between the packets per second calculated using netflow data compared to what's reported by SNMP. I see no evidence of UDP errors, so all the input messages seem to be processed.

Interestingly, I have older collectors using logstash 2.3.4 and logstash-codec-netflow-2.1.1 where there is no pps mismatch.

@colinsurprenant
Copy link
Contributor

In your above example, the event is tagged with _rubyexception which means that one of the ruby code section produced an exception. You probably want to verify, using conditionals in your filter section, that an event received is valid before applying ruby code to it.

@colinsurprenant
Copy link
Contributor

It is quite hard to tell why you have this mismatch between your SNMP metrics and this netflow data. You could try to enable Persistent Queue to prevent possible back pressure which could result in dropped UDP packets. Monitoring the PQ will tell you if there are indication of back pressure situations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants