writing to influxdb with batches

[maksadbek]

Required for all PRs:

• CHANGELOG.md updated
• Sign CLA (if not already signed)
• README.md updated (if adding a new plugin)

[sparrc]

can you explain what this is doing and why it's necessary?

[sebito91]

This parameter gives the option to break up the number of metrics sent per message from nstat to influxdb. In the case without the param, the nstat plugin grabs every...single...stat from /proc/netstat which results in about 75+ fields sent per measurement.

Although we can tweak the size of the UDP payload being sent/received, this is an insanely large number of metrics to send per message. This feature request allows us to split the fields into multiple messages (and as an optional param only) so we can send, say 20 fields per measurement AND stay within the existing UDP payload size

BEFORE
[root@fpia-compute0006 ~]# telegraf -test -input-filter nstat
2016/06/03 11:13:27 Using config file: /etc/telegraf/telegraf.conf
* Plugin: nstat, Collection 1
> nstat,bu=testing,cls=server,dc=fpia,env=grid,host=fpia-compute0006,name=netstat,sr=grid_compute,trd=false IpExtInBcastOctets=955177988i,IpExtInBcastPkts=2757485i,IpExtInCEPkts=0i,IpExtInCsumErrors=0i,IpExtInECT0Pkts=0i,IpExtInECT1Pkts=0i,IpExtInMcastOctets=352590093276i,IpExtInMcastPkts=4579162470i,IpExtInNoECTPkts=4600477188i,IpExtInNoRoutes=2i,IpExtInOctets=358603630526i,IpExtInTruncatedPkts=0i,IpExtOutBcastOctets=0i,IpExtOutBcastPkts=0i,IpExtOutMcastOctets=145028112i,IpExtOutMcastPkts=2019374i,IpExtOutOctets=17868838460i,TcpExtArpFilter=0i,TcpExtBusyPollRxPackets=0i,TcpExtDelayedACKLocked=14i,TcpExtDelayedACKLost=244i,TcpExtDelayedACKs=1130369i,TcpExtEmbryonicRsts=0i,TcpExtIPReversePathFilter=0i,TcpExtListenDrops=0i,TcpExtListenOverflows=0i,TcpExtLockDroppedIcmps=0i,TcpExtOfoPruned=0i,TcpExtOutOfWindowIcmps=0i,TcpExtPAWSActive=0i,TcpExtPAWSEstab=0i,TcpExtPAWSPassive=0i,TcpExtPruneCalled=0i,TcpExtRcvPruned=0i,TcpExtSyncookiesFailed=101i,TcpExtSyncookiesRecv=0i,TcpExtSyncookiesSent=0i,TcpExtTCPACKSkippedChallenge=0i,TcpExtTCPACKSkippedFinWait2=0i,TcpExtTCPACKSkippedPAWS=0i,TcpExtTCPACKSkippedSeq=0i,TcpExtTCPACKSkippedSynRecv=0i,TcpExtTCPACKSkippedTimeWait=0i,TcpExtTCPAbortFailed=0i,TcpExtTCPAbortOnClose=4i,TcpExtTCPAbortOnData=65717i,TcpExtTCPAbortOnLinger=0i,TcpExtTCPAbortOnMemory=0i,TcpExtTCPAbortOnTimeout=30i,TcpExtTCPAutoCorking=600176i,TcpExtTCPBacklogDrop=0i,TcpExtTCPChallengeACK=58i,TcpExtTCPDSACKIgnoredNoUndo=620i,TcpExtTCPDSACKIgnoredOld=0i,TcpExtTCPDSACKOfoRecv=0i,TcpExtTCPDSACKOfoSent=0i,TcpExtTCPDSACKOldSent=244i,TcpExtTCPDSACKRecv=4509i,TcpExtTCPDSACKUndo=398i,TcpExtTCPDeferAcceptDrop=0i,TcpExtTCPDirectCopyFromBacklog=1448i,TcpExtTCPDirectCopyFromPrequeue=4161525i,TcpExtTCPFACKReorder=0i,TcpExtTCPFastOpenActive=0i,TcpExtTCPFastOpenActiveFail=0i,TcpExtTCPFastOpenCookieReqd=0i,TcpExtTCPFastOpenListenOverflow=0i,TcpExtTCPFastOpenPassive=0i,TcpExtTCPFastOpenPassiveFail=0i,TcpExtTCPFastRetrans=1856i,TcpExtTCPForwardRetrans=205i,TcpExtTCPFromZeroWindowAdv=1i,TcpExtTCPFullUndo=16i,TcpExtTCPHPAcks=8317552i,TcpExtTCPHPHits=4240421i,TcpExtTCPHPHitsToUser=461i,TcpExtTCPHystartDelayCwnd=24i,TcpExtTCPHystartDelayDetect=1i,TcpExtTCPHystartTrainCwnd=54871i,TcpExtTCPHystartTrainDetect=229i,TcpExtTCPLossFailures=0i,TcpExtTCPLossProbeRecovery=4586i,TcpExtTCPLossProbes=9731i,TcpExtTCPLossUndo=118i,TcpExtTCPLostRetransmit=86i,TcpExtTCPMD5NotFound=0i,TcpExtTCPMD5Unexpected=0i,TcpExtTCPMemoryPressures=0i,TcpExtTCPMinTTLDrop=0i,TcpExtTCPOFODrop=0i,TcpExtTCPOFOMerge=0i,TcpExtTCPOFOQueue=1251i,TcpExtTCPOrigDataSent=14517154i,TcpExtTCPPartialUndo=147i,TcpExtTCPPrequeueDropped=0i,TcpExtTCPPrequeued=1640i,TcpExtTCPPureAcks=1224155i,TcpExtTCPRcvCoalesce=396656i,TcpExtTCPRcvCollapsed=0i,TcpExtTCPRenoFailures=0i,TcpExtTCPRenoRecovery=0i,TcpExtTCPRenoRecoveryFail=0i,TcpExtTCPRenoReorder=0i,TcpExtTCPReqQFullDoCookies=0i,TcpExtTCPReqQFullDrop=0i,TcpExtTCPRetransFail=0i,TcpExtTCPSACKDiscard=0i,TcpExtTCPSACKReneging=0i,TcpExtTCPSACKReorder=5i,TcpExtTCPSYNChallenge=58i,TcpExtTCPSackFailures=62i,TcpExtTCPSackMerged=450i,TcpExtTCPSackRecovery=1511i,TcpExtTCPSackRecoveryFail=131i,TcpExtTCPSackShiftFallback=8754i,TcpExtTCPSackShifted=510i,TcpExtTCPSchedulerFailed=0i,TcpExtTCPSlowStartRetrans=155i,TcpExtTCPSpuriousRTOs=51i,TcpExtTCPSpuriousRtxHostQueues=0i,TcpExtTCPSynRetrans=371i,TcpExtTCPTSReorder=148i,TcpExtTCPTimeWaitOverflow=0i,TcpExtTCPTimeouts=149i,TcpExtTCPToZeroWindowAdv=1i,TcpExtTCPWantZeroWindowAdv=1i,TcpExtTW=155492i,TcpExtTWKilled=0i,TcpExtTWRecycled=0i 1464970407152416596
> nstat,bu=testing,cls=server,dc=fpia,env=grid,host=fpia-compute0006,name=snmp,sr=grid_compute,trd=false IcmpInAddrMaskReps=0i,IcmpInAddrMasks=0i,IcmpInCsumErrors=0i,IcmpInDestUnreachs=1069711i,IcmpInEchoReps=96i,IcmpInEchos=88487i,IcmpInErrors=192062i,IcmpInMsgs=1158294i,IcmpInParmProbs=0i,IcmpInRedirects=0i,IcmpInSrcQuenchs=0i,IcmpInTimeExcds=0i,IcmpInTimestampReps=0i,IcmpInTimestamps=0i,IcmpMsgInType0=96i,IcmpMsgInType3=1069711i,IcmpMsgInType8=88487i,IcmpMsgOutType0=88487i,IcmpMsgOutType3=331i,IcmpMsgOutType8=96i,IcmpOutAddrMaskReps=0i,IcmpOutAddrMasks=0i,IcmpOutDestUnreachs=331i,IcmpOutEchoReps=88487i,IcmpOutEchos=96i,IcmpOutErrors=0i,IcmpOutMsgs=88914i,IcmpOutParmProbs=0i,IcmpOutRedirects=0i,IcmpOutSrcQuenchs=0i,IcmpOutTimeExcds=0i,IcmpOutTimestampReps=0i,IcmpOutTimestamps=0i,IpDefaultTTL=64i,IpForwDatagrams=0i,IpForwarding=2i,IpFragCreates=843840i,IpFragFails=0i,IpFragOKs=337536i,IpInAddrErrors=0i,IpInDelivers=4598461991i,IpInDiscards=0i,IpInHdrErrors=0i,IpInReceives=4598461998i,IpInUnknownProtos=0i,IpOutDiscards=489456i,IpOutNoRoutes=0i,IpOutRequests=28590757i,IpReasmFails=0i,IpReasmOKs=0i,IpReasmReqds=0i,IpReasmTimeout=0i,TcpActiveOpens=613575i,TcpAttemptFails=386323i,TcpCurrEstab=1280i,TcpEstabResets=81288i,TcpInCsumErrors=0i,TcpInErrs=58i,TcpInSegs=15214484i,TcpMaxConn=-1i,TcpOutRsts=451736i,TcpOutSegs=21247146i,TcpPassiveOpens=100156i,TcpRetransSegs=8065i,TcpRtoAlgorithm=1i,TcpRtoMax=120000i,TcpRtoMin=200i,UdpInCsumErrors=0i,UdpInDatagrams=4579313100i,UdpInErrors=2752352i,UdpLiteInCsumErrors=0i,UdpLiteInDatagrams=0i,UdpLiteInErrors=0i,UdpLiteNoPorts=0i,UdpLiteOutDatagrams=0i,UdpLiteRcvbufErrors=0i,UdpLiteSndbufErrors=0i,UdpNoPorts=240i,UdpOutDatagrams=13493588i,UdpRcvbufErrors=0i,UdpSndbufErrors=487575i 1464970407152853214

AFTER
[root@fpia-compute0006 ~]# telegraf -test -input-filter nstat
2016/06/03 11:13:54 Using config file: /etc/telegraf/telegraf.conf
* Plugin: nstat, Collection 1
> nstat,bu=testing,cls=server,dc=fpia,env=grid,host=fpia-compute0006,name=netstat,sr=grid_compute,trd=false IpExtInCEPkts=0i,IpExtInECT1Pkts=0i,IpExtInOctets=358612130653i,IpExtOutMcastOctets=145031744i,TcpExtPAWSActive=0i,TcpExtPAWSEstab=0i,TcpExtSyncookiesRecv=0i,TcpExtSyncookiesSent=0i,TcpExtTCPACKSkippedFinWait2=0i,TcpExtTCPAbortOnData=65721i,TcpExtTCPDirectCopyFromBacklog=1448i,TcpExtTCPFastOpenCookieReqd=0i,TcpExtTCPFullUndo=16i,TcpExtTCPHystartDelayDetect=1i,TcpExtTCPMemoryPressures=0i,TcpExtTCPOFODrop=0i,TcpExtTCPOFOMerge=0i,TcpExtTCPPrequeued=1640i,TcpExtTCPRenoFailures=0i,TcpExtTCPRetransFail=0i,TcpExtTCPSackMerged=450i,TcpExtTCPSackShifted=510i,TcpExtTCPSlowStartRetrans=155i,TcpExtTCPTSReorder=148i,TcpExtTCPToZeroWindowAdv=1i 1464970434584866579
> nstat,bu=testing,cls=server,dc=fpia,env=grid,host=fpia-compute0006,name=netstat,sr=grid_compute,trd=false IpExtInNoRoutes=2i,TcpExtArpFilter=0i,TcpExtDelayedACKLost=244i,TcpExtIPReversePathFilter=0i,TcpExtLockDroppedIcmps=0i,TcpExtOutOfWindowIcmps=0i,TcpExtRcvPruned=0i,TcpExtSyncookiesFailed=101i,TcpExtTCPAutoCorking=600198i,TcpExtTCPDSACKOfoRecv=0i,TcpExtTCPDSACKRecv=4509i,TcpExtTCPDSACKUndo=398i,TcpExtTCPDirectCopyFromPrequeue=4161525i,TcpExtTCPFastOpenActive=0i,TcpExtTCPForwardRetrans=205i,TcpExtTCPHPHits=4240522i,TcpExtTCPOrigDataSent=14517362i,TcpExtTCPPureAcks=1224198i,TcpExtTCPRenoRecoveryFail=0i,TcpExtTCPSACKReneging=0i,TcpExtTCPSACKReorder=5i,TcpExtTCPSackFailures=62i,TcpExtTCPSackRecoveryFail=131i,TcpExtTCPTimeouts=149i,TcpExtTWKilled=0i 1464970434584957145
> nstat,bu=testing,cls=server,dc=fpia,env=grid,host=fpia-compute0006,name=netstat,sr=grid_compute,trd=false IpExtInBcastOctets=955200212i,IpExtInBcastPkts=2757549i,IpExtInMcastPkts=4579271973i,IpExtInNoECTPkts=4600587128i,IpExtOutBcastPkts=0i,TcpExtListenDrops=0i,TcpExtPruneCalled=0i,TcpExtTCPACKSkippedSynRecv=0i,TcpExtTCPAbortFailed=0i,TcpExtTCPAbortOnTimeout=30i,TcpExtTCPBacklogDrop=0i,TcpExtTCPDSACKIgnoredOld=0i,TcpExtTCPDSACKOfoSent=0i,TcpExtTCPDSACKOldSent=244i,TcpExtTCPFastOpenPassiveFail=0i,TcpExtTCPHystartTrainDetect=229i,TcpExtTCPLossFailures=0i,TcpExtTCPLossUndo=118i,TcpExtTCPOFOQueue=1251i,TcpExtTCPPrequeueDropped=0i,TcpExtTCPRenoReorder=0i,TcpExtTCPReqQFullDoCookies=0i,TcpExtTCPSchedulerFailed=0i,TcpExtTCPSpuriousRtxHostQueues=0i,TcpExtTW=155497i 1464970434585022241
> nstat,bu=testing,cls=server,dc=fpia,env=grid,host=fpia-compute0006,name=netstat,sr=grid_compute,trd=false IpExtInTruncatedPkts=0i,IpExtOutBcastOctets=0i,TcpExtEmbryonicRsts=0i,TcpExtPAWSPassive=0i,TcpExtTCPACKSkippedSeq=0i,TcpExtTCPACKSkippedTimeWait=0i,TcpExtTCPAbortOnClose=4i,TcpExtTCPAbortOnLinger=0i,TcpExtTCPDSACKIgnoredNoUndo=620i,TcpExtTCPFACKReorder=0i,TcpExtTCPFastOpenActiveFail=0i,TcpExtTCPFastOpenListenOverflow=0i,TcpExtTCPHPAcks=8317682i,TcpExtTCPHPHitsToUser=461i,TcpExtTCPLossProbes=9731i,TcpExtTCPMD5NotFound=0i,TcpExtTCPMD5Unexpected=0i,TcpExtTCPPartialUndo=147i,TcpExtTCPRcvCoalesce=396664i,TcpExtTCPSYNChallenge=58i,TcpExtTCPSackRecovery=1511i,TcpExtTCPSpuriousRTOs=51i,TcpExtTCPSynRetrans=371i,TcpExtTCPWantZeroWindowAdv=1i,TcpExtTWRecycled=0i 1464970434585081600
> nstat,bu=testing,cls=server,dc=fpia,env=grid,host=fpia-compute0006,name=netstat,sr=grid_compute,trd=false IpExtInMcastOctets=352598524858i,IpExtOutMcastPkts=2019425i,IpExtOutOctets=17869090832i,TcpExtDelayedACKLocked=14i,TcpExtDelayedACKs=1130401i,TcpExtListenOverflows=0i,TcpExtOfoPruned=0i,TcpExtTCPACKSkippedChallenge=0i,TcpExtTCPACKSkippedPAWS=0i,TcpExtTCPAbortOnMemory=0i,TcpExtTCPChallengeACK=58i,TcpExtTCPDeferAcceptDrop=0i,TcpExtTCPFastOpenPassive=0i,TcpExtTCPFastRetrans=1856i,TcpExtTCPFromZeroWindowAdv=1i,TcpExtTCPHystartDelayCwnd=24i,TcpExtTCPLossProbeRecovery=4586i,TcpExtTCPLostRetransmit=86i,TcpExtTCPMinTTLDrop=0i,TcpExtTCPRcvCollapsed=0i,TcpExtTCPRenoRecovery=0i,TcpExtTCPReqQFullDrop=0i,TcpExtTCPSACKDiscard=0i,TcpExtTCPSackShiftFallback=8754i,TcpExtTCPTimeWaitOverflow=0i 1464970434585140099
> nstat,bu=testing,cls=server,dc=fpia,env=grid,host=fpia-compute0006,name=netstat,sr=grid_compute,trd=false IpExtInCsumErrors=0i,IpExtInECT0Pkts=0i,TcpExtBusyPollRxPackets=0i,TcpExtTCPHystartTrainCwnd=54871i 1464970434585187876
> nstat,bu=testing,cls=server,dc=fpia,env=grid,host=fpia-compute0006,name=snmp,sr=grid_compute,trd=false IcmpInCsumErrors=0i,IcmpInDestUnreachs=1069739i,IcmpInMsgs=1158326i,IcmpInParmProbs=0i,IcmpInSrcQuenchs=0i,IcmpInTimestamps=0i,IcmpMsgInType8=88491i,IcmpMsgOutType0=88491i,IcmpOutAddrMaskReps=0i,IcmpOutEchoReps=88491i,IcmpOutEchos=96i,IcmpOutMsgs=88918i,IcmpOutSrcQuenchs=0i,IcmpOutTimestampReps=0i,IpFragFails=0i,IpFragOKs=337548i,IpInDelivers=4598571930i,IpInHdrErrors=0i,IpInReceives=4598571938i,IpOutNoRoutes=0i,TcpEstabResets=81293i,TcpMaxConn=-1i,TcpPassiveOpens=100161i,TcpRtoMin=200i,UdpLiteInCsumErrors=0i 1464970434585453166
> nstat,bu=testing,cls=server,dc=fpia,env=grid,host=fpia-compute0006,name=snmp,sr=grid_compute,trd=false IcmpInAddrMaskReps=0i,IcmpInEchos=88491i,IcmpInErrors=192071i,IcmpInRedirects=0i,IcmpMsgInType3=1069739i,IcmpOutErrors=0i,IcmpOutRedirects=0i,IcmpOutTimeExcds=0i,IpDefaultTTL=64i,IpInAddrErrors=0i,IpOutRequests=28591489i,IpReasmTimeout=0i,TcpAttemptFails=386333i,TcpInErrs=58i,TcpInSegs=15214825i,TcpOutRsts=451749i,TcpRtoMax=120000i,UdpInCsumErrors=0i,UdpInDatagrams=4579422597i,UdpLiteInErrors=0i,UdpLiteRcvbufErrors=0i,UdpLiteSndbufErrors=0i,UdpNoPorts=240i,UdpOutDatagrams=13493947i,UdpSndbufErrors=487586i 1464970434585518448
> nstat,bu=testing,cls=server,dc=fpia,env=grid,host=fpia-compute0006,name=snmp,sr=grid_compute,trd=false IcmpInAddrMasks=0i,IcmpInEchoReps=96i,IcmpMsgInType0=96i,IcmpMsgOutType3=331i,IcmpMsgOutType8=96i,IcmpOutDestUnreachs=331i,IcmpOutParmProbs=0i,IcmpOutTimestamps=0i,IpForwDatagrams=0i,IpForwarding=2i,IpFragCreates=843870i,IpInUnknownProtos=0i,IpOutDiscards=489467i,IpReasmOKs=0i,IpReasmReqds=0i,TcpActiveOpens=613594i,TcpCurrEstab=1281i,TcpInCsumErrors=0i,TcpOutSegs=21247533i,TcpRtoAlgorithm=1i,UdpInErrors=2752416i,UdpLiteInDatagrams=0i,UdpLiteNoPorts=0i,UdpLiteOutDatagrams=0i,UdpRcvbufErrors=0i 1464970434585578005
> nstat,bu=testing,cls=server,dc=fpia,env=grid,host=fpia-compute0006,name=snmp,sr=grid_compute,trd=false IcmpInTimeExcds=0i,IcmpInTimestampReps=0i,IcmpOutAddrMasks=0i,IpInDiscards=0i,IpReasmFails=0i,TcpRetransSegs=8065i 1464970434585621550

[sparrc]

This seems like it should be an option for the influxdb output, rather than only for the nstat plugin.

@maksadbek can you look into making this change on the influxdb output plugin? You probably only want to do this if the user is writing via UDP.

[maksadbek]

@sparrc Okay, I will look at it.

[sebito91]

@sparrc, how would you recommend we do that? If any measurement has over x
fields we would use this split option?

[sparrc]

@sebito91 yes, that's what I had in mind, but doing this probably should be "off" by default. It's also probably less relevant when making http writes.

[sebito91]

I'll get you a commit shortly, this will absolutely be an optional setting.

[maksadbek]

@sparrc I moved the logic of breaking into batches to Influxdb output plugin.
It is configured with 2 variables in config file: batch_size & batch_limit.
Metrics are broken into multiple parts, and each part has batch_size of fields.
This feature turned off by default, and turned on by setting batch_size > 0 value.
It breakes metrics into multiple formats only when measurements has over batch_limit metrics.

BTW, the build actually failed because of another plugin:

FAIL    github.com/influxdata/telegraf/plugins/outputs/opentsdb 0.017s
--- FAIL: TestWrite (0.00s)
    Error Trace:    opentsdb_test.go:58
    Error:      Received unexpected error "OpenTSDB: Telnet connect fail"

[sparrc]

sorry about the inconvenience but please rebase and repush, thanks!

[sebito91]

Thank you @maksadbek @sparrc!!

[sebito91]

Any idea when we can merge this? Waiting for v1.0.0?

[sparrc]

aren't BatchSize and BatchLimit pretty much the same thing? Seems like the same functionality could be accomplished with just one of the two, unless I'm misunderstanding something.

[sparrc]

also the term "Batch" is already being used in telegraf to refer to a "batch" of points, so maybe call it something that explicitly indicates that it's only limiting the number of fields sent per metric, MetricFieldLimit?

[maksadbek]

@sparrc,
BatchSize is the number of metrics that each batch must include maximum. If there are 180 metrics in, and batch size is 30, then these metrics will be splitted into 6 batches that each contain 30 metrics.

BatchLimit is the limit when metrics must be splitted into multiple with smaller size, if it is 200, and there are 200 metrics, then 200 metrics are written at once, it will not be splitted into batches. If there are 250 metrics then, it will be splitted.

I can change this logic if it is required.

Yes, I agree with the name, MetricFieldLimit is better, relatively MetricFieldNum instead of BatchSize

[sparrc]

it seems to me that only BatchLimit is necessary for this use-case then. IMO we should simply have a maximum limit on how many fields can be in a single sent metric.

[maksadbek]

Okay, I will apply this and set BatchSize to 25.

[sparrc]

Looks good @maksadbek.

Apologies for not seeing this earlier, but I think this is actually a bug in the InfluxDB UDP client, isn't it? The UDP client should not be writing packets that are too large to write. You should be able to drop your code more or less in-place here: https://github.com/influxdata/influxdb/blob/master/client/v2/client.go#L414.

Doing this, you would remove the option of configuring MetricFieldLimit, and instead you could calculate how many packets the single packet should be split into based on how many times larger it is than the max packet size.

let me know what you think about implementing that, and then the Telegraf change would simply be updating the influxdb dependency commit.

[maksadbek]

OK, I will look up at InfluxDB client code and apply changes there.
I will get you implementation very soon.

[sparrc]

thanks @maksadbek, ping me when you have a PR for that on the influxdb repo.

[sparrc]

Closing as this will be a change to the influxdb client. Feel free to reopen if that's not the case.

[maksadbek]

@sparrc @sebito91 I've sent a PR to InfluxDB, influxdata/influxdb#6913