Incorrectly parses FormErr packet

[danielgriggs]

Hi,

Every few billion packets I find one that is incorrectly parsed. I would have thought by the next packet it would reset the state (or something) and correctly parse the next packet. But I have an example packet capture where this happens. Decoding the the packets with packetq shows,

1,1507765391,270720,2048,4175,53,"46.101.12.191","185.159.197.130",17,61,4,0,"pintael.nz.","",60087,39,0,0,0,0,0,4096,1,0,0,1,2,1,0,0,0,0,0,0,1,0,0,1,1,0,0,0,0,0,""
2,1507765391,271020,2048,20712,53,"91.209.84.9","185.159.197.130",17,118,4,0,"integro.co.nz.","",435,42,0,2,0,0,0,4000,1,0,0,1,15,1,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,""
3,1507765391,271070,2048,53,4175,"185.159.197.130","46.101.12.191",17,64,4,0,"pintael.nz.","",60087,1011,0,3,0,0,0,4096,1,0,8,1,2,1,0,0,0,1,0,0,1,0,0,1,1,1,0,0,0,0,""
4,1507765391,271103,2048,53,20712,"185.159.197.130","91.209.84.9",17,64,4,0,"pintael.nz.","",435,18,0,1,0,0,0,0,0,0,0,0,2,1,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,""

As you can see the forth packet as the incorrect domain in it, wireshark decodes the packet correctly.
PCAP sample attached.
incorrectly_parse_2.pcap.zip

[jelu]

Thanks for such a detailed report and a PCAP to test against, can you verify the fix from my branch:

git remote add jelu https://github.com/jelu/PacketQ.git
git fetch --all
git checkout dnsmsginit

[danielgriggs]

Hi Jerry,

I went through a couple of days of data and all of the changes in the results are positive and have fixed the errors.

Keep up the good work!

Daniel.

[danielgriggs]

Although it does appear to be quite a bit slower :(

daniel@test-m1:~$ /usr/bin/time packetq-jelu -s "select * from dns" ~/trace_test_2017-10-07_00\:43\:33.pcap > /dev/null
2.78user 0.23system 0:03.01elapsed 99%CPU (0avgtext+0avgdata 173360maxresident)k
0inputs+0outputs (0major+42801minor)pagefaults 0swaps
daniel@test-m1:~$ /usr/bin/time packetq -s "select * from dns" ~/trace_test_2017-10-07_00\:43\:33.pcap > /dev/null
1.68user 0.12system 0:01.80elapsed 100%CPU (0avgtext+0avgdata 173252maxresident)k
0inputs+0outputs (0major+42794minor)pagefaults 0swaps

[jelu]

That would be the price, yeah. Let me tinker with this in a few weeks too see if I can solve it in another way.

[jelu]

@danielgriggs Please rerun your processing tests :)

[danielgriggs]

Hi @jelu

Thats looking great,

Before patching;
real 2m40.713s
user 2m24.384s
sys 0m9.080s

Just with the behaviour correction;
real 3m5.727s
user 2m56.852s
sys 0m8.516s

Latest revision of that branch;
real 2m32.347s
user 2m23.112s
sys 0m8.360s

With both of the patched versions having the correct behaviour.
I'll run it over some larger datasets, but I think this is it :)

[danielgriggs]

Yep, looks all good.

[jelu]

@danielgriggs Great, thanks! I'll get a release out with this fix today then.