Proposal: Additional option fields for wireless packet capture #48
Comments
So presumably if the capture device isn't moving, the location can be in the IDB rather than in every EDB, and if it's not doing scanning, the frequency information can be in the IDB rather than in every EDB. Would the same apply to the channel number? |
What if you're receiving on more than one antenna? |
What indicates how the channel number is to be interpreted? I.e., is this an 802.11 channel, a Bluetooth channel, etc.? |
|
Thanks for the quick response! In order: WRT channel number in EPB or IDB, yep that would make sense. WRT RSSI, I'm not aware of hardware that actually receives the same packet on N antennae rather than just picking the best one at a given point in the preamble, but, support for multiple RSSI options per packet would work? (And if they're actually parallel, that may be better modelled as N interfaces). WRT. channel number interpretation, this is I guess backwards from the current way. That does raise the question of support for further metadata (like channel b/w, spacing), as it stands I have been putting that kind of data in the file and interface comments since it's not /required/ for parsing / understanding the packet capture. |
|
I would also be interested in GPS data; I'd propose pushing it to include some additional info, modelled on the PPI GPS framework: https://media.blackhat.com/bh-us-11/Cache/BH_US_11_Cache_PPI-Geolocation_WP.pdf The lat/lon/alt/ts is very similar to what is proposed above; the addition of VECTOR would allow for precision aimed antennas (something the original ppi geolcoation method was designed for). Position in both IDB and EDB would account for all my needs (fixed and moving sensors); IDB alone wouldn't be enough. I'd like to suggest breaking the record into 'gps' and 'wifi specific'; I've got non-wifi packet sources in Kismet which still have location data. For the wifi-specific component, you may find it difficult to represent channel simply; in wifi alone you've got 5, 10 15 (non-standard) widths, 20, HT20, HT40+ HT40-, HT80+center, HT160+center, and HT80+80 with dual non-contiguous center frequencies. For RSSI, any N/AC hardware with mimo will report multiple signal levels in the radiotap header; you've got an arbitrary number of antenna signal levels (1-3 on current hardware but more in the future is certainly a possibility). |
|
@kismetwireless good points! IDK whether the pcapng team would rather one big issue or multiple smaller ones (or how to actually enact any of this 😂)? Totally agreed about position options for both IDB and EDB. So the VECTOR addition would add pitch/roll/yaw? I would make the suggestion that perhaps that would better be called orientation, and I have been wondering whether speed was useful to encompass doppler shift of measurements. Also that hopefully it could be more simple. I don't understand what you mean by 'gps' and 'wifi specific'? The goal here is to have useful physical data available for any packet capture source (I work on a variety of ISM band mesh networks which is the use case for me) That is true, but the question is whether pcap-ng should include that, or, have some kind of generic enough mapping for it to be implemented for arbitrary protocols. Another option would be to add channels to the IDB and reference them from each EPB, but that's adding quite a lot of complexity, and means that wifi channels will end up getting encoded in every pcap file. For RSSI, would the addition of an antenna index to the rssi option cover that? |
Updated Additional Options
RSSI could be 2x 16-bit to save a 4 bytes, but, 32-bit floats are probably easier to deal with in most situations (unless you're missing an FPU). |
|
re: wifi vs gps - we need gps for non-wifi packets, so it doesn't make sense to couple them. We should have a GPS EPB and IDB, and a 'frequency' or similarly named EPB and IDB. For the GPS frame I'd suggest all the data mapped in the GPS PPI spec which I linked above, starting on page 22. |
|
See also another proposal for a "Location Information Block" in the archives of the pcapng mailing list. Here are the thread from March 2015 and the continuation of that thread in April 2015. |
|
@kismetwireless absolutely and that's why I am proposing this? AFAIK it's currently coupled because the recommended way of getting location data into wireshark is wrapping whatever data you have in a radiotap header? @guyharris thanks for linking those, do you know what happened following that discussion? (For other readers, this post contains a fairly thorough formatted proposal with responses following, and this covers many of the same options as we're talking about with different formats/units (with much of the PPI spec stuff). I think the distinction between a |
There were still questions outstanding, so nothing happened after that.
"This discussion" meaning the discussion here on GitHub, or the e-mail discussion from 2015? The best thing to do with the 2015 discussion is to pick up from the proposal and answers to questions any ideas that look good and, for open questions, possibly resolve them (e.g., the questions about the Source Types) and incorporate the results into the your proposal if they look good.
As in "we need to offer both types of option" or as in "the option should be called "location" rather than "gps", as it might not come from GPS or even from any of the satellite-based positioning systems"?
So it sounds as if you mean ""the option should be called "location" rather than "gps""; if so, I agree.
I'll leave that for others to answer.
Options in the IDB, like options in the LPB, can be thought of a a form of compression - instead of having to put values for those options in every EPB, if they'll be the same for every packet on a given interface, or for every packet geotagged by a given location provider, there can be one instance of them in the IDB or LPB and the EPB can just point to the IDB or LPB by ID. You already have to do indexing to refer to the IDB to get, for example, the link-layer header type; similar code an do indexing to refer to an LPB, so I don't consider it an extreme amount of extra complexity. I guess the "compressed" options could be attached to an IDB, but if you have, for example, a laptop with two WI-Fi adapters and a GPS receiver, you have one location provider and two interfaces, so you conceptually have one location provider, so I could see having an LPB as being a bit cleaner in that sense. |
There are two ways to provide per-packet metadata: provide them in a link-layer-header-type-specific metadata header preceding the link-layer header, or provide them in pcapng options. The first one has the advantages that:
The second one has the advantage that, if it's not specific to a particular link-layer type, you don't have to add it to N different link-layer types. The location/angle/velocity information is definitely not link-layer specific, and strikes me as an obvious candidate for inclusion in pcapng (at the expense of it not being representable in pcap, but there's already information that's not representable in pcap). |
|
@guyharris thanks for the responses! The discussion being all of the discussions in all the places ^_^
Cool, and what format should the proposal be? Would a PR against the base XML file work?
Exactly.
IDK how much more it is for parsers, but, my suggestion is that putting these in the IDB and EPB is still simpler compared to introducing an LPB as the IDB lookup already exists. As mentioned in the mailinglist adding the LPB is also going to increase the complexity of merging captures, whereas the other approach should work as without modification to existing tools.
It is probably cleaner, but for N adaptors in a static (not moving) configuration at the same location, in which case it's going to save N-1 location headers. IMO it would be a mistake to end up allowing both, so, the to conclude this the options are either:
The former will (probably) result in better compression for some use cases, but requires modification of existing tools to support merging (and parsing?) pcap-ng files containing LPBs. The latter may result in data duplication where N > 1 interfaces are used from the same place under certain conditions, but should not require any modification to support merging or splitting of files. IMO it's also easier to implement for parsing / encoding, but, that doesn't matter a lot. That seems like a question of ethos as much as anything?
That's why we're here ^_^ I couldn't quite stomach the thought of wrapping my 6lowpan packets in radiotap headers / writing more oddly specific encoders. |
If there's a convenient way of formatting the updated XML file into the resulting HTML and viewing it, before the chances are merged, yes. Not being able to preview the result would make it more difficult to comment on. |
I think there's some confusion here. Kenneth D. Renard's original proposal had a Location Information Block (LIB), which provides location information. I later proposed a Location Provider Information Block (LPIB) which gives a description of a provider of location information. A capture would have one LPIB per source of location information, no matter how many interfaces or packets there are, or how fast the capturing machine moves. The location update rate would control the number of LIBs, not the number of LPIBs. So there are at least two questions:
I was mainly discussing the LPIB vs. IDB issue; you appear to be discussing the EPB vs. LIB issue. For LPIB vs. IDB, as per my comments, I see having a separate LPIB as conceptually cleaner, even if it requires that the reader keep track of lists of location providers as well as lists of interfaces. For location information in EPBs vs. location information in LIBs, that's where the packet rate and location update rate matter. In the thread about Renard's proposal, I proposed per-EPB location information in this message:
and Renard responds in this message:
I see the LIB as useful only if either 1) the machine is moving slowly enough, and capturing enough traffic, that you'd get less redundant location information (that's the packet rate >> location update case you mention) or 2) the user wants to know how the machine is moving even if it doesn't happen to be capturing any packets (that's a packet rate << location update rate case, but it's not necessarily all packet rate << location update rate cases - if the user doesn't care where the machine is when it's not seeing traffic, the extra LIBs aren't useful). If the two rates are comparable, the LIB doesn't provide any compression. |
Yep, I somehow got the two of them mashed up in my head. s/LPB/LIB/g in my previous comment, and thanks for clearing that up!
IMO the mechanism of location capture is metadata that isn't particularly relevant to the captured packets, though it would make pcap files more standalone.
I suspect given enough interfaces it's probably pretty easy to receive >> packets than location updates (GPS is usually ~10Hz which is /not very fast/).
I don't think this is a use case for a pcap-ng file? If you want a GPS track you can save one using a format for that. I need to map out the implications of each approach again but it's way too late here. Will try and tabulate it tomorrow.
It's gotta be achievable, convenience though 😂 |
|
Hi Guys, cheers tao |
|
Not yet, still need it / intending to work on it but buried under other things at the moment. |
|
After a long time of not having the time to think about this I had the idea that perhaps rather than building this to be location specific, it would make more sense to have a Capture Information Block (CIB) that we can extend with options for location or whatever else is required, and attach to interface description block(s) or refer to from packet block(s). So, a draft of that idea is in progress in #51. It doesn't address all the questions here or in the mailing list yet (because tbqh I'm finding keeping track of all the discussion rather difficult), but, we'll get there hopefully. The immediate questions I have are:
What do y'all think? |
So we now have discussion on:
It might be easier to keep track of it if it's all in one place; if so, where should that be? |
|
I think it would make sense to continue here? |
I'm fine with that so, unless somebody else objects, let's do that (and not further discuss this proposal on the mailing list or in comments on the pull request). |
So what does "capture information" refer to? It's presumably not information about a particular packet, because that should be done as options for packet blocks of various sorts (Enhanced Packet Blocks or any future packet block types). So it could be thought of as a form of compression for cases where the same information applies to multiple packets, under the assumption that the information in question changes rarely enough that you save space by not duplicating the information in multiple EPBs. The current proposal has only position, orientation, and velocity information; presumably those change slowly enough that you would get sufficient compression by using CIBs (and so that, in a capture taken over a very long period of time, you don't get more than 2^32 changes of position, orientation, or velocity!). Do you foresee other forms of information where this putting the information in a CIB, rather than in per-packet options, would result in compression rather than expansion? |
What sort of time stamps? A time stamp of the time at which the information was determined?
Does that change infrequently enough that putting it in a CIB rather than in per-packet options would result in compression rather than expansion? Does the answer depend on which particular RF information is being discussed? |
If the intent is to provide compression of per-packet information that changes sufficiently infrequently, attaching them to EPBs would cover that use case. Attaching them to an IDB would presumably be useful for the case where the information never changes, e.g. a capture done on a machine that's not moving relative to the surface of the earth could have a CIB giving the position, orientation, and velocity of the device (the latter presumably being the zero vector, so maybe it needn't even be reported) attached to all of the IDBs for the capture. BTW, if an EPB has a CIB attached to it and is attached to an IPB with a CIB attached to it, if a given option is present in both CIBs, should the option value in the CIB attached to the EPB be the one used? |
So presumably you're proposing CIB options that could describe a location provider. How often would that information change? If it never changes, putting that in a CIB attached to the IDB would work, but if it can change occasionally, but changes more slowly than other CIB information, it'd be duplicated in all CIBs starting with the first change from values specified in the CIB attached to the IDB. A quick look at Wikipedia suggests that not all satellite-based navigation systems have global coverage yet, so it might be possible for a mobile device capturing packets to switch from one provider to another if the device moves out of the area covered by the nav system it's currently using. By 2020 that might not be a problem (I don't know whether there would be other reasons, technical or political :-), for a device to change from one nav system with global coverage to another while in the process of doing a capture). |
|
Given its absence from the proposal, presumably angular velocity (d{orientation}/dt) isn't of interest? |
Except for obvious editorial changes (typoes, copy-and-pasteos, etc.), which are probably best done in comments on the pull request. |
Updated Summary (TLDR)Thanks for all the commends and feedback! On working through them I think there's a simpler approach, presented first so it's more obvious it's here. Overview
I can update the PR to reflect these changes if you like to see them, or wait until they've been further discussed. Comments / Replies
It could be called "Capture Metadata Block (CMB)" or anything similar, I am open to alternate names if there is a more appropriate one. My original proposal was just to add options to the Enhanced Packet Block (EPB) or Interface Descriptor Block (IDB), however, I think it makes sense to use a separate block:
Definitely more location-related ones, I'm not sure about other types of information. Some RF measurements (like RSSI) are definitely better per packet, some like channel bandwidth are per-interface and/or change with time, but this name supports it if we come up with other options.
Yeah, if we're using global lat/lng/alt from a GPS it'll probably be nice to have the GPS time along with it.
Yep. Bandwidth / Frequency / Channel spacing are often per-interface for simple networks, but are not for wifi etc (as @kismetwireless pointed out).
I was thinking to link the information blocks to the interface blocks (rather than vice versa), so you can re-issue updates to an interface, and the latest CIB applies to all packets on that interface until the next one is read. This also works kinda nicely from the writing side, if you're capturing packets you just have to interleave CIBs when you receive updated location information and they'll apply to everything until your next CIB, which also works nicely for the static case. It does mean if you have multiple interfaces in the same place, you need multiple CIBs. We could allow them to be attached to N interfaces if that was desired, though it seems a little more complicated.
The CIB is really just a list of options, so whatever is relevant can be reported and whatever is not can be elided.
The EPB attached block would override and of the same options specified in the IDB attached block, so the EPB version lets us override the IDB version. On thinking more I'm not completely sure we need to be able to attach them to EPB at all, it might be better to rip out the strictly per-packet options like RSSI and channels as EPB options and remove the ability to link this with an EPB, which would vastly simplify the complexity of parsing logic.
In my experience, most multi-constellation systems still output lat/lng/alt, I don't think which system is providing the information is necessarily relevant. We could add an enum to support that identification, but, I don't know how much sense that makes when you're using a variety of networks. I'm not convinced this is a problem that should be solved within pcap.
I'm not interested in it, but, once we have decided how we're going to do this / split up interface or packet related options, introducing new options is reasonably trivial. |
|
So, for the local location information, is there a need to specify where the origin is? |
"Reused" as in "not having to separately define IDB and EPB options"?
If the metadata is per-packet, then there's nothing to update, so this appears to refer to updating values specified in an IDB. |
So let's not bother with providing a mechanism to identify the location provider until somebody requests it and gives a good argument why it's necessary. |
The specific scenario I am thinking of is the arbitrary grid space of RF simulators, so, no I don't think so.
Yeah, though I think the alternative here is a simpler approach.
Not specified in, but attached to?
Agreed. So the options afaik now are:1. Add a bunch of options to EPBs (and presumably other packet types in future)This would seem to be the simplest option, though does require all metadata to be repeated every packet. An example file would be:
It's very easy, though not super efficient and does not support location sampling >> packet receipt rate. 2. Add a metadata block that EPBs can refer toCreate a metadata block and associated options that are linked to a given packet using an option in the EPB. This would look something like:
This requires some ordering rules (metadata defined before packet so the data is available when streaming), and tracking of metadata block IDs. The static case requires only one metadata block (per interface), however, packet block sizes will increase slightly with metadata link information. I don't think you can link the other way without introducing unique EPB IDs or an EPB ID counter, which generalises poorly for other packet types. 3. Add a metadata block that is associated with an IDB as well as options to EPBs where appropriateFor persistent metadata (eg. location), create a metadata block that is always applied to an interface and applies to any packets following it. For packet-related metadata (eg. RSSI), add options to EPBs. An example file might look like:
This allows compression for the static use case (eg. where location doesn't change) and where packet rate >> change in metadata rate. Where this is not true, up to one metadata block may be required per packet block. Persistent metadata is also generalized over all packet types, and interface location can be tracked at a higher rate than packet capture if required. What nowMy preference is for option three as I think it's reasonably simple/efficient/covers all use cases, or option one for simplicity, and I would be happy to go ahead and rework the PR to match either. |
|
@guyharris any thoughts on these updates? |
|
Sorry, I was out of the discussion for a while for various reasons, including not being very familiar with all the WiFi stuff (which started the discussion). I think option three is the best, even though it adds the most complexity. Do I read the proposal correctly when I assume that MDBs are required when we introduce them (always having at least one MDB per IDB?) I haven't completely made up my mind about it but it may be desirable to still be able to write pcapng files with no MDBs at all. In which case the linking needs to be done via block options, which I would prefer anyway (compared to adding static link fields in the block headers). |
|
For option three, my suggestion is that the MDB includes the link to the interface it applies to (ie. contains the interface id), and if MDBs are available and the parser supports it, metadata gets applied, otherwise they're just another opaque block. That way there should be no compatibility issues (, and you can simply strip or merge pcap files and metadata if desired). |
|
Yes, that sounds like the right way to do it. |
|
Please implement? |
|
@packetfoo did you have a chance to look at #51? I guess the things left are: edit: todo list moved to #51 |
No, I haven't, but I'll look at it this week in greater detail. I'm fighting with git and Github (the workflow concepts of Github refuse to get into my head, apparently), but I'll get there eventually... |
|
Cool cool cool, if it'd be useful we could run through it in a skype call or something, just let me know if I can help (my email is my username at gmail.com if you'd like to get in contact that way) ^_^ |
|
May I ask what happened to the proposed EPB RF options (RSSI, channel, freq, etc)? The PR #51 only addresses the location information in a CIB and the per packet options have fallen by the wayside. I would like to also propose additional metadata for time slotted channel hopping radios which include an absolute slot number and start of slot timestamp (separate from start of frame). |
It turned out that there were a bunch of separate concerns with location information (which relates to interfaces) vs. rssi (which relates only to packets), so I decided to split the problem and attempt the location information in #51 and follow that up with another PR for the radio information. As far as I can see, RSSI and Channel can just be added as fields on the EPB, which I plan to propose once this has landed. |
|
Since PR #51 is created for the CIB options shall discussion on the EPB RF options continue in this issue? Some comments to those specific options.
a. 32-bit is insufficient - even as an unsigned integer it wouldn't represent frequency above ~4.295GHz (i.e. 5GHz band), perhaps an option in CIB/IDB for the multiplier factor for frequency values?
a. separate antennas could also be handled as separate interfaces but perhaps not for MIMO configuration? So multiple instances of the RSSI option could appear in each EPB.
I agree with the first part... Is RSSI as float needed? Also add for TDMA and TSCH channel access methods.
|
|
Re: per-antenna data and MIMO; we've already got 4 signal levels per packet
for some wifi cards now and it'll only get more complex.
Right now, if I recall, radiotap handles this by appending additional
capabilities blocks w/ only the antenna signal filled out; generating
multiple records per packet in pcapng wouldn't be a problem.
…On Thu, Dec 20, 2018 at 2:40 PM James Ko ***@***.***> wrote:
Since PR #51 <#51> is created for
the CIB options shall discussion on the EPB RF options continue in this
issue?
Some comments to those specific options.
Channel / Medium options
a. Frequency in Hz as 1x 32-bit float for EPB and IDB
b. Channel Spacing in Hz as 1x 32-bit float for IDB or EPB
c. Channel Bandwidth in Hz as 1x 32-bit float for IDB or EPB
d. Channel encoding as string for IDB or EPB
e. Channel name as string for IDB or EPB
a. 32-bit is insufficient - even as an unsigned integer it wouldn't
represent frequency above ~4.295GHz (i.e. 5GHz band), perhaps an option in
CIB/IDB for the multiplier factor for frequency values?
b. are there sub-Hz channel spacing implementations? Is 32-bit integer
sufficient?
c. is 32-bit integer sufficient?
f. Data rate in bps or kbps as 32-bit integer for IDB/CIB, is a multiplier
needed here as well?
Capture / Measurement options
a. RSSI in dBm as 1x 32-bit antenna index and 1x 32-bit float for EPB
b. Channel index as 1x 16-bit uint for EPB
a. separate antennas could also be handled as separate interfaces but
perhaps not for MIMO configuration? So multiple instances of the RSSI
option could appear in each EPB.
RSSI could be 2x 16-bit to save a 4 bytes, but, 32-bit floats are probably
easier to deal with in most situations (unless you're missing an FPU).
I agree with the first part... Is RSSI as float needed?
Do any chipsets report RSSI in values other than integers?
Most situations not involving low powered embedded network devices without
an FPU.
I would prefer to see values represented as integers where possible.
Also add for TDMA and TSCH channel access methods.
1. slot number as 64-bit integer
2. start of slot timestamp (separate from start of frame - which is
packet timestamp) - same format an resolution as EPB timestamp (high & low).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#48 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AVScY_a6DglHyJQWS-gLUfSyQ0duZGSkks5u6-eZgaJpZM4QbL0b>
.
|
Yes.
Presumably, by analogy to what radiotap does, you mean "generating multiple options per packet block in pcapng wouldn't be a problem". |
|
Yep; sorry for being imprecise. Appending an option block per antenna
should be fine.
Having an antenna number may be useful in each block, rather than iterating
them; I don't know of any cards currently which report non-sequential
antennas but there's always surprises.
…On Thu, Dec 20, 2018 at 3:09 PM Guy Harris ***@***.***> wrote:
Right now, if I recall, radiotap handles this by appending additional
capabilities blocks w/ only the antenna signal filled out;
Yes.
generating multiple records per packet in pcapng wouldn't be a problem.
Presumably, by analogy to what radiotap does, you mean "generating
multiple options per packet block in pcapng wouldn't be a problem".
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#48 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AVScY9gaVDB9p-N-wdUQSVuzIwBPWjaqks5u6-6VgaJpZM4QbL0b>
.
|
|
I agree.. multiple rssi (antenna#/dBm) blocks rather than "list" of dBm with number of antennas derived from the option length. Additionally, an End of Frame Timestamp (EOF) for time-slotted access methods is desirable since a packet transmission may overflow the timeslot length. |
|
On your first set of comments, for all of those I suggested using an IEEE standard 32-bit float, not a 32-bit integer, which should comfortably store all the values we might require. I think given their prevalence even on embedded systems nowdays it makes sense to use an floats rather than come up with our value / exponent encoding. Should you not have an FPU many of these values could be pre-computed at startup or compile time. For RSSI, if you're playing with localisation there's a big difference between -10dBm and -10.5dBm, and simulations can have arbitrary accuracy, so again I would personally rather see floats.
Seems like a good approach ^_^ also means you can elide antennas if the packet wasn't seen on them. |
Update:
This has been split into two components, per-interface information is approached by introducing a Capture Information Block (CIB) attached to a given interface in #51, per-packet information through the addition of new options to the existing Extended Packet Block (EPB) in #56
The following is left for reference
Original:
Hi there,
I would like to propose the addition of a set of option fields useful for wireless packet capture.
A number of these are available in the 802.11 radiotap (meta)header, however are generally useful outside the 802.11 protocol.
There is an argument here that "there's already code to handle that in programs that read pcap and pcap-ng files, there's already code in drivers to generate that information when capturing packets" which doesn't really stand for non-802.11 protocols.
By supporting these options as part of the core pcap-ng protocol we would enable development of interesting tools for visualisation and analysis of wireless networks that are protocol independent.
The option fields I would propose initially are:
Supporting GPS Location and frequency in the IDB or EPB allows for specification of multiple static capture points or moving captures at different frequencies.
Cheers,
Ryan
The text was updated successfully, but these errors were encountered: