Flow packet has only one defined interface

[aapostoliuk]

Debian 12.2.0-14. VyOS 1.5 rolling release
I build and install the latest package of hsflowd 2.1.12-4
Interface configuration.

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 0c:c2:47:b9:00:01 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname ens5
    inet 192.168.30.1/24 brd 192.168.30.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::ec2:47ff:feb9:1/64 scope link
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 0c:c2:47:b9:00:02 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    altname ens6
    inet 192.168.40.1/24 brd 192.168.40.255 scope global eth2
       valid_lft forever preferred_lft forever
    inet6 fe80::ec2:47ff:feb9:2/64 scope link
       valid_lft forever preferred_lft forever

ICMP traffic is generated from host 192.168.30.10 to host 192.168.40.10 via this router.
sFlow config:

sflow {
  polling=30
  sampling=1000
  sampling.bps_ratio=0
  collector { ip = 192.168.111.1 udpport = 6343 }
  pcap { dev=eth1 }
  psample { group=1 egress=on }
  dbus { }
}

If I run hsflowd in debug mode I see only one interface is defined. The other interface is not found.

_root:takeSample: hook=0 tap=eth1 in=<not found> out=eth1 pkt_len=84 cap_len=84 mac_len=14 (0C51DC210000 -> 0CC247B90001 et=0x0800)
_root:selected sampler eth1 ifIndex=3
_root:takeSample: hook=0 tap=eth1 in=<not found> out=eth1 pkt_len=84 cap_len=84 mac_len=14 (0C51DC210000 -> 0CC247B90001 et=0x0800)
_root:selected sampler eth1 ifIndex=3
_root:takeSample: hook=0 tap=eth1 in=eth1 out=<not found> pkt_len=84 cap_len=84 mac_len=14 (0CC247B90001 -> 0C51DC210000 et=0x0800)
_root:selected sampler eth1 ifIndex=3
_root:takeSample: hook=0 tap=eth1 in=eth1 out=<not found> pkt_len=84 cap_len=84 mac_len=14 (0CC247B90001 -> 0C51DC210000 et=0x0800)
_root:selected sampler eth1 ifIndex=3

As a result, one of the interface indexes is 1073741823 in the flow packets.
Is it a normal situation or not? If yes, can you explain why?

[sflow]

Yes. The interface number 0x3FFFFFFF means "internal" in sFlow. In a regular network switch we use "internal" for all packets that were to/from the switch CPU, such as SNMP request/response packets. That's because the SNMP MIB-II ifTable (usually) does not have an entry that represents the local CPU.

When monitoring a server like this, the model is similar. It's like a switch with one port. Everything to/from the host is considered to be to/from the "internal" interface. When you sample using pcap { dev=eth1 } you are enabling bidirectional sampling on that NIC and adopting that model.

If you think of the whole packet-switched network as a graph then what you are saying is that this NIC represents the entry-point to the graph. It's a little different if you ask hsflowd to monitor a virtual switch such as Open vSwitch or the Docker bridge, because then you are allowing the network-graph to extend to the virtual switch ports. But after that the model is the same.

Note that this issue is similar to this one about proxy forwarding:
#70

[aapostoliuk]

It seems that I do not understand something.
The network is
host(192.168.30.10)---(eth1=192.168.30.1)VyOS(eth2=192.168.40.1)----host(192.168.40.10)
ICMP traffic is routed via VyOS from eth1 interface to eth2 interface
I added eth2 to sFlow too.

sflow {
  polling=30
  sampling=1000
  sampling.bps_ratio=0
  collector { ip = 192.168.111.1 udpport = 6343 }
  pcap { dev=eth1 }
  pcap { dev=eth2 }
  psample { group=1 egress=on }
  dbus { }
}

And I see the same situation.

_root:takeSample: hook=0 tap=eth2 in=eth2 out=<not found> pkt_len=84 cap_len=84 mac_len=14 (0CC247B90002 -> 005079666800 et=0x0800)
_root:selected sampler eth2 ifIndex=4
_root:takeSample: hook=0 tap=eth2 in=<not found> out=eth2 pkt_len=84 cap_len=84 mac_len=14 (005079666800 -> 0CC247B90002 et=0x0800)
_root:selected sampler eth2 ifIndex=4
_root:takeSample: hook=0 tap=eth1 in=<not found> out=eth1 pkt_len=84 cap_len=84 mac_len=14 (0C51DC210000 -> 0CC247B90001 et=0x0800)
_root:selected sampler eth1 ifIndex=3
_root:takeSample: hook=0 tap=eth2 in=eth2 out=<not found> pkt_len=84 cap_len=84 mac_len=14 (0CC247B90002 -> 005079666800 et=0x0800)
_root:selected sampler eth2 ifIndex=4
_root:takeSample: hook=0 tap=eth1 in=eth1 out=<not found> pkt_len=84 cap_len=84 mac_len=14 (0CC247B90001 -> 0C51DC210000 et=0x0800)
_root:selected sampler eth1 ifIndex=3

Should it be one filled interface in raw and the other interface should be , if traffic is forwarded between two interfaces (eth1, eth2)?

[sflow]

Yes, strictly speaking that would be more correct, but hsflowd does not know all the details of the forwarding decision in each case. An sFlow agent embedded in the FRR routing engine that VyOS runs could probably populate in+out ports correctly, but here hsflowd is running as a regular systemd service only knows so much.

If hsflowd were to query the routing tables via netlink then it might do better, but this is where Linux can be tricky. You would have to know which routing table to query, and that might depend on a mark set by iptables/nftables. Furthermore, even if you can look up the L3 route, there might still be ECMP options and the decision on which L2 component link to send the packet out on might be made somewhere else. So the disaggregated nature of Linux networking makes this harder. When sFlow is implemented on an ASIC it can be architected to attach the right metadata to each sample down in the silicon (although some ASICs do not do any better than what you are seeing here). Open vSwitch and VPP agents have a better chance of knowing more about the forwarding too. Doing the same thing for generic Linux networking is a challenge.

The good news is that most of the value provided by sFlow is still there. You know what the traffic is on each link, and your collector has the information to resolve the traffic matrix for the whole network.

If your collector can run BGP peering then some of the missing routing information can be spliced in that way.