Multi-homed support

[siraaris]

A project I'm working on involves a configuration with a 4-port ethernet PCI card (Intel i350) on a DSP host working with 32ch audio, with four endpoints handling 8ch audio each, connected to the server by ethernet XO cables (no switch in the setup). PTP is locking on each of the endpoint and server aes67-daemon instances, using software time stamping and modified LinuxPTP to support loopback multicast.

However as aes67-daemon only provides one ip_addr and interface name, only one of the endpoints is able to see the server.

I'm wondering whether aes67-daemon could be modified to support multiple ethernet interfaces.

Would be great to be able to connect 4 devices to a "broadcast" host without a switch.

Aris

[bondagit]

This configuration is not supported at the moment as the kernel module is tied to a single network interface.
It's possible to add support for multiple network interfaces but it would require modification to the kernel module, its interface, the daemon and the daemon REST API.

[siraaris]

Nod. Appreciate the analysis and comment back.

Not sure if you mark issues as feature requests. I'll keep working on my project with a switch proper and loop back here once the setup is working in the standard configuration.

Kind regards,

Aris

[bondagit]

yes, we can use the issue as feature request. Let's keep it open then.

[siraaris]

Just dropping in to say that the standard configuration (fed by a separate PTP master) works as expected :)

[doughecka]

Can it be tied to a bridge interface, instead of a physical interface, and then the physical interfaces could be joined to that bridge?

[bondagit]

Can it be tied to a bridge interface, instead of a physical interface, and then the physical interfaces could be joined to that bridge?

You can give it a try, it would be interesting to see the result.

[siraaris]

I think PTP doesn't work with bridge networking. See this.

[bondagit]

As long as on the bridge interface you can receive multicast traffic it should work.
The driver implements a PTP slave only.

[siraaris]

The configuration I was running was based on distributing 4 sets of 8ch audio from a main AES67-DAEMON instance, using a an appropriate switch to which 4 end point devices are patched in.

I removed the switch, replacing with a bridged setup on all ports of a 4-port PCIe card, and set the main AES67-DAEMON instance to use this bridge.

I tried running PTP4L on one of the physical 4-port interfaces that are part of the bridge (no sync).
I tried running PTP4L on another ethernet interface available on the machine, and adding that to the bridge. Less success, as PTP relies on a connected interface I think.

I did connect a separate device running PTP to one of the bridged interfaces - and yes, that works nicely.

So, I've removed the switch (nice), but still need a separate physical device that patches into the bridge to provide a clock.

Maybe I misunderstood the comments above?

[bondagit]

did you patch ptp4l in order to enable IP_MULTICAST_LOOP ?
See: #34

Without the patch you will not be able to receive PTP packets sent by ptp4l on the same interface where the daemon is running.

[siraaris]

Thanks. I have patched ptp4l, and retried.

Regardless of the interface I run ptp4l on, I observe the following:

• Main aes67-daemon process always syncs.
• The aes67-daemon process on the same physical device as ptp4l is set also syncs but the others don't.

Maybe multicast is not setup correctly...

[bondagit]

can you describe your setup better? are you running multiple daemon instances on the same host?

[siraaris]

System consists of:

• primary DSP host, running Debian Linux 12

• This host runs AES67-daemon configured with 4 sources, each with 8ch

• it has a 4 port Ethernet card (Intel i350)

• it also has two other Ethernet devices, one used as default gateway

• 4 satellite hosts also on Debian 12 running aes67-daemon, configured to pick up one of the 8ch streams each

Scenario 1:

• do not configure the 4-port Ethernet at all
• configure the 2nd Ethernet dev with some network
• Configure aes67-daemon to this interface
• configure satellite hosts and their aes67-daemon with same network
• connect the primary and satellite hosts to a switch
• configure an additional PTP host and patch it into the switch

This configuration works just fine.

Scenario 2:

• configure 4-port card devices into a bridge (br0)
• configure br0 with some network
• configure the 4 satellite hosts to the same network and patch these into ports on the 4-port Ethernet device
• Run patched PTP on any of the 4port card interfaces

This configuration results in two aes67-daemon PTP syncs only; the primary host always syncs, as does the satellite system that is associated with the port on which the patched PTP is run.

Hopefully this is clear, can make a diagram if needed!

[bondagit]

I cannot reproduce your environment here, try to disable the multicast snooping on the interface br0:

echo 0 > sys/class/net/br0/bridge/multicast_snooping

and also, maybe:

echo 2 > /sys/class/net/br0/bridge/multicast_router

[bondagit]

any update on this ?

[siraaris]

I'll try this tonight and let you know how it goes.

[siraaris]

Well, those options didn't do anything.

But good news - what seems to work is if I run PTP4L in JBOD mode. Then it seems to work just fine - all clocks sync.

[siraaris]

My command (on the primary aes67-daemon host that has the 4port ethernet):

root@controller:~/bin# /usr/local/sbin/ptp4l -i eno2 -i enp1s0f0 -i enp1s0f1 -i enp1s0f2 -i enp1s0f3 -f /etc/linuxptp/ptp4l.conf --domainNumber 0 --dscp_event 46 --dscp_general 34 --logAnnounceInterval 1 --logSyncInterval 0 --logMinDelayReqInterval 0 --logMinPdelayReqInterval 0 --announceReceiptTimeout 3 --boundary_clock_jbod 1 -m

Maybe not all those options are necessary, I re-used an old config I had from previous projects.

[siraaris]

Maybe I spoke too soon, the endpoints sync, but not the primary host. Sigh.

[siraaris]

Very weird, checking the settings:

root@controller:# cat /sys/class/net/br0/bridge/multicast_snooping
1
root@controller:# cat /sys/class/net/br0/bridge/multicast_router
2

and running PTP4L:

/usr/local/sbin/ptp4l -i enp1s0f0 -i enp1s0f1 -i enp1s0f2 -i enp1s0f3 -f /etc/linuxptp/ptp4l.conf --domainNumber 0 --dscp_event 46 --dscp_general 34 --logAnnounceInterval 1 --logSyncInterval 0 --logMinDelayReqInterval 0 --logMinPdelayReqInterval 0 --announceReceiptTimeout 3 --boundary_clock_jbod 1 -m

All aes67-daemon's sync (the primary & the 4 satellite hosts).

I'll monitor it for a while and report back if everything stays synced.

The (patched) ptp4l output seems well behaved:

ptp4l[252799.913]: selected /dev/ptp1 as PTP clock
ptp4l[252799.913]: port 2 (enp1s0f1): just a bunch of devices
ptp4l[252799.913]: port 3 (enp1s0f2): just a bunch of devices
ptp4l[252799.913]: port 4 (enp1s0f3): just a bunch of devices
ptp4l[252799.914]: port 1 (enp1s0f0): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[252799.915]: port 2 (enp1s0f1): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[252799.916]: port 3 (enp1s0f2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[252799.917]: port 4 (enp1s0f3): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[252799.917]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[252799.917]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[252807.254]: port 1 (enp1s0f0): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[252807.254]: selected local clock a0369f.fffe.51b55c as best master
ptp4l[252807.254]: port 1 (enp1s0f0): assuming the grand master role
ptp4l[252807.277]: port 2 (enp1s0f1): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[252807.277]: port 1 (enp1s0f0): assuming the grand master role
ptp4l[252807.277]: port 2 (enp1s0f1): assuming the grand master role
ptp4l[252807.838]: port 3 (enp1s0f2): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[252807.838]: port 1 (enp1s0f0): assuming the grand master role
ptp4l[252807.838]: port 2 (enp1s0f1): assuming the grand master role
ptp4l[252807.838]: port 3 (enp1s0f2): assuming the grand master role
ptp4l[252807.891]: port 4 (enp1s0f3): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[252807.891]: port 1 (enp1s0f0): assuming the grand master role
ptp4l[252807.891]: port 2 (enp1s0f1): assuming the grand master role
ptp4l[252807.891]: port 3 (enp1s0f2): assuming the grand master role
ptp4l[252807.891]: port 4 (enp1s0f3): assuming the grand master role

[siraaris]

Well, I've stopped ptp4l a few times this evening (and all aes67-daemons unsync), start ptp4l up (and all aes67-daemons sync again, and hold).

If I find any failures I'll report back.

But for now, running in JBOD mode, with the patched ptp4l mentioned above seems to work for me for running aes67-daemon and Ravenna kernel module via a Linux Bridge configuration of a multi-port card or cards without using a separate switch.

Note the ptp4l.conf configuration is pretty much stock.

[bondagit]

/usr/local/sbin/ptp4l -i enp1s0f0 -i enp1s0f1 -i enp1s0f2 -i enp1s0f3 -f /etc/linuxptp/ptp4l.conf --domainNumber 0 --dscp_event 46 --dscp_general 34 --logAnnounceInterval 1 --logSyncInterval 0 --logMinDelayReqInterval 0 --logMinPdelayReqInterval 0 --announceReceiptTimeout 3 --boundary_clock_jbod 1 -m

I guess you cannot run ptp4l n on the br0 interface directly because of the missing support for the requested timestamp mode, correct ?

[siraaris]

Yes, that's my understanding. ptp4l notices a bridge interface and dumps out.

Assuming the JBOD with loopback mod for ptp4l approach is stable with aes67-daemon, it would satisfy the need that this Issue was raised for, and you can either close, or keep open if you want to consider a different longer term approach down the track.

[bondagit]

solution for multi-homed support via bridge interface