can't create backup path

[vandit86]

Hello,
I try to create a backup path but with no success, i.e. mptcp always use two available paths.
Will appreciate any suggestion..

my setup :
kernel version : 5.14.0
ip utility, iproute2-5.14.0

routing

n1 :

root@n1:/tmp/pycore.33653/n1.conf# ip r
default via 11.0.0.1 dev eth0 
11.0.0.0/8 dev eth0 proto kernel scope link src 11.0.0.2 
13.0.0.0/8 via 11.0.0.1 dev eth0 
14.0.0.0/8 via 15.0.0.1 dev eth1 
15.0.0.0/8 dev eth1 proto kernel scope link src 15.0.0.2 

n4 (server) :

root@n4:/tmp/pycore.33653/n4.conf# ip r
default via 13.0.0.1 dev eth0 
11.0.0.0/8 via 13.0.0.1 dev eth0 
13.0.0.0/8 dev eth0 proto kernel scope link src 13.0.0.2 
14.0.0.0/8 dev eth1 proto kernel scope link src 14.0.0.2 
15.0.0.0/8 via 14.0.0.1 dev eth1 

ip mptcp

n1:

root@n1:/tmp/pycore.33653/n1.conf# ip mptcp e
15.0.0.2 id 1 subflow backup dev eth1 
11.0.0.2 id 2 subflow dev eth0 
root@n1:/tmp/pycore.33653/n1.conf# ip mptcp limits 
add_addr_accepted 1 subflows 1 

n4 (server)

root@n4:/tmp/pycore.33653/n4.conf# ip mptcp e
13.0.0.2 id 1 signal dev eth0 
14.0.0.2 id 2 signal backup dev eth1
root@n4:/tmp/pycore.33653/n4.conf# ip mptcp l
add_addr_accepted 1 subflows 1

ip mptcp monitor on n4 (running iperf3 tool for traffic generation)

root@n4:/tmp/pycore.33653/n4.conf# ip mptcp m
[       CREATED] token=38f500f7 remid=0 locid=0 saddr6=::ffff:13.0.0.2 daddr6=::ffff:11.0.0.2 sport=5201 dport=54542
[   ESTABLISHED] token=38f500f7 remid=0 locid=0 saddr6=::ffff:13.0.0.2 daddr6=::ffff:11.0.0.2 sport=5201 dport=54542
[SF_ESTABLISHED] token=38f500f7 remid=0 locid=2 saddr6=::ffff:14.0.0.2 daddr6=::ffff:15.0.0.2 sport=5201 dport=47359 backup=0
[       CREATED] token=2684661a remid=0 locid=0 saddr6=::ffff:13.0.0.2 daddr6=::ffff:11.0.0.2 sport=5201 dport=54544
[   ESTABLISHED] token=2684661a remid=0 locid=0 saddr6=::ffff:13.0.0.2 daddr6=::ffff:11.0.0.2 sport=5201 dport=54544
[SF_ESTABLISHED] token=2684661a remid=0 locid=2 saddr6=::ffff:14.0.0.2 daddr6=::ffff:15.0.0.2 sport=5201 dport=58367 backup=0
[        CLOSED] token=2684661a
[        CLOSED] token=38f500f7

Also, the backup flag is not set on tcpdump output.

The similar issue was reported in here https://githubmemory.com/repo/multipath-tcp/mptcp_net-next/issues/191, but I have not found any solution.

[matttbe]

Hi @vandit86 ,

Thank you for this bug report!

Do you think you could re-do this experiment with a 5.15 kernel (or ideally with our export branch)?
Some improvements have been done around the use of the backup flag (Packet scheduler improvements (especially with backup subflows)). It will also help debugging the issue if you have this dev version.

[matttbe]

(For Github tracking: Possibly linked to #191 )

[pabeni]

It looks there are a few problems leading to the above behavior:

• the ADD_ADDR sub-option does not carry (per-rfc) the backup flag
• the signaled address received by the client is processed before any local endpoint (as per kernel implementation)
• the client creates only a single subflow (as per user configuration). Such subflow is a non backup one due to the above.

The solution should be removing the 'signal' endpoint from the server, so that the client can use the 'subflow' endpoint.

Note that to allow the client 'subflow' endpoints to work correctly you additionally need some routing setup. Specifically the server address you connect to must be reachable from both the client's links.

Assuming the client connects to 13.0.0.2, something alike the following should work (on top of the existing routing config):

[ on client n1]

# the non default metric is need to avoid a 'File exists' error due to existing route on the other link
ip route add 13.0.0.8/8 via 15.0.0.1/8 dev eth1 metric 100

[ on router n3]
ip route add 13.0.0.8/8 via 14.0.0.2

[vandit86]

Hi @vandit86 ,

Thank you for this bug report!

Do you think you could re-do this experiment with a 5.15 kernel (or ideally with our export branch)? Some improvements have been done around the use of the backup flag (Packet scheduler improvements (especially with backup subflows)). It will also help debugging the issue if you have this dev version.

Thank you for the quick answers! I will try switch to newer kernel and test mptcp.. I'm not so experienced working with kernel, thus it might take some time before I give you feedback.

[vandit86]

Hi @pabeni.
Thanks for the quick answer.

• the ADD_ADDR sub-option does not carry (per-rfc) the backup flag

Here is like Add Address looks on first path ( i.e. server send ADD_ADDR request to client, and no backup flag is set)

Multipath Transmission Control Protocol: Add Address
    Kind: Multipath TCP (30)
    Length: 16
    0011 .... = Multipath TCP subtype: Add Address (3)
    .... 0000 = IP version: 0
    Address ID: 2
    Truncated HMAC: 1008806329321223511

The solution should be removing the 'signal' endpoint from the server, so that the client can use the 'subflow' endpoint.

Sorry, I don't understand your suggestion. If I remove "signal" flag from server endpoint, how the client will know to create new subflow, ? i.e., no ADD_ADDR will be sent, thus no subflow will be created.. Am I right ??

• the client creates only a single subflow (as per user configuration). Such subflow is a non backup one due to the above.

Client connect to server 11.0.0.2 <-->13.0.0.2 (initial connection). Server sends ADD_ADDR to client, and client always create additional subflow (through 15.0.0.2 <-->14.0.0.2), even if backup flag is set by ip mptcp , on server and on client..
I have experimented with different configurations, changing endpoints and limits parameters.

[pabeni]

Hi @pabeni. Thanks for the quick answer.

• the ADD_ADDR sub-option does not carry (per-rfc) the backup flag

Here is like Add Address looks on first path ( i.e. server send ADD_ADDR request to client, and no backup flag is set)

Multipath Transmission Control Protocol: Add Address
    Kind: Multipath TCP (30)
    Length: 16
    0011 .... = Multipath TCP subtype: Add Address (3)
    .... 0000 = IP version: 0
    Address ID: 2
    Truncated HMAC: 1008806329321223511

The solution should be removing the 'signal' endpoint from the server, so that the client can use the 'subflow' endpoint.

Sorry, I don't understand your suggestion. If I remove "signal" flag from server endpoint, how the client will know to create new subflow, ? i.e., no ADD_ADDR will be sent, thus no subflow will be created.. Am I right ??

No.

The client creates additional subflows are created if:

• receives one or more ADD_ADDR options
  or
• there is a 'subflow' endpoint configured on the client side.

Note the above is a simplification of the actual rules used, as mptcp limits are also taken in account, etc... The relevant part is that client side 'subflow' endpoints are sufficient for additional subflows creation. The client will try to create new subflow using the endpoint address as source and the known server address as destination.

Please try the configuration specified in the previous comment and let us know the result.

• the client creates only a single subflow (as per user configuration). Such subflow is a non backup one due to the above.

Client connect to server 11.0.0.2 <-->13.0.0.2 (initial connection). Server sends ADD_ADDR to client, and client always create additional subflow (through 15.0.0.2 <-->14.0.0.2), even if backup flag is set by ip mptcp , on server and on client.. I have experimented with different configurations, changing endpoints and limits parameters.

That happens because the ADD_ADDR option does not carry the 'backup' flag. The client connects to the address specified by the ADD_ADDR option with default (non backup) flag. Any additional client endpoint is irrelevant for such subflow creation.

[vandit86]

Thank you for explanation, @pabeni !

Please try the configuration specified in the previous comment and let us know the result.

I configure routing of topology as you suggests (also removing "endpoints" from server and adding one endpoint on client with ip mptcp endpoint add 15.0.0.2 dev eth1 subflow backup, leaving just subflow limits on both sides) and backup flag actually appears in dump. However, mptcp is continuing to "split" traffic through two available subflows, ignoring the backup flag setted on the path.

n1:

root@n1:/tmp/pycore.33653/n1.conf# ip mptcp e
15.0.0.2 id 1 subflow backup dev eth1 
root@n1:/tmp/pycore.33653/n1.conf# ip mptcp l
add_addr_accepted 0 subflows 2 

Server (n4) :

root@n4:/tmp/pycore.33653/n4.conf# ip mptcp e
root@n4:/tmp/pycore.33653/n4.conf# ip mptcp l
add_addr_accepted 0 subflows 2 

ip mptcp monitor on n4 (with iperf3):

[       CREATED] token=b90bc994 remid=0 locid=0 saddr6=::ffff:13.0.0.2 daddr6=::ffff:11.0.0.2 sport=5201 dport=54690
[   ESTABLISHED] token=b90bc994 remid=0 locid=0 saddr6=::ffff:13.0.0.2 daddr6=::ffff:11.0.0.2 sport=5201 dport=54690
[SF_ESTABLISHED] token=b90bc994 remid=1 locid=0 saddr6=::ffff:13.0.0.2 daddr6=::ffff:15.0.0.2 sport=5201 dport=50917 backup=1
[       CREATED] token=78e93e3a remid=0 locid=0 saddr6=::ffff:13.0.0.2 daddr6=::ffff:11.0.0.2 sport=5201 dport=54692
[   ESTABLISHED] token=78e93e3a remid=0 locid=0 saddr6=::ffff:13.0.0.2 daddr6=::ffff:11.0.0.2 sport=5201 dport=54692
[SF_ESTABLISHED] token=78e93e3a remid=1 locid=0 saddr6=::ffff:13.0.0.2 daddr6=::ffff:15.0.0.2 sport=5201 dport=43125 backup=1
[        CLOSED] token=78e93e3a
[        CLOSED] token=b90bc994

[pabeni]

Thank you for explanation, @pabeni !

Please try the configuration specified in the previous comment and let us know the result.

I configure routing of topology as you suggests (also removing "endpoints" from server and adding one endpoint on client with ip mptcp endpoint add 15.0.0.2 dev eth1 subflow backup, leaving just subflow limits on both sides) and backup flag actually appears in dump. However, mptcp is continuing to "split" traffic through two available subflows, ignoring the backup flag setted on the path.

That is indeed a bug, fixed with the upstream commit 0460ce2 present in Linux v5.15-rc1and newer. Please try the most up2date kernel, thanks!

[matttbe]

So as a conclusion, if I'm not mistaken, it looks like no modification is needed in the kernel code but probably best to improve the ip-mptcp man page or other doc related to that.

[vandit86]

Thank you, @matttbe @pabeni .
It seems that backup path works well with Linux kernel v5.15.0-rc7. I think this issue can be closed.
As I understand, there is no way to create a sublfow in backup mode after client receives ADD-ADDR from server. The endpoint should be pre-configured as backup on the client before any mptcp session starts??

[pabeni]

As I understand, there is no way to create a sublfow in backup mode after client receives ADD-ADDR from server. The endpoint should be pre-configured as backup on the client before any mptcp session starts??

Indeed subflows created in response to incoming ADD_ADDR will always be non backup. We could improve the scenario with an additional PM netlink setting, that will allow configuring the backup status for such subflows.

I'm not sure about the better way to expose this to user-space. A dummy, default (0.0.0.0) endpoint that will match only incoming ADD_ADDR ?!? Also I'm wondering if that should be instead handled at the protocol (RFC) level, expanding the ADD_ADDR option definition with a backup flag.
@matttbe, @mjmartineau: WDYT?

[matttbe]

I'm not sure about the better way to expose this to user-space. A dummy, default (0.0.0.0) endpoint that will match only incoming ADD_ADDR ?!? Also I'm wondering if that should be instead handled at the protocol (RFC) level, expanding the ADD_ADDR option definition with a backup flag.

I don't think we need to modify the RFC: both the client and sender can add the backup in the MPJ. So when the ADD_ADDR is received, the client can add the backup flag if the endpoint it is using has this flag. Same on the server side, no?

Eventually, there could be an option to create new subflow using the initial IP with the backup flag. But here too, we could create an endpoint with the same IP and set the backup flag. WDYT?

[mjmartineau]

@pabeni: I agree with @matttbe, at the protocol level both peers have the ability to set the backup bit during the MP_JOIN handshake so there's no need to change the advertisement.

When sending MP_JOIN in response to an ADD_ADDR, the backup bit and choice of local endpoint seem to be closely coupled. For example, if the new subflow and existing subflows are on the same interface for the MP_JOIN sender, that sender doesn't really need to use the backup bit to influence the peer's packet scheduler. If all new subflows are established by the in-kernel PM without considering the interface it's on, it seems like we don't gain much by having a default 0.0.0.0 endpoint backup setting.

It seems like the missing link is using the in-kernel's PM existing endpoint information for outgoing subflow establishment to influence the source address for outgoing MP_JOINs rather than depending entirely on route lookup (as you noted on IRC).

[mjmartineau]

Regarding my proposal to use existing endpoint information, I think it falls within the scope of the ip mptcp endpoint documentation:

      subflow
              if additional subflow creation is allowed by MPTCP limits,
              the endpoint will be used as the source address to create
              an additional subflow after that the MPTCP connection is
              established.

[vandit86]

Indeed subflows created in response to incoming ADD_ADDR will always be non backup. We could improve the scenario with an additional PM netlink setting, that will allow configuring the backup status for such subflows.

ok, regarding the same topology showed above:
Server (n4) is configured to advertise additional address, and when the client receives ADD_ADDR, the PM of the client creates subflow (MP_JOIN) with Backup flag disabled.. Always.. And there is no way to set new subflow as backup for client.

C    <--   ADD_ADDR  --    S 
C    ---    MP_JOIN, Backup=0  --> S

I don't think we need to modify the RFC: both the client and sender can add the backup in the MPJ. So when the ADD_ADDR is received, the client can add the backup flag if the endpoint it is using has this flag. Same on the server side, no?

Please, can you show me how to do this with ip mptcp configuration tool .. Or using mptcpd PM is the only way to do this from user space??

[matttbe]

Please, can you show me how to do this with ip mptcp configuration tool .

If I'm not mistaken, it is currently not possible to do that with 'ip'.
It should "soon" be possible to do that with mptcpd but #186 needs to be implemented first.

[matttbe]

So as a conclusion, if I'm not mistaken, it looks like no modification is needed in the kernel code but probably best to improve the ip-mptcp man page or other doc related to that.

This has been done by @pabeni https://lore.kernel.org/all/247b8dfb7254d4a1fb435b5efa756cea989b62cb.1637922870.git.pabeni@redhat.com/