Add HPSJam's XOR packet loss solution to Jamulus?

[chrisrimple]

In #763 is mention of HPSJam's XOR solution to handle up to 30% packet loss without affecting audio quality. Has that been back-ported to Jamulus, or should it be?

[ann0see]

If it's a (simple) implementation which would enable using Jamulus with WiFi or massively decrease dropouts without more latency that would be great!

I haven't tried HPSJam yet, unfortunately.

[corrados]

What is a "XOR solution"?

[chrisrimple]

@hselasky Can you share info on your packet loss solution?

[hselasky]

Hi,

Imagine you send two equally lengthed audio packets: a, b

To achieve redundancy you can either send packets twice:
a a b b

Or send the XOR of the two packets:
a b a^b

In my HpsJam I'm using a fixed number of packets per second.

To send a, b, a^b, I need 3ms. That means that the application also needs to wait 3ms in order to receive those packets correctly.

If any of a, b, or a^b is lost, it can be reconstructed.

a = b ^ a^b
b = a ^ a^b

If a^b is lost, then no problem.

Packet a = 72 samples b = 72 samples a ^ b = 72 samples. (72+72)/3 = 48 samples/ms - 48 kHz

The local timer is adjusted to tick in rate with the average sample speed from the audio device.

On the server, the average of all clients is used for clock synchronization.

--HPS

[gene96817]

Usually, the specific message redundancy method is chosen to match the packet loss mechanism. Does anyone have any idea of what kind of packet loss behavior we need to compensate for? I am also dealing with audio problems. However, my guess is that the dominant cause is late packet delivery rather than lost packets. It would be very interesting to know what kind of failure is causing the audio problem.

Note that in my case, where I believe I am seeing late packets, this would be caused by traffic congestion at one or more links. There are other users who have been discussing "buffer bloat" and this would also be categorized as a link (or router) congestion. If this is the case, then this XOR method could make things worst because the XOR method uses a 50% traffic increase. Generally, low computational work requires more traffic redundancy and low traffic redundancy requires more computing (and more delay). To summarize, if some of us are seeing network congestion problems, increasing traffic by 50% would be bad.

[gene96817]

It would be very helpful for Jamulus to keep loss packet and late packet statistics. We would learn a lot about what is disrupting the audio quality.

[ghost]

About 'buffer bloat': This also happens on the computer that Jamulus is running on to a very large extent.
Here is an old linux/unix command to see your transmit queue length: ifconfig (see txqueuelen in output)
Here is how to set it to 16 on eth0 (for the halibut :) sudo ifconfig eth0 txqueuelen 16
There are books and peer reviewed papers written about error correction and redundancy, thus, I expect it to be implemented properly.

[gene96817]

@DavidSavinkoff Yes, essentially every port in the data path has a transmit and receive buffer. There are a variety of conditions that contribute to buffering overflow (which is how I interpret the descriptive image of "buffer bloat"). In the client device, the transmit buffer can fill up if the application (e.g. Jamulus) puts frames in the buffer and the communications layer doesn't run often enough to transmit the buffers. These are symptoms that the client system is either overloaded or is having scheduling/dispatching problems.

No need to look at peer-reviewed papers. They are usually hard to read. Just look at a textbook. The explanations are easier to understand.

[corrados]

Jamulus is designed for low network traffic. As this XOR method will significantly increase the network traffic I would not like to support this.

Is it possible to switch this ON/OFF in HPSJam so that you can test how much better the audio quality is with using this XOR method?

[hselasky]

If you do it right, it just means 33% more data is needed. 1 of 3 packets is redundancy with HpsJAM.
No, XOR redundancy is the default with HpsJAM. It is built into the foundation of the protocol.

[gene96817]

I advise against using the XOR redundancy.
1- it increases the traffic, especially at the server
2- for large sessions (e.g. >50 participants) it represents significant traffic
3- Did someone specify the recovery algorithm when a packet is lost? How many packets does it take to recover from a random packet loss? (I am a bit busy right now. I think the recovery takes 5 -6 packets.) Lets make sure we know how much delay we are inserting into the flow. The analysis is simple for three packets if we have a marker for the first of the triplet. It is more complicated if a packet is lost (i.e. is the next packet the first of a triplet or something else).
(Sorry I don't have time to think this through this moment.)

[hselasky]

Recovering from more than 3 packets increases delay significantly and is not recommended.

[corrados]

I have the same opinion like gene96817. In April 2020, the server traffic on my virtual server was about 800 GB and my limit of the server at the time was 1 TB. So an increase of 33 % would have overloaded my server. Similar, at home I am running Jamulus with stereo/high quality which is around 900 kbps. My max. upstream rate is 1 Mbps. So, if Jamulus requires 33 % more bandwidth, I cannot use stereo/high quality anymore.

So, let's keep this method a unique feature of the HPSJam software.

[ann0see]

Can we close this as wontfix?