MPEGTS write the pcr in right way.

[winlinvip]

https://github.com/winlinvip/simple-rtmp-server/issues/311

[winlinvip]

A: For example, when you use a player, it uses pts (presentation timestamp) and dts (decoding timestamp) for display. However, there is a problem. If the encoding end and decoding end each use their own clock, after running for a while, they may become inconsistent, resulting in abnormal display. It may play too fast or too slow. To address this, you can insert pcr (program clock reference) into the stream. When decoding, you can adjust the clock of the decoding end to match the encoding end and display normally.

Q: Why would using their own clocks result in inconsistency? As long as both clocks are synchronized, for example, one is at 7:00 and the other is at 8:00, and dts and pts are relative times starting from 0, then after playing for 10 minutes, when dts becomes 600 seconds, as long as the two clocks do not accumulate faster or slower, there should be no problem, right? For example, one is at 7:10 and the other is at 8:10, what's the issue? Are you saying that the clocks on TVs and set-top boxes are not very accurate, so pcr calibration is needed? But the clocks on computers connected to the internet are accurate, so it's not necessary?

A: Yes, you are correct. If the clocks on both ends are synchronized and do not accumulate any discrepancies, there should be no issue. However, in practice, the clocks on devices like TVs and set-top boxes may not be very accurate and can drift over time. This is why pcr calibration is often used to ensure synchronization between the encoding and decoding ends. On the other hand, computers connected to the internet usually have more accurate clocks, so the need for pcr calibration may be less necessary in those cases.

A: Yes, as long as there is no accumulation of faster or slower timing, there should be no problem.

Q: What is this clock? Is it the system clock? Is it the clock returned by the gettimeofday(&now, NULL) API?

A: Yes, it is the gettimeofday() clock.

Q: The root cause of the problem is that this time drifts, right? Even if we use crystal oscillators to obtain the time, it can still become faster or slower. And even if we synchronize with a time server, there can still be errors, right? It seems that the time in my car easily drifts and does not match the time recorded by the 360 dashcam.

A: Yes, that's correct. It's easier to understand now. If your car is the encoding end and the 360 dashcam is the decoding end, there will be a mismatch in time, causing anomalies in the displayed time, such as playing too fast or too slow. To synchronize the time between your car and the 360 dashcam, you need something called a PCR (Program Clock Reference).

Q: Actually, DTS and PTS have the same meaning. It's just that both are needed because of P and B frames, so only one is mentioned, like DTS. Why can't DTS be associated with PCR? For example, why can't the encoder directly use the system clock gettimeofday as DTS? Then we wouldn't need PCR, right?

For instance, if the encoder is running slower, DTS starts from 0, and after 10 minutes, the encoder is only at 5 minutes. So, the DTS value would be 500 seconds. At the same time, the decoder has a normal clock, and its timeline has reached 10 minutes, requiring a video with a timestamp of 600 seconds. However, there is only 500 seconds available, so the decoder would be ahead. Couldn't this be corrected by simply adjusting the decoder's clock to be 500 seconds slower? If we use DTS = PCR, then we wouldn't need PCR. So, why do we still need PCR?

A: However, when you only use DTS, it's like having only one dimension. You cannot determine whether it is for decoding and displaying, or if the clocks on both sides are out of sync. Let's take an example: you mentioned that the encoder is running slower, but what if the encoder is running faster? The encoder reaches 10 minutes, but the normal time is only 5 minutes. In this case, should the decoder wait for another 5 minutes to display, or should it directly synchronize the clocks? If we use DTS both for display time and synchronization, the decoder itself would not be able to determine if its clock is the one with the issue, and it would become confused.

Q: If the encoder is running faster, for example, at a clock speed of 2x, which means the normal time should be 5 minutes, but its PCR reaches 10 minutes. What should its DTS be? Assuming it starts from 0, should it be 300 seconds or 600 seconds? If it is 600 seconds, then it would be the same as the PCR value. If it is 300 seconds, where does this 300 seconds come from?

A: The DTS should also be 600. So when the decoding end receives the PCR and sees that the PCR is 600, while its own DTS is only 300, it realizes that it is 300 units behind the encoder. It quickly adjusts itself to 600, so that it can follow the clock of the encoder.

Q: So, if we use relative time, which means the encoder sets the clock to zero when encoding starts, then the PCR and DTS should be the same, right? Because it is accurate when it looks at its own clock, the DTS and PCR are the same. Of course, the DTS also starts from zero.

A: If there is no PCR, and the decoding end receives a DTS of 600, but its own clock is only at 300, then the decoding end will be confused. Is my clock slow? Or should this frame of data be displayed after waiting for another 300? It cannot make a decision.

Q: Assuming the encoder samples the image device at a frequency of 25 frames per second, which means there is one frame every 40 milliseconds, so there are 25 video frames per second. In reality, for these 5 minutes (600 seconds), the encoder sets it to one frame every 40 milliseconds, 25 frames per second, resulting in 1500 frames for 600 seconds. However, the encoder actually outputs 750 frames, but with a DTS interval of 80 milliseconds, and the displayed values range from 0 to 600 seconds.

When the encoder decodes these 1500 seconds of video for the player, if the player follows its own correct clock, it will definitely be incorrect. Unless the player also adjusts its clock to match the encoder's. I understand this part.

Since DTS and PCR are calculated relatively and are consistent, meaning PCR = DTS, when the player sees this 600, it realizes that the correct value should be 300. For live streaming, it will know that the PCR is also 600, so it should adjust its own time accordingly. Right? It seems that PCR and DTS are the same, so why do we need PCR?

A: At the encoding end, it took 5 seconds to encode 5 frames of data. However, due to an issue with the encoding end's clock, the DTS values for these 5 frames should have been 0, 1, 2, 3, 4, but instead became 0, 2, 4, 6, 8. The decoding end receives 0, 2, 4, 6, 8. If the decoding end assumes there is no issue with the clock, it will decode the frames as follows: the first frame at 0 seconds, the second frame at 2 seconds, the third frame at 4 seconds, the fourth frame at 6 seconds, and the fifth frame at 8 seconds. However, as mentioned earlier, assuming the decoding end's clock is correct is not the reality. The clocks are out of sync between the encoding and decoding ends. How can the decoding end be made aware that there is a synchronization issue with the clock? Simply relying on the DTS values of 0, 2, 4, 6, 8 is not enough. There needs to be another way for the decoding end to know that the clocks are out of sync, and this is where PCR comes in. In order for the decoding end to decode and display correctly, it must adjust its own clock based on the PCR. Everything should be aligned with the encoding end (even if the encoding end is abnormal).

Q: With PCR, it is indeed possible to ensure normal playback even when there is a clock synchronization issue, even if the encoding end is abnormal. I understand this concept now and have a basic understanding of the purpose of PCR. To be honest, there is still one question that I haven't fully understood: why can't DTS be used as PCR? Why can't the sequence 0, 2, 4 indicate a clock issue? If we can derive PCR from DTS, then DTS should be able to replace PCR. Or in other words, why can't DTS replace PCR? Why are PCR and DTS unrelated? In the example you provided, how would PCR be indicated? What specific numbers would be used for PCR?

A: PCR is usually indicated before DTS. It can be in a separate packet or in the same packet as DTS, but the latter is more common. PCR is slightly smaller than DTS and has a frequency of 27,000. According to the standard, it should be inserted every 40 milliseconds. The value I just mentioned, which is slightly smaller than DTS, is not suitable because there are many PCR values.

A: If the playback speed increases, the actual interval is still 40 milliseconds, but the displayed value at the encoding end will not be 40 milliseconds.

Q: For a 25fps video, the normal DTS values would be 0 40 80 120 160. If the speed is doubled, it would be 0 80 160 240 320. The PCR values should also be 0 80 160 240 320, right? I understand now. This 25 frames at double speed is equivalent to PCR every 40 milliseconds, so DTS can be used to derive PCR. If they are not equal, for example, for a 10fps video with a 100-millisecond interval, the normal DTS values would be 0 100 200 300 400 500, and at double speed it would be 0 200 400 800 1000. The PCR insertion values would be 0 80 160 240 320. When PCR and DTS are mixed together, it would be 0 (0 80 160) 200 (240 320) 400 800 1000. Although DTS and PCR are related, PCR has much higher accuracy, right? So, besides DTS, PCR is also useful? If it's a 100fps video with a 10-millisecond interval, the accuracy of DTS would be even higher than PCR. In this case, wouldn't it be better to use DTS? However, high frame rates like this are not common. Is it just because of the accuracy issue, or are there other factors? It seems that PCR and DTS are still related, right?

A: It's not an accuracy issue. PCR is used to compare with the player's clock, while DTS is not uniform. When there are B frames, the PTS does not progress uniformly with DTS.

TRANS_BY_GPT3

[winlinvip]

In MPEG-DASH, it also states:

A.8 Encoder clock drift control
Over a longer operation time, a difference in clock accuracy of the encoder and decoder
may cause the playback to lag behind real-time or to interrupt temporarily due to the
client trying to access data faster than real-time.
For ISO base media file format based, clients may avoid these anomalies by using the
Producer Reference Time boxes as follows.

Haha, use PCR quickly...

TRANS_BY_GPT3

[winlinvip]

In summary:

1. When the clocks of the encoder and the player are not synchronized, PCR is needed for synchronization. For example, some boxes or devices without a time server may have inaccurate time, which can lead to inaccurate clocks for the player and the encoder. In an extreme case, if the encoder is running at double speed, a 5-minute video's DTS will show as 10 minutes. If the player plays this content based on the 10-minute timestamp, issues will arise, and vice versa.
2. DTS/PTS is used as the timestamp for content. Depending on the frame rate setting, the encoder may not uniformly apply timestamps, especially in cases of variable frame rates or B-frames. The TS standard inserts a PCR every 40 milliseconds, but there may not always be a usable DTS within that time frame. Therefore, having a separate PCR clock is necessary.
3. For HLS, a streaming format used on the internet, the clocks between devices are relatively accurate. Although the time may differ, the synchronization is consistent. Therefore, there is no need to use PCR, or PCR can be directly set as DTS without any issues, as there is no error. Some internet players may even ignore PCR altogether.

TRANS_BY_GPT3

[winlinvip]

PCR is mainly used in broadcasting and television to solve the clock drift problem of devices such as set-top boxes. However, mobile phones and other devices have NTP clock calibration, satellite timing, etc., so PCR is not needed.

TRANS_BY_GPT3