Support of growing MXF files

[donkon]

Hi!

We are trying to encode growing mxf files (recorded by our broadcast system) and written to some storage) to some streaming formats (mp4). The idea is to use bmxlib as a buffer (option --gf) and pipe the stream to ffmpeg. Basicly the following commandline works:

mkfifo my-pipe & (bmxtranswrap -t op1a --gf -o my-pipe growing-input.mxf | ffmpeg -y -i my-pipe -f mp4 out.mp4)

With this approach, 2 problems show up:

• If the growing-input.mxf is not growing (completely written), at the end following error messages come up (but encoding works):
  ERROR (libMXF): 'mxf_file_seek(_cFile, position, whence)' failed, at libMXF++/libMXF++/File.cpp:558
  ERROR: MXF exception caught: 'mxf_file_seek(_cFile, position, whence)' failed, at libMXF++/libMXF++/File.cpp:558
• If the file is growing (simulated by pv -L 1000000 complete-input.mxf > growing-input.mxf), bmxtranswrap complains
  Warning: Failed to read all partitions. File may be incomplete or invalid
  Warning: Input file is incomplete
  ERROR: 'num_read == len' check failed
  ERROR: near bmx/src/mxf_reader/EssenceReader.cpp:641
  Info: Duration: 62 (00:00:02:12 @25hz)
  Warning: Read fewer samples (62) than expected (200)
  So it seems that is it doesn't support growing files.

Any ideas? Any other approaches to find a solution?

[philipnbbc]

For the first problem you could try the --single-pass option to stop it trying to seek back and update the header metadata.

For the second problem, what rate is the file growing at? The bmxtranswrap growing file support was designed to support a file growing at a particular media rate (default realtime, i.e. the file's edit rate). It retries reads that fail, but if the data is arriving too slowly it will hit the retry count limit and error. E.g. if the growing file is a file transfer that is slower than realtime then bmxtranswrap will likely fail with the default settings. I haven't tried it but I assume -L 1000000 is 8Mb/s and therefore probably too low for typical broadcast video to simulate realtime? You can use the --rt, --gf-retries, --gf-delay and --gf-rate options to configure how bmxtranswrap follows the growing file progress.

[donkon]

Yes, "normally" the file is written in realtime, but it's not gauranteed, so for stability reasons I'm trying to figure out, what happens, when the files grows slowly.

If I set the realtime rate lower (eg. --rt 0.5), it works with my simulation (pv -L 5000000), but as soon as bmx is faster than the stream is written, the command fails (ERROR: 'num_read == len' check failed).

I tried
mkfifo my-pipe & (bmxtranswrap -t op1a --rt 0.1 --gf-delay 20 --gf-rate 0.1 --gf-retries 3 --single-pass --gf -o my-pipe growing-input.mxf | ffmpeg -y -i my-pipe -f mp4 out.mp4)

I'm wondering why I don't notice a delay of 20 seconds (--gf-delay 20) which should occur 3 times (--gf-retries 3)?

[philipnbbc]

For the applications I had in mind I didn't expect the delay to be that much - see the rt_sleep function that ignores values above 10 seconds. Try --gf-delay 5?

[donkon]

ok, I see the wait period when executing

mkfifo my-pipe & (bmxtranswrap --rt 1.0 --gf --gf-delay 5 --gf-retries 4 --single-pass -o my-pipe growing-input.mxf | ffmpeg -y -i my-pipe -f mp4 out.mp4)

but after 5 Seconds:

ERROR (libMXF): 'mxf_file_read(mxfFile, (uint8_t*)key, 16) == 16' failed, in libMXF/mxf/mxf_file.c:564
ERROR (libMXF): 'mxf_read_k(mxfFile, key)' failed, in libMXF/mxf/mxf_file.c:601
ERROR (libMXF): 'mxf_read_kl(_cFile, key, llen, len)' failed, at libMXF++/libMXF++/File.cpp:528
Warning: Failed to read all partitions. File may be incomplete or invalid
Warning: Input file is incomplete
ERROR: 'num_read == len' check failed
ERROR:     near bmx/src/mxf_reader/EssenceReader.cpp:641
ERROR: Single pass write failed because OP-1A container duration 36 does not equal set input duration 200
ERROR:     near bmx/src/mxf_op1a/OP1AFile.cpp:549
ERROR: BMX exception caught: Single pass write failed because OP-1A container duration 36 does not equal set input duration 200

[philipnbbc]

I reproduced it by running bmxtranswrap --rt 0.5 -o growing_input.mxf source.mxf to simulate the growing file at slower than realtime speed.

A quick fix that seemed to work is to change this line to if (mRequestSampleCounts[offset] == num_samples && mReadSampleCounts[offset] > 0) {. mReadSampleCounts[offset] was 0 which caused bmxtranswrap to exit because 0 samples were returned and there was no read error. I'm not sure this fix is correct (e.g. should it check for != num_samples?) but will take another look when I get some time.

[donkon]

would be great if you could have a deeper look, you could also provide a fix in a seperate branch and we could test it in more detail

[philipnbbc]

I've opened #90 with changes following #87 (reply in thread).

[donkon]

Thank, we will have a look

[donkon]

I tried #90 and it looks good to me, I give you final feedback in the PR