subprocess.Popen not closed when using subclip

[gyglim]

Hi

I have another issue with system calls that remain open, eventually leading to an OSError: [Errno 24] Too many open files.
It happens when using subclip. This is the call trace, leading to the call which is not closed:

/usr/local/lib/python2.7/dist-packages/moviepy-0.2.2.13-py2.7.egg/moviepy/video/io/ffmpeg_reader.py, line 97 in function initialize.
/usr/local/lib/python2.7/dist-packages/moviepy-0.2.2.13-py2.7.egg/moviepy/video/io/ffmpeg_reader.py, line 178 in function get_frame.
/usr/local/lib/python2.7/dist-packages/moviepy-0.2.2.13-py2.7.egg/moviepy/video/io/VideoFileClip.py, line 99 in function <lambda>.
/usr/local/lib/python2.7/dist-packages/moviepy-0.2.2.13-py2.7.egg/moviepy/Clip.py, line 95 in function get_frame.
/usr/local/lib/python2.7/dist-packages/moviepy-0.2.2.13-py2.7.egg/moviepy/decorators.py, line 89 in function wrapper.
<decorator-gen-14>, line 2 in function get_frame.
/usr/local/lib/python2.7/dist-packages/moviepy-0.2.2.13-py2.7.egg/moviepy/Clip.py, line 188 in function <lambda>.
/usr/local/lib/python2.7/dist-packages/moviepy-0.2.2.13-py2.7.egg/moviepy/Clip.py, line 136 in function <lambda>.
/usr/local/lib/python2.7/dist-packages/moviepy-0.2.2.13-py2.7.egg/moviepy/Clip.py, line 95 in function get_frame.
/usr/local/lib/python2.7/dist-packages/moviepy-0.2.2.13-py2.7.egg/moviepy/decorators.py, line 89 in function wrapper.
<decorator-gen-14>, line 2 in function get_frame.
/usr/local/lib/python2.7/dist-packages/moviepy-0.2.2.13-py2.7.egg/moviepy/video/VideoClip.py, line 659 in function set_make_frame.
/usr/local/lib/python2.7/dist-packages/moviepy-0.2.2.13-py2.7.egg/moviepy/decorators.py, line 14 in function outplace.
<decorator-gen-57>, line 2 in function set_make_frame.
/usr/local/lib/python2.7/dist-packages/moviepy-0.2.2.13-py2.7.egg/moviepy/Clip.py, line 136 in function fl.
/usr/local/lib/python2.7/dist-packages/moviepy-0.2.2.13-py2.7.egg/moviepy/Clip.py, line 189 in function fl_time.
/usr/local/lib/python2.7/dist-packages/moviepy-0.2.2.13-py2.7.egg/moviepy/Clip.py, line 391 in function subclip.
/usr/local/lib/python2.7/dist-packages/moviepy-0.2.2.13-py2.7.egg/moviepy/decorators.py, line 41 in function apply_to_audio.
<decorator-gen-37>, line 2 in function subclip.
/usr/local/lib/python2.7/dist-packages/moviepy-0.2.2.13-py2.7.egg/moviepy/decorators.py, line 29 in function apply_to_mask.
<decorator-gen-38>, line 2 in function subclip.
/usr/local/lib/python2.7/dist-packages/moviepy-0.2.2.13-py2.7.egg/moviepy/decorators.py, line 89 in function wrapper.
<decorator-gen-39>, line 2 in function subclip.

Relates to these:
#501
#255
#57

@Zulko Any suggestion as to here I should go looking for a missing del or close?

Regards,
Michael

[Zulko]

Can you provide a minimal code to reproduce the effect ?
Do you have this error even with PR #501 fixing the audio ?
How many subclips are you using at the same time ?

[gyglim]

Yes, it's with the audio fix.
I will try to create a minimal example to reproduce it, sure.

I am actually just using one subclip at a time. It's just a function that is called over an over again, thus after sufficient calls leading to an error.

[gyglim]

@Zulko
There you go:

from moviepy.editor import VideoFileClip

def iterate_subclip():
    video = VideoFileClip('media/big_buck_bunny_432_433.webm')
    subclip = video.subclip(0.5, 1)

    for _ in subclip.iter_frames():
        break

    del subclip
    del video

if __name__=='__main__':
    for idx in range(10000):
        print idx
        iterate_subclip()

On my local development machine I get a OSError: [Errno 12] Cannot allocate memory instead, but the cause is the same: Popen calls that are not closed (verified by using https://gist.github.com/gyglim/340c9d276d7f53578eb9673156e556df)

[ghost]

@gyglim I've ran your code above, and after 1,000 loops, it still seems stable (I'm running the program top, and I see the number of processes fluxuate between 220 and 250, but always comes back to 220 after a minute or so. Can you double check that you still get errors with this code, with the current repo version?
I'll let the program run until it reaches 10,000 but I'm not sure I will duplicate your results.

[ghost]

@gyglim I let this run to completion, and I do not get the error you reported. I will admit that memory usage got somewhat high, and at times we have 20 or so ffmpeg processes open at a time.

[gyglim]

Ok, I will have another look. I believe the issue comes from using wait() in
https://github.com/Zulko/moviepy/blob/master/moviepy/video/io/ffmpeg_reader.py#L188

From the subprocess documentation:

Popen.wait)
Wait for child process to terminate. Set and return returncode attribute.

Note This will deadlock when using stdout=PIPE or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use Popen.communicate() when using pipes to avoid that.

It is my understanding that the buffer can run full when using subclip and iter_frames, as not all frames are read from the buffer in this case. Correct me if I am wrong @Zulko.

Anyways, we changed to communicate() internally and it helped. I will verify again and then make a PR, once I get around to it

[keikoro]

Closing this as it's a very old issue without any new activity in several years.

Feel free to reopen it – or better yet, open a new issue following our issue template and referencing this issue – if the problem persists with the latest recommended version of MoviePy.