Request.post() stores data in temp files

[popravich]

Long story short

web_reqrep.Request.post() reads whole payload into memory and than stores uploads into temp files — implicitly by using cgi.FieldStorage.

Expected behaviour

Either no temporary files or explicit behavior.

Actual behaviour

cgi.FieldStorage creates temporary file for each upload (file).
The FieldStorage docstring states the following:

    The class is subclassable, mostly for the purpose of overriding
    the make_file() method, which is called internally to come up with
    a file open for reading and writing.  This makes it possible to
    override the default choice of storing all files in a temporary
    directory and unlinking them as soon as they have been opened.

So it is possible to control how uploads are handled — either stored into temp files or any other way...

PS: python documentation is missing FieldStorage description.

[asvetlov]

I believe getting rid of temp files is the proper solution: we already have req.multipart() coroutine for handling endless data streams.

@popravich would you create a Pull Request?

[popravich]

Will take a look this weekend

[fafhrd91]

@popravich did you look into this issue?

[popravich]

Sorry, had no time yet

[fafhrd91]

I added new .post() implementation to #1664, it stores data in temp files without loading whole payload into memory (our multipart implementation is better than cgi)

I do not think we should remove post() in favor of multipart(), ergonomics of post() is much better.
but we can combine post() with client_max_size attribute and raise exception if payload exceeds it

[popravich]

Ouch, now imagine containerized application running with readonly filesystem...
I thought it should be possible to control this behavior.

[fafhrd91]

Good point, I'll add config for that.
But i don't think it is a problem, developer should know what he is doing. .post() is convinience.

[fafhrd91]

on other hand it is same behavior as in 1.x versions

[fafhrd91]

added client_max_size support. but I think we should leave post in-place.
.post() is for simplicity, .multipart() is for streaming

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.