Support for POST files and Upload

[r1m]

Is there any support for uploading files with POST requests ?

Bonus question : could we update the device with a manual firmware upload ?

[Vity01]

I just wanted to create new thread for this with a same RFE.
@ram-one There is only HttpClient class which you can use for uploading really small files.
See HttpClient->setPostBody and FileSystem->fileGetContent. See related sample... Note that it's not suitable for bigger files, because it means to load whole file content into memory (+ it's using MemoryDataStream internally).
BUT for big files (eg. >10kB) there is still no support. There should be some new function aka
HttpClient->setPostBody(new FileStream("my file")) which would allow to upload bigger file.

Also you can also use FTP server and download data from external server, if this usecase fits your needs (not for mine, since my AP is not working 24/7).

[r1m]

What I meant is uploading TO the ESP.
Httpclient class is for sending a request.

I used code from PR #499 to be able to read the incoming post request body, but everything goes to the memory. It's not suitable for large files.

[Vity01]

So there are 2 use cases:

1. Sming HttpServer - receiving large files
2. Simple "ajax" request (using HttpClient) for sending large files
   both needs enhancement

[hreintke]

@ram-one @Vity01 :
I agree that the option for sending a file with HTTP Post would be a viable option.
If you have a proposal/initial design for that I would be happy to discuss that.

With current functionality there is probably a workaround.
For that see the rboothttpupdate class which receives a full file, now sending it to flash but creating a similar class which sends to file should not be to difficult,
If you trigger this by sending a http_post with the "url of the file to be loaded" you have a "poor mans http post file"

[r1m]

This workaround has a big drawback : It needs a webserver on the other side.

[hreintke]

@ram-one :
I know, but it is available now without major additions to sming.
If this is not an option you either have to create a PR (wiling to discuss the design) or wait until someone else does.

[patrickjahns]

I would be interested in this too - but for a different use case scenario: enabling OTA Updates via http post. This way people do not need a webserver of their own and request would never need to leave their own network

[patrickjahns]

@hreintke

I had a look at the current http server implementation and this is what I found so far/ideas for changes:

HttpRequest::parseHeader needs to be extended to properly detect multipart headers (multipart/form-data) - also extract the boundary from there to detect the parts

HttpServerConnection::onReceive needs to be extended to check for multipart and set a flag i.e. FILE_UPLOAD_START/FILE_UPLOAD_INPROGRESS/FILE_UPLOAD_FINISHED

But what I did not quite understand from the current design is, if we can receive small packages and then write them to filesystem (form the onReceive method i guess?)

[hreintke]

@patrickjahns :
Great if you can work on the implementation of this feature. Would be a good extension and useful for many applications.
You are looking at the right spot to implement. But I think the updates can be limited to the parseHeader function. The httpserver:onreceive does already call the parseHeader for every tcp fragment it receives when the header is not yet complete.

    if (state == eHCS_Ready)
    {
        HttpParseResult res = request.parseHeader(server, buf);
        if (res == eHPR_Wait)
         .......
        .......

state being initialized as eHCS_Ready at instantiaton.of the httpserverconnection object.

For saving the data you should use a stream, see DataSourceStream.h. That would make it possible to write to a file (FileStream) or when using for OTA a FlashStream (to be made).

UPDATE :
We are moving Sming framework from NONOS SDK to RTOS sdk. The SmingRTOS is in beta state and already contains all functionality of the NONOS version. I prefer to have larger improvements done in the RTOS version as support for adding new to NONOS will cease shortly.
There is no difference in the http-request/server classes so your work up till now is still valid.

[patrickjahns]

@hreintke

Thank you for the input - I had a closer look at the implementation just now.

Here is an example multipart request:

POST /update HTTP/1.1
Host: 192.168.2.109
Connection: keep-alive
Content-Length: 2690
Pragma: no-cache
Cache-Control: no-cache
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://192.168.2.109
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.47 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryA4qie7aqfuYPyKbB
DNT: 1
Referer: http://192.168.2.109/test.html
Accept-Encoding: gzip, deflate
Accept-Language: en-GB,en;q=0.8,en-US;q=0.6,de;q=0.4

------WebKitFormBoundaryA4qie7aqfuYPyKbB
Content-Disposition: form-data; name="update"; filename="README.md"/r/n
Content-Type: application/octet-stream/r/n
content /r/n
------WebKitFormBoundaryA4qie7aqfuYPyKbB--/r/n

My approach in the testcode is to check in parseHeader if the request is multipart - if yes, set a flag. Futurewise we can still call parsePostData
From there on it is straight forward - checking inside parsePostData if the POST is actually MultiPart - search for the boundary beginning, extract Content-Disposition - if it has filename write the following bytes to file until we arrive at next boundary beginning / or at --+boundary+-- (which marks the end of the multipart

Since this ties directly to the way we parse form data - would it be possible to extend the HttpServer Interface so the when defining the Path via server.addPath("/"upload, onUpload); - we can can hook up in the parser somehow?

Otherwise I am a bit lost when extending the post to upload a new firmware. (I wouldn`t want to write to spiffs first and then move the rom)

[hreintke]

@patrickjahns
I don't understand what/how you want to achieve with the addpath extension.
Are you "asking confirmation to allow download" of application ?

For extending to post firmware. There is a base class IDataSourceStream, and a derived class Filestream. My idea is that we add a "write functionality" to the filestream class (like the memorydatastream has) . That provides the way to get the posted file on spiffs. When also adding a derived Flashstream class, which writes directly to flash (no spiffs) we have the same interface for use in the http classes.

PS : see the update in the remark above.

[patrickjahns]

Copying some information from gitter to keep track of it:

I will take the action for adding the functionality to the filestream class.
For your implementation just do :

FileStream uploadFile;
bool uploadFile.attach(String file) -> false if fails.
int uploadFile.write(char*,size) -> will always append to the end of the file, returns number of written chars.

Will be available in SmingRTOS shortly.
Passwords -> http basic auth is the way to go.
On addPath : You are right about URI.
On callbacks, indeed it is up to the server to see whether a POST is valid for a URI.
But if the situation is that we by saying with addpath -> uploads OK. Then anyone can just post to that URL and upload a file.
That is what I tried to say in the "prevent abuse" But the Httprequest is not yet complete, so the callback(httprequest,httpresponse) cannot be handled at that moment.

Also interesting for HTTP Multipart is RFC2388

[patrickjahns]

Closed and move discussion to #694