Upload recorded videos to S3

[okaycj]

Summary

Currently, Pip handles uploading video files to our S3 store. As we're replacing Pipe, this functionality will need to be replaced using the AWS S3 Javascript SDK.

Implementation

The S3 documentation recommends using the SDK over wrapping the REST API. Additionally, using S3's multipart upload has it's benefits and draw backs. The most notable draw back is that if we don't stop/abort an upload, we're charged for storing the parts of the upload, seemingly, indefinitely.

S3 might have a "timeout" setting that we could use to remove "abandoned" multipart uploads.

AC

• Replace upload placeholder with upload to S3
• Use S3's multipart upload
• Attempt to abort upload when browser is closed before end of experiment (BG created separate issue: Handle in-progress uploads when study ends unexpectedly #348)
• Script to check if there are uploads and determine if an upload has been "abandoned" (BG crossed off as being out of scope for this EFP issue)

[mekline]

This got moved into the "Done" column - is it actually done or still pending finishing the rest of the video upload pipeline?

[becky-gilbert]

I think we're done with the majority of this issue, which is the implementation and integration of the S3 class and successful uploading (i.e first two items acceptance criteria checklist). But I would like to leave this issue open because we still need to look at the last two items in the checklist.

Also, a question about this item:

Attempt to abort upload when browser is closed before end of experiment

When the browser tab/window is closed early, do we want to abort the upload or quickly try to complete it? Does that depend on whether there are any upload parts that are still in progress?

[becky-gilbert]

Status update:
I would like to keep this issue open so that we can address this task from the AC checklist at some point in the future.

Attempt to abort upload when browser is closed before end of experiment

I think the task below is also important but probably won't entail changes in the EFP repo, so I've crossed it off the AC list above (I don't know whether this script should be part of the lookit-api or elsewhere?)

Script to check if there are uploads and determine if an upload has been "abandoned"

Note: there are some remaining problems getting the actual S3 uploads to work on staging/production because of problems with the environment variables. But those problems are not in the scope of this EFP issue as they are addressed in separate issues:
lookit/lookit-api#1074
lookit/lookit-orchestrator#36