5Gb limit

[esraneufeld]

Problem:

AWS S3 has a 5 GB limit for a sinlge put operation. The single pre-signed link strategy fails. This affects:

• Uploads through filepicker
• Uploads through nodeports
• Uploads from comp. backend

Remedy:

• rclone instad of nodeports?
• temporary S3 credentials for uplaods into specific subfolder of the bucket?
• multiple upload links?

Baklava

Preview Give feedback

1. Computational backend: Use S3 Envs for AWS clusters that live inside the simcore stack osparc-simcore#4640
   1 of 1
   a:dask-service +1
   
2. computational backend shall make use of multipart links for external clusters osparc-simcore#3217
   a:dask-service a:director-v2 t:maintenance +3
   

[newton1985]

Does "uploads from comp. backend" also cover the case of saving work.zip to S3 when a study is closed? Afaik, this is also affected.

[sanderegg]

Update on sprint Macarons

Done

• Creation of S3 links instead of single presigned links by the storage service: ✨Storage can return a S3 link instead of a presigned link on demand osparc-simcore#3004
• Computational sidecar understands S3 links: ✨ Use S3 links in internal computational backend cluster, prepare for temporary tokens (⚠️ devops) osparc-simcore#3006

Ongoing

• Computational sidecars in internal cluster use S3 links: ✨ Use S3 links in internal computational backend cluster, prepare for temporary tokens (⚠️ devops) osparc-simcore#3006
• Modification of dynamic sidecar to use RClone library to synchronise state files and inputs/outputs using direct S3 access: ✨♻️ (⚠️ devops) Remove 5gb limit when uploading data via nodeports osparc-simcore#2993

Open

• Creation of temporary S3 credentials per user (limited to 1000 concurrent temp credentials by AWS)
• Investigate multipart upload with presigned links (for frontend, and potentially for computational sidecars in external clusters)
• Implement solution for frontend/external computational sidecars

[sanderegg]

Update on sprint Croissant

Done

• Computational sidecars in internal cluster use S3 links: ✨ Use S3 links in internal computational backend cluster, prepare for temporary tokens (⚠️ devops) osparc-simcore#3006
• Modification of dynamic sidecar to use RClone library to synchronise state files and inputs/outputs using direct S3 access: ✨♻️ (⚠️ devops) Remove 5gb limit when uploading data via nodeports osparc-simcore#2993
• Creation of temporary S3 credentials per user (limited to 1000 concurrent temp credentials by AWS)
• Investigate multipart upload with presigned links (for frontend, and potentially for computational sidecars in external clusters)
• ♻️ Enhancement/storage refactoring part 1 osparc-simcore#3077

Ongoing

Implement multipart upload links to break 5Gb limit:

• ✨ Storage: allow creation of multipart presigned upload links osparc-simcore#3021 (FilePicker)
  • First tests allowed to upload >10Gb file over the file picker
  • Backup solution for dynamic sidecar if RClone solution is too unstable

Open

• Implement solution for frontend/external computational sidecars
• Implement solution for public API

[sanderegg]

Update on sprint Diolkos

Done

• Enhancement/storage refactoring step2 osparc-simcore#3100
• ✨♻️ Storage refactoring step3 osparc-simcore#3144

Ongoing

• ✨ Storage: allow creation of multipart presigned upload links osparc-simcore#3021

[sanderegg]

Update on sprint Meteora

Done

• ✨ Storage: allow creation of multipart presigned upload links osparc-simcore#3021
• api-server: Use multipart upload  osparc-simcore#3166

Ongoing

• WIP: Enhancement/stream_upload osparc-simcore#3224

Open

• computational backend shall make use of multipart links for external clusters osparc-simcore#3217

[esraneufeld]

todo

• fix bug in API that limits to 2GB

[pcrespov]

Update on sprint Brutalism

No progress this sprint on this issue

Open

• fix bug in API that limits to 2GB
• computational backend shall make use of multipart links for external clusters osparc-simcore#3217

[sanderegg]

Update on sprint Vaporwave

Done

• API: add interface in webserver for tokens with expiration #705

Ongoing

• computational backend shall make use of multipart links for external clusters osparc-simcore#3217
• fix bug that limits to 2GB (workaround exists)

Open

• WIP: Enhancement/stream_upload osparc-simcore#3224 Abandonned in favor of api-server: uploaded files are temporarily located in the server osparc-simcore#3223, API: add entrypoint for multipart in api-server osparc-simcore#4540

[esraneufeld]

@sanderegg
dear sylvain. is there an update on this one?

[sanderegg]

@sanderegg dear sylvain. is there an update on this one?

@esraneufeld no there isn't we decided in PM1 to concentrate on other priorities, since this one has a workaround and that we concentrate on dynamic services at the moment.

[newton1985]

@esraneufeld - just to add some detail, the workaround is to use the "curl" command's "put" feature. It's a substitution of 1 or 2 lines of code, and seems to work just fine without any drawbacks. Though at some point it would be nice to fix it, since the average user won't know the trick.

[matusdrobuliak66]

@matusdrobuliak66 Investigate options AWS temporary credientials for S3

[sanderegg]

Current status is that presigned link are created with a very long valid time. this is not optimal. Best would be to use the osparc API directly, therefore we would not rely on a hard-coded time value. it would also be more secure.