From 1c6c7b66f1928d0daa0cb771081942511de0d77e Mon Sep 17 00:00:00 2001 From: Johannes Maron Date: Sat, 19 Mar 2022 12:46:51 +0100 Subject: [PATCH] Use Markdown readme instead or RST to use native mermaid diagrams --- Makefile | 2 - README.md | 226 +++++++++++++++++++++++++++++++++++++ README.rst | 217 ----------------------------------- http-message-flow.sequence | 8 -- http-message-flow.svg | 1 - pyproject.toml | 2 +- 6 files changed, 227 insertions(+), 229 deletions(-) delete mode 100644 Makefile create mode 100644 README.md delete mode 100644 README.rst delete mode 100644 http-message-flow.sequence delete mode 100644 http-message-flow.svg diff --git a/Makefile b/Makefile deleted file mode 100644 index 07fd945..0000000 --- a/Makefile +++ /dev/null @@ -1,2 +0,0 @@ -http-message-flow.svg: http-message-flow.sequence - diagrams sequence http-message-flow.sequence http-message-flow.svg diff --git a/README.md b/README.md new file mode 100644 index 0000000..01b89be --- /dev/null +++ b/README.md @@ -0,0 +1,226 @@ +# django-s3file + +A lightweight file upload input for Django and Amazon S3. + +Django-S3File allows you to upload files directly AWS S3 effectively +bypassing your application server. This allows you to avoid long running +requests from large file uploads. This is particularly helpful for if +you run your service on AWS Lambda or Heroku where you have a hard +request limit. + +[![PyPi +Version](https://img.shields.io/pypi/v/django-s3file.svg)](https://pypi.python.org/pypi/django-s3file/) +[![Test +Coverage](https://codecov.io/gh/codingjoe/django-s3file/branch/master/graph/badge.svg)](https://codecov.io/gh/codingjoe/django-s3file) +[![GitHub +license](https://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/codingjoe/django-s3file/master/LICENSE) + +## Features + +- lightweight: less 200 lines +- no JavaScript or Python dependencies (no jQuery) +- easy integration +- works just like the built-in +- extendable JavaScript API + +## For the Nerds + +```mermaid +sequenceDiagram + autonumber + actor Browser + Browser->>S3: POST large file + activate S3 + S3->>Browser: RESPONSE AWS S3 key + Browser->>Middleware: POST AWS S3 key + activate Middleware + Middleware->>S3: GET AWS S3 key + S3->>Middleware: RESPONSE large file promise + deactivate S3 + Middleware->>Django: request incl. large file promise + deactivate Middleware + activate Django + opt only if files is procssed by Django + Django-->>S3: GET large file + activate S3 + S3-->>Django: RESPONSE large file + deactivate S3 + end + Django->>Browser: RESPONSE success + deactivate Django +``` + +## Installation + +Make sure you have [Amazon S3 +storage](http://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html) +setup correctly. + +Just install S3file using `pip`. + +```bash +pip install django-s3file +# or +pipenv install django-s3file +``` + +Add the S3File app and middleware in your settings: + +```python +# settings.py + +INSTALLED_APPS = ( + '...', + 's3file', + '...', +) + +MIDDLEWARE = ( + '...', + 's3file.middleware.S3FileMiddleware', + '...', +) +``` + +## Usage + +S3File automatically replaces Django's `ClearableFileInput` widget, you +do not need to alter your code at all. + +The `ClearableFileInput` widget is only than automatically replaced when +the `DEFAULT_FILE_STORAGE` setting is set to `django-storages`' +`S3Boto3Storage` or the dummy `FileSystemStorage` is enabled. + +### Setting up the AWS S3 bucket + +#### Upload folder + +S3File uploads to a single folder. Files are later moved by Django when +they are saved to the `upload_to` location. + +It is recommended to [setup +expiration](http://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html) +for that folder, to ensure that old and unused file uploads don't add up +and produce costs. + +The default folder name is: `tmp/s3file` You can change it by changing +the `S3FILE_UPLOAD_PATH` setting. + +#### CORS policy + +You will need to allow `POST` from all origins. Just add the following +to your CORS policy. + +```json +[ + { + "AllowedHeaders": [ + "*" + ], + "AllowedMethods": [ + "POST" + ], + "AllowedOrigins": [ + "*" + ], + "ExposeHeaders": [], + "MaxAgeSeconds": 3000 + } +] +``` + +### Progress Bar + +S3File does emit progress signals that can be used to display some kind +of progress bar. Signals named `progress` are emitted for both each +individual file input as well as for the form as a whole. + +The progress signal carries the following details: + +```javascript +console.log(event.detail) + +{ + progress: 0.4725307607171312 // total upload progress of either a form or single input + loaded: 1048576 // total upload progress of either a form or single input + total: 2219064 // total bytes to upload + currentFile: File {…} // file object + currentFileName: "text.txt" // file name of the file currently uploaded + currentFileProgress: 0.47227834703299176 // upload progress of that file + originalEvent: ProgressEvent {…} // the original XHR onprogress event +} +``` + +The following example implements a Boostrap progress bar for upload +progress of an entire form. + +```html +
+
0%
+
+``` + +```javascript +(function () { + var form = document.getElementsByTagName('form')[0] + var progressBar = document.getElementsByClassName('progress-bar')[0] + + form.addEventListener('progress', function (event) { + // event.detail.progress is a value between 0 and 1 + var percent = Math.round(event.detail.progress * 100) + + progressBar.setAttribute('style', 'width:' + percent + '%') + progressBar.setAttribute('aria-valuenow', percent) + progressBar.innerText = percent + '%' + }) +})() +``` + +### Using S3File in development + +Using S3File in development can be helpful especially if you want to use +the progress signals described above. Therefore, S3File comes with a AWS +S3 dummy backend. It behaves similar to the real S3 storage backend. It +is automatically enabled, if the `DEFAULT_FILE_STORAGE` setting is set +to `FileSystemStorage`. + +To prevent users from accidentally using the `FileSystemStorage` and the +insecure S3 dummy backend in production, there is also an additional +deployment check that will error if you run Django\'s deployment check +suite: + +```shell +python manage.py check --deploy +``` + +We recommend always running the deployment check suite as part of your +deployment pipeline. + +### Uploading multiple files + +Django does have limited support for [uploading multiple +files](https://docs.djangoproject.com/en/stable/topics/http/file-uploads/#uploading-multiple-files). +S3File fully supports this feature. The custom middleware makes ensure +that files are accessible via `request.FILES`, even though they have +been uploaded to AWS S3 directly and not to your Django application +server. + +### Using optimized S3Boto3Storage + +Since `S3Boto3Storage` supports storing data from any other fileobj, it +uses a generalized `_save` function. This leads to the frontend +uploading the file to S3 and then copying it byte-by-byte to perform a +move operation just to rename the uploaded object. For large files this +leads to additional loading times for the user. + +That\'s why S3File provides an optimized version of this method at +`storages_optimized.S3OptimizedUploadStorage`. It uses the more +efficient `copy` method from S3, given that we know that we only copy +from one S3 location to another. + +```python +from s3file.storages_optimized import S3OptimizedUploadStorage + +class MyStorage(S3OptimizedUploadStorage): # Subclass and use like any other storage + default_acl = 'private' +``` diff --git a/README.rst b/README.rst deleted file mode 100644 index d2d2211..0000000 --- a/README.rst +++ /dev/null @@ -1,217 +0,0 @@ -============= -django-s3file -============= - -A lightweight file upload input for Django and Amazon S3. - -Django-S3File allows you to upload files directly AWS S3 effectively -bypassing your application server. This allows you to avoid long running -requests from large file uploads. This is particularly helpful for if -you run your service on AWS Lambda or Heroku where you have a hard request -limit. - -|PyPi Version| |Test Coverage| |GitHub license| - --------- -Features --------- - -- lightweight: less 200 lines -- no JavaScript or Python dependencies (no jQuery) -- easy integration -- works just like the built-in -- extendable JavaScript API - -------------- -For the Nerds -------------- - -.. image:: http-message-flow.svg - ------------- -Installation ------------- - -Make sure you have `Amazon S3 storage`_ setup correctly. - -Just install S3file using ``pip``. - -.. code:: bash - - pip install django-s3file - # or - pipenv install django-s3file - -Add the S3File app and middleware in your settings: - -.. code:: python - - - INSTALLED_APPS = ( - '...', - 's3file', - '...', - ) - - MIDDLEWARE = ( - '...', - 's3file.middleware.S3FileMiddleware', - '...', - ) - ------ -Usage ------ - -S3File automatically replaces Django’s ``ClearableFileInput`` widget, -you do not need to alter your code at all. - -The ``ClearableFileInput`` widget is only than automatically replaced -when the ``DEFAULT_FILE_STORAGE`` setting is set to -``django-storages``\ ’ ``S3Boto3Storage`` or the dummy ``FileSystemStorage`` -is enabled. - -Setting up the AWS S3 bucket ----------------------------- - -Upload folder -~~~~~~~~~~~~~ - -S3File uploads to a single folder. Files are later moved by Django when -they are saved to the ``upload_to`` location. - -It is recommended to `setup expiration`_ for that folder, to ensure that -old and unused file uploads don’t add up and produce costs. - -The default folder name is: ``tmp/s3file`` You can change it by changing -the ``S3FILE_UPLOAD_PATH`` setting. - -CORS policy -~~~~~~~~~~~ - -You will need to allow ``POST`` from all origins. Just add the following -to your CORS policy. - -.. code:: json - - [ - { - "AllowedHeaders": [ - "*" - ], - "AllowedMethods": [ - "POST" - ], - "AllowedOrigins": [ - "*" - ], - "ExposeHeaders": [], - "MaxAgeSeconds": 3000 - } - ] - -Progress Bar ------------- - -S3File does emit progress signals that can be used to display some kind of progress bar. -Signals named ``progress`` are emitted for both each individual file input as well as -for the form as a whole. - -The progress signal carries the following details: - -.. code:: javascript - - console.log(event.detail) - - { - progress: 0.4725307607171312 // total upload progress of either a form or single input - loaded: 1048576 // total upload progress of either a form or single input - total: 2219064 // total bytes to upload - currentFile: File {…} // file object - currentFileName: "text.txt" // file name of the file currently uploaded - currentFileProgress: 0.47227834703299176 // upload progress of that file - originalEvent: ProgressEvent {…} // the original XHR onprogress event - } - - -The following example implements a Boostrap progress bar for upload progress of an -entire form. - -.. code:: html - -
-
0%
-
- -.. code:: javascript - - (function () { - var form = document.getElementsByTagName('form')[0] - var progressBar = document.getElementsByClassName('progress-bar')[0] - - form.addEventListener('progress', function (event) { - // event.detail.progress is a value between 0 and 1 - var percent = Math.round(event.detail.progress * 100) - - progressBar.setAttribute('style', 'width:' + percent + '%') - progressBar.setAttribute('aria-valuenow', percent) - progressBar.innerText = percent + '%' - }) - })() - - -Using S3File in development ---------------------------- - -Using S3File in development can be helpful especially if you want to use the progress -signals described above. Therefore, S3File comes with a AWS S3 dummy backend. -It behaves similar to the real S3 storage backend. It is automatically enabled, if the -``DEFAULT_FILE_STORAGE`` setting is set to ``FileSystemStorage``. - -To prevent users from accidentally using the ``FileSystemStorage`` and the insecure S3 -dummy backend in production, there is also an additional deployment check that will -error if you run Django's deployment check suite:: - - python manage.py check --deploy - -We recommend always running the deployment check suite as part of your deployment -pipeline. - -Uploading multiple files ------------------------- - -Django does have limited support for `uploading multiple files`_. S3File -fully supports this feature. The custom middleware makes ensure that -files are accessible via ``request.FILES``, even though they have been -uploaded to AWS S3 directly and not to your Django application server. - -.. _Amazon S3 storage: http://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html -.. _setup expiration: http://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html -.. _uploading multiple files: https://docs.djangoproject.com/en/stable/topics/http/file-uploads/#uploading-multiple-files - -.. |PyPi Version| image:: https://img.shields.io/pypi/v/django-s3file.svg - :target: https://pypi.python.org/pypi/django-s3file/ -.. |Test Coverage| image:: https://codecov.io/gh/codingjoe/django-s3file/branch/master/graph/badge.svg - :target: https://codecov.io/gh/codingjoe/django-s3file -.. |GitHub license| image:: https://img.shields.io/badge/license-MIT-blue.svg - :target: https://raw.githubusercontent.com/codingjoe/django-s3file/master/LICENSE - -Using optimized S3Boto3Storage ------------------------------- - -Since ``S3Boto3Storage`` supports storing data from any other fileobj, -it uses a generalized ``_save`` function. This leads to the frontend uploading -the file to S3 and then copying it byte-by-byte to perform a move operation just -to rename the uploaded object. For large files this leads to additional loading -times for the user. - -That's why S3File provides an optimized version of this method at -``storages_optimized.S3OptimizedUploadStorage``. It uses the more efficient -``copy`` method from S3, given that we know that we only copy from one S3 location to another. - -.. code:: python - - from s3file.storages_optimized import S3OptimizedUploadStorage - - class MyStorage(S3OptimizedUploadStorage): # Subclass and use like any other storage - default_acl = 'private' diff --git a/http-message-flow.sequence b/http-message-flow.sequence deleted file mode 100644 index 1365113..0000000 --- a/http-message-flow.sequence +++ /dev/null @@ -1,8 +0,0 @@ -Client->AWS S3:POST large file -AWS S3->Client:RESPONSE AWS S3 key -Client->S3FileMiddleware:POST AWS S3 key -S3FileMiddleware->AWS S3:GET AWS S3 key -AWS S3->S3FileMiddleware:RESPONSE large file -S3FileMiddleware->DjangoView:request incl. large file -DjangoView->Client:RESPONSE success - diff --git a/http-message-flow.svg b/http-message-flow.svg deleted file mode 100644 index 4914b4e..0000000 --- a/http-message-flow.svg +++ /dev/null @@ -1 +0,0 @@ -Created with Raphaël 2.1.4ClientClientAWS S3AWS S3S3FileMiddlewareS3FileMiddlewareDjangoViewDjangoViewPOST large fileRESPONSE AWS S3 keyPOST AWS S3 keyGET AWS S3 keyRESPONSE large filerequest incl. large fileRESPONSE success \ No newline at end of file diff --git a/pyproject.toml b/pyproject.toml index 9be4818..ada07ed 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -7,7 +7,7 @@ name = "django-s3file" authors = [ { name = "Johannes Maron", email = "johannes@maron.family" } ] -readme = "README.rst" +readme = "README.md" license = { file = "LICENSE" } dynamic = ["version", "description"] classifiers = [