[DigitalOcean] Private objects signing does not work with custom domain

[lifenautjoe]

When using a custom domain on your DigitalOcean space, such as cdn.peekalink.com, enabled by setting AWS_S3_CUSTOM_DOMAIN=cdn.peekalink.com, when having a storage with the property default_acl='private', no signature is emitted on the object url and it cannot be accessed.

This is because on the S3BotoStorage, the url method checks whether theres a custom domain, and if there is, it tries to sign the object url with the CloudFront Signer, thing which obviously does not work for DigitalOcean Spaces.

       # On s3boto.py line 564
        if self.custom_domain:
            url = "{}//{}/{}".format(
                self.url_protocol, self.custom_domain, filepath_to_uri(name))

            if self.querystring_auth and self.cloudfront_signer:
                expiration = datetime.utcnow() + timedelta(seconds=expire)

                return self.cloudfront_signer.generate_presigned_url(url, date_less_than=expiration)

            return url

I'm trying to figure out how to best do this but the most sensible thing probably to do is to create a digitalocean_signer that handles this edge case and can be set on an storage digitalocean_signer attribute.

Opening this issue so that there's some evidence that this problem exists and serving private resources stored on DigitalOcean won't work until this is solved.

[lifenautjoe]

I'm not sure anymore whether this is even supported by DigitalOcean.

https://ideas.digitalocean.com/ideas/DO-I-2914

[ZuSe]

We are running into the same problem on OVH.
Basically the whole Storage is based in Swift 3, but works well with most of the S3 API functions.

It would be great if signatures would work for custom domains as well. Even though the token will change after 1,24 or whatever hours it would still save GB of bandwidth.

[lifenautjoe]

According to some of the new replies on that issue raised for digital ocean, signing the object with the non-cdn url and then just replacing it by the cdn-url works.

This would imply that we could "hotfix" this in this library.

I don't have some free space to do this at the moment, so feel free to give it a try.

[awhileback]

Arriving here from google researching this on mixing public/private objects: throwing in an idea I'm about to test...

Could this not be solved by changing the storage class without having to fork/change the code?

@deconstructible
class PrivStorage(S3Boto3Storage):

    custom_domain = None
    bucket_name = settings.AWS_PRIVSTORAGE_BUCKET_NAME
    location = settings.MEDIAFILES_LOCATION

s3_priv_storage = PrivStorage()

The above would accomplish the same thing: you have stripped the "custom_domain" from the storage object for a particular model, which will cause the signing method to fall back to url.bucket_name.digitaloceanspaces.tld

Then you're just a template tag filter away from putting the url back in django templates.

[lifenautjoe]

Anyone had any success with this?

[awhileback]

You'll have to verify all of this on Digital Ocean but based on my previous idea, I have implemented multi-storage on AWS S3/Cloudfront with a mixture of private and public buckets, like so:

custom_storages.py (at the root of your django project):

from django.conf import settings
from storages.backends.s3boto3 import S3Boto3Storage
from storages.utils import setting
from django.utils.deconstruct import deconstructible

@deconstructible
class StaticStorage(S3Boto3Storage):

    bucket_name = settings.AWS_STORAGE_BUCKET_NAME
    location = settings.STATICFILES_LOCATION

s3_static_storage = StaticStorage()


@deconstructible
class MediaStorage(S3Boto3Storage):

    bucket_name = settings.AWS_STORAGE_BUCKET_NAME
    location = settings.MEDIAFILES_LOCATION

s3_media_storage = MediaStorage()


@deconstructible
class PrivStorage(S3Boto3Storage):

    custom_domain = None
    signature_version = 's3v4' # (1. see link below this code section)
    bucket_name = settings.AWS_PRIVSTORAGE_BUCKET_NAME
    location = settings.MEDIAFILES_LOCATION

s3_priv_storage = PrivStorage()

(1. you may have to experiment with signing versions, 's3v4' works on AWS S3 without a cloudfront URL, 's3' works on D.O. with a custom domain)

And then settings.py:

STATIC_URL = f'https://{CDN_URL}/static/'
MEDIA_URL = f'https://{CDN_URL}/media/'
AWS_S3_ACCESS_KEY_ID = os.environ.get('DO_ACCESS_KEY_ID')
AWS_S3_SECRET_ACCESS_KEY = os.environ.get('DO_SECRET_ACCESS_KEY')
AWS_DEFAULT_ACL = None
STATICFILES_LOCATION = 'static'
MEDIAFILES_LOCATION = 'media'
DEFAULT_FILE_STORAGE = 'custom_storages.MediaStorage'
STATICFILES_STORAGE = 'custom_storages.StaticStorage'
AWS_STORAGE_BUCKET_NAME = 'cdn-mydomain-com'
AWS_PRIVSTORAGE_BUCKET_NAME = 'priv-mydomain-com'
AWS_IS_GZIPPED = True

AWS_HEADERS = {
    'CacheControl': 'max-age=86400'
}
# only required with Digital Ocean spaces, not on S3-Cloudfront
#AWS_S3_REGION_NAME = os.environ.get('DO_AWS_REGION')
#AWS_S3_ENDPOINT_URL = f'https://{AWS_S3_REGION_NAME}.digitaloceanspaces.com'
AWS_S3_USE_SSL = True
AWS_S3_CUSTOM_DOMAIN = 'cdn.mydomain.com'

With these you can import and use the individual storage class functions as storage backends. For instance, here's a custom media upload model for wagtail-media that uses the private storage bucket:

from custom_storages import s3_priv_storage

class CustomMedia(AbstractMedia):

    file = FileField(storage=s3_priv_storage, verbose_name=_('file'))
    thumbnail = FileField(upload_to='media_thumbnails', blank=True, verbose_name=_('thumbnail')
    )

    duration = CharField(
        blank=False,
        null=False,
        verbose_name=_('duration'),
        max_length=25,
        help_text=_('Duration in seconds. Valid input formats: HH:MM:SS, MM:SS, or SS.'),
    )

    admin_form_fields = (
        "title",
        "file",
        "collection",
        "duration",
        "width",
        "height",
        "thumbnail",
        "tags",
    )

    def clean(self, *args, **kwargs):

        if self.duration:
            media_duration = sum(int(x) * 60 ** i for i, x in enumerate(reversed(self.duration.split(':'))))
            self.duration = media_duration

        super().clean(*args, **kwargs)

So what this accomplishes is I have two buckets, with media files in both. "location" points to the file structure within the bucket, not the bucket itself, so both my cdn-mydomain-com and priv-mydomain-com buckets have "media" folders that hold images or audio or video or whatever. In the above custom media model example, the media file itself will go to the priv bucket, and the thumbnail will go to the wagtail default public media location specified by 'media_thumbnails'.

By setting "custom_domain = None" for the priv bucket, it is forced to generate signed url when calls to the URL method are made. For items in the cdn bucket, they inherit the cdn url from the settings.py file and do not generate signed urls, but rather cdn.mydomain.com/{{ media.location }}

If you want to go a step further and implement signing of temp urls with the CDN url, as mentioned above just do it in a template tag.

from django.conf import settings

@register.filter
def cdn_url(value):
    if settings.AWS_S3_ENDPOINT_URL in value:
        cdn_domain = 'https://' + settings.AWS_S3_CUSTOM_DOMAIN
        new_url = value.replace(settings.AWS_S3_ENDPOINT_URL, cdn_domain)
        return new_url
    else:
        return value

usage in a template would be like:

{% load myapp_tags %}

{{ media_field.url|cdn_url }}

[awhileback]

FYI the above method is what is recommended in the Digital Ocean docs for generating signed urls on a custom-CDN domain object.

https://docs.digitalocean.com/products/spaces/resources/s3-sdk-examples/#presigned-url

You can use presigned URLs with the Spaces CDN. To do so, configure your SDK or S3 tool to use the non-CDN endpoint, generate a presigned URL for a GetObject request, then modify the hostname in the URL to be the CDN hostname

[lifenautjoe]

Hi @awhileback ,

Thank you so much for your detailed response !

The overriding custom_domain = None works for signing objects with the non-CDN domain.

However if I take the signed url like

https://sfo3.digitaloceanspaces.com/somus/media/private/posts/860fa3ce-f9d4-4337-b168-43e7343cc13f/7e34073f-9dec-40c3-842c-12b954f8d61a.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ALVILVO7ERRUH6ZIJNCK%2F20211014%2Fsfo3%2Fs3%2Faws4_request&X-Amz-Date=20211014T203823Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=f26f659674fea4c8bde399a53106acd4a90798c4d2df9afa5020d5cf83d1d37a

And replace https://sfo3.digitaloceanspaces.com/somus for https://cdn.somus.app like:

https://cdn.somus.app/private/posts/860fa3ce-f9d4-4337-b168-43e7343cc13f/7e34073f-9dec-40c3-842c-12b954f8d61a.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ALVILVO7ERRUH6ZIJNCK%2F20211014%2Fsfo3%2Fs3%2Faws4_request&X-Amz-Date=20211014T203823Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=f26f659674fea4c8bde399a53106acd4a90798c4d2df9afa5020d5cf83d1d37a

The signature is no longer valid, which means the assumption of replacing the host is not valid :-(.

I'm using AWS_S3_SIGNATURE_VERSION = "s3v4" . Maybe has something to do with this?

Thanks a million!

[lifenautjoe]

Woah! If I use AWS_S3_SIGNATURE_VERSION = "s3" and replace the hostname it works!

Thanks a lot!

[awhileback]

Updated guide with your findings, you're welcome!

[jschneier]

A docs PR would be appreciated

…

On Thursday, October 14, 2021, awhileback ***@***.***> wrote: Updated with your findings, I can verify that "s3v4" works on AWS S3, so I suppose people just have to experiment with this setting on a per-provider basis. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#944 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAREDWFKN6BVKCISSP326DTUG5ASVANCNFSM4SLGIULA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[amoralesc]

Worth noting that the signature version 2 (AWS_S3_SIGNATURE_VERSION = "s3") has been deprecated by AWS: https://aws.amazon.com/blogs/aws/amazon-s3-update-sigv2-deprecation-period-extended-modified/

[mriabchenko]

I can confirm that using digital ocean spaces & cusom domain & cdn & acl private & signedUrl is still impossible using up-to-date "regular" approach and contemporary library like @aws-sdk/client-s3 (I am from typescript nodejs world). The only way to make it work is using approach described above - s3 config s3ForcePathStyle: true, signatureVersion: s3 -> generate signed url -> replace / with <custom_endpoint>. Thank you for rising this issue and describing the solution! Also, they are deprecating library that makes such implementation possible https://aws.amazon.com/blogs/developer/announcing-end-of-support-for-aws-sdk-for-javascript-v2/