What's the fastest way to setup static files in a moto S3 bucket or dump all files?

[couling]

Is access to S3 objects only possible via the API. For testing purposes is there any way to access (read or write) S3 objects without going through boto3?

Using Moto to unit test S3 clients is great! The one problem I run into is test performance with 20 to 100ms per test spent primarily on pytest setup executing boto3 requests to moto.

I have a test fixture which requires an S3 bucket to be pre-populated with a bunch of static files and the tests then tests themselves mutate those static files. The setup currently needs to create_bucket execute a bunch of put_object. While this may only take 50ms per test, that's a lot of time when running a lot of parameterised tests.

I was wondering if, when using the with moto.mock_s3, there's any way to get files in through a back door, or likewise get a dump of everything rapidly.

Is my only option to talk to moto via boto3, or is there something quicker for bulk transfers?

[bblommers]

Hi @couling! The simple answer: yes, there is only one official/supported way to access S3 objects.

We are in the process of releasing Moto v5, which should have much better performance. Are you able to upgrade to moto==5.0.0alpha2 to test this out?
See the announcement with upgrade notes here: #7198

[couling]

I'm afraid to say that experimenting with 5.0.0alpha2 has resulted in slower tests, not quicker tests. +50% runtime.

[couling]

I can't rule out that is may be me using it incorrectly. I was executing moto.mock_aws() every unit test.

[bblommers]

Interesting. Can you share a reproducible test case, or even the repository?

[couling]

Okay here's a similar example I can show you in my own open source project. There's fewer AWS unit tests in this project but the individual unit tests have similar times. Total runtime 5.0.0alpha2 13.5 seconds vs 4.2.13 8.0 seconds.

pip install poetry
git clone https://github.com/couling/MkdocsDeploy/
cd MkdocsDeploy

Then branch main

git checkout 75fd1981460ba445441667cc86afbe3c4769e6c1
poetry install
poetry run pytest

126 passed, 1 skipped in 8.09s

Vs branch experimental-moto-5

git checkout c2a931a7348ff4e9b0bbc65e398375f09e1bf0ab
poetry install
poetry run pytest

126 passed, 1 skipped in 13.67s

[bblommers]

Ah, gotcha. Thanks for sharing an example. The latest branch (not yet released) does have some performance improvements over v4:

poetry remove moto
poetry add git+https://github.com/getmoto/moto.git#v5
poetry run pytest

126 passed, 1 skipped in 6.62s

[bblommers]

It is possible to use internal API to upload all files to S3. But note that, as it is internal, the API will change/break without warning. So use at your own risk.

The benefit is that you can skip the boto3-initialization/validation for these operations, which tends to take a lot of time.

The MkdocsDeploy-project sees marginal gains with this approach, around 10/20 ms - but maybe that improves for projects with more test files.

from moto.core import DEFAULT_ACCOUNT_ID
from moto.s3 import s3_backends

def upload_whole_site(s3_bucket: str, target_prefix: str, site_spec: versions.DeploymentSpec | None):

    if site_spec is None:
        return None
    metadata = shared_implementations.generate_meta_data(site_spec)
    s3_backend = s3_backends[DEFAULT_ACCOUNT_ID]["global"]
    # Initialise the bucket with the site's metadata
    for filename, content in metadata.items():
        s3_backend.put_object(s3_bucket, key_name=target_prefix + filename, value=content)
    # Put a dummy index.html file for every version and alias:
    for key in itertools.chain(site_spec.versions, site_spec.aliases):
        s3_backend.put_object(s3_bucket, key_name=target_prefix + key + "/index.html", value=b"")
    if site_spec.default_version:
        s3_backend.put_object(s3_bucket, key_name=target_prefix + "index.html", value=b"")

[robwhiteston]

Would this approach work if moto is running in a docker container, but the tests are running on the host?

[bblommers]

No @robwhiteston, that would only work when using the decorators. There is no quick way to setup S3 files if Moto is running inside Docker/as a server.