S3 FileSystem Questions

[mbklein]

Since I'll be running Livebook in an AWS Elastic Container Service task, I plan to use S3 to store our livebooks, but I've run into a couple questions I can't figure out by reading either the docs or the code.

1. Is there a way to bootstrap a new Livebook with a preconfigured S3 FileSystem as the default? Since ECS containers are ephemeral and we don't have a persistent volume attached for storage, we need to have some way to tell Livebook where the books are going to live without having to do it through the UI. I'm not at all opposed to creating a custom image with a pre-populated livebook_config.ets, but if there's a better way to do it, I'd love to hear it.
2. Is there a way to tell Livebook to store config changes to S3 as well, or is that always expected to be part of the local filesystem?
3. It looks like the S3 FileSystem currently requires hard-configured keys. As suggested by AWS, we use temporary credentials and IAM roles rather than keys for everything we can. Ideally, it would be fairly easy to use something like the aws_credentials package to retrieve credentials, but the code doesn't seem to work with the current EC2/ECS metadata endpoints. [UPDATE: I was wrong. aws_credentials is working fine.] Are there plans to add ephemeral/role-based credential discovery to the S3 FileSystem provider?

As I've typed all this out, I've realized that the first two issues (and the third, to some extent) are things Livebook Teams is looking to solve. But I'm not really looking to synchronize configurations across multiple Livebook instances – I just need to keep a single ephemeral instance configured between restarts – so I don't think I'd be able to justify a subscription to those who would have to approve it. I also know they'd be wary of storing S3 access keys with a third party provider, even with strong encryption involved.

[josevalim]

We can’t store the config on S3 because the S3 config is stored in the config itself.

1 is meant to be tackled by Livebook Teams but, if someone updates aws_credentials, we can explore adding integration to it through Livebook, so it supports to EC2 endpoints.

[mbklein]

if someone updates aws_credentials, we can explore adding integration to it through Livebook, so it supports to EC2 endpoints

Actually, it looks like I was incorrect in my previous post. I've tested aws_credentials with ECS, EC2, and environment credentials, and it's working fine after all. I had a misconfiguration in my previous test that was messing with things. (The error messages in the setup log are just for all the credential methods that aren't present in the current environment.)