Why the @service macro rather than just defining submodules of AWS.jl ?

[oxinabox]

I struggle to understand why there is not just e.g. AWS.S3, AWS.Cloudwatch etc.
And rather the user has to do @service AWS.Cloudwatch.
Which as well as being weird, is a bit spooky in that it is creating a new instance of that module in their namespace.

My guess is it is left-over from when julia's precompilation was worse, and so it was to minimize the amount of code that had to be parsed.
If so now, that precompilation is cached across environments (as of Julia 1.3),
and run in parallel, and at Pkg.add time (as of Julia 1.6) it would seem that motivation (if that was the motivation) is negated.
And further that by requiring it, if anything we are increasing pre-compile time since it can no longer be shared.

If I am correct and it is indeed vestigial we could change this to generate all services within AWS.jl and then mark the @service macro as deprecated.

[oxinabox]

Doing this would also negate the need for #435
since all services would be in the AWS module rather than in the user's namespace

[mattBrzezinski]

When I was originally doing this including all modules was substantially slower when loading in the package. I wanted to make this package more light weight. I knew there would only really be a handful of AWS services which users would actually use in their code at one given time it seemed better to introduce this @service macro.

I wouldn't mind removing the @service macro in favour of including these submodules if we can show that it doesn't slow things down.

[oxinabox]

We should open an issue on the JuliaLang/julia repo if this work doesn't basically just precompile away.
I expect it might incur a lot of precompile time, but it shouldn't add much load time

[omus]

A prerequisite for this:

• Choose a canonical names for services #462

[ericphanson]

I’ve been wondering about this too. One difference is how many copies of the code and of the functions there are. The current way is 1 copy per user instead of 1 copy total, eg if two downstream libraries both use s3, they each get their own copy of the module by calling the macro. So e.g. if one pirates something, the other is unaffected.

I guess this is kind of a feature in that it enables #457 to be possible— one downstream library could ask for slightly different variants of the functions than another (eg with and without the new Response objects @omus is working on).

It seems kinda crazy and unidiomatic to have # copies of a module scaling with # of downstream users, but on the other hand it is autogenerated code, and the functions aren’t really treated as generic functions in practice. It kind of reminds me of the “private modules” thing the VSCode folks want (/ have by vendoring copies of all their dependencies).

So maybe @service could be seen as the user asking for code generation rather than loading a module. And things like AWSS3 could take advantage of this by adding convenience methods to its copy of the S3 module and re-exporting that, rather than defining its own functions that wrap around it.

It seems like #457 basically takes this approach and says “if we’re doing code generation for the caller, we might as well let them customize it”. Whereas this issue is saying that we shouldn’t be doing code generation at call-time and instead should be doing it exclusively upfront in the PRs to here.

[omus]

It seems like #457 basically takes this approach and says “if we’re doing code generation for the caller, we might as well let them customize it”. Whereas this issue is saying that we shouldn’t be doing code generation at call-time and instead should be doing it exclusively upfront in the PRs to here.

As I mentioned in #457 we can both have the high-level API accessible via submodules in AWS.jl and also support embedding the same submodules in other packages. The only real reason I can see for having both would be to support customization like in #457.

I am definitely not opposed to having services exist as submodules as long as we're not seeing performance issues. If we are seeing such issues I would say we can stick with @service for now and file an issue.

[iamed2]

Motivating factors for the original decision (not an argument for keeping status quo):

First, AWSSDK.jl was slow to load. On my computer, load time (not including precompilation) on Julia 1.0/1.1 was 1.8 seconds. Older versions of Julia were even slower, but Julia 1.0 is the oldest version of Julia that I have easily accessible. Load time was more than precompile time. Load time for AWSSDK.jl is now 0.5 seconds in Julia 1.6. AWSSDK.jl contains basically the same contents as the service files so this is a good proxy. Also worth noting that, for some reason, both precompile and load times are significantly increased in Docker on EC2.

The @service macro makes it possible to have the service code precompiled in the user's application, since lazy-loaded code cannot be precompiled in an existing package. It's important to clarify that the @service macro does not generate any code, it just loads it. This is the best solution I could see for loading services only when necessary, but having the entire application be precompilable.

[c42f]

While @service is kind of weird, I do feel happy not to load all of src/services into the Julia runtime - it's about 22 MB of source code! That's kind of an absurd amount of work to be asking of the Julia runtime, given that I'm usually using about three services at a time.

I've seen older code which refuses to use AWSSDK.jl and just uses AWSCore.jl directly, presumably to avoid this. (Also perhaps because the high level services defined in AWSSDK don't make the AWS APIs a lot easier to use.)