consider switching the JSON3

[ExpandingMan]

This might be a fairly disruptive change, but I really think this package should consider dumping JSON for JSON3. JSON3 is lazy by default, very fast, and it might eventually be very useful here that it has integration with StructTypes.jl.

I've recently had a few cases where AWS returned lists of hundreds of thousands of items and JSON.jl is incredibly slow.

Might be able to help with this, but I wanted to gauge willingness. Seems likely it will involve a breaking change, but hopefully not "badly" breaking.

[omus]

I think switching out the JSON library would be fine. One consideration is that the JSON dict key ordering shouldn't change as that can be annoying with IAM policies. As JSON3 already maintains the ordering I don't see any major issues with changing things.

Alternatively improving raw output support (#384) would allow you to use whatever JSON parsing library you prefer

[mattBrzezinski]

This is something that I was thinking about recently. I wanted to benchmark the performances between these three packages;

• JSON
• JSON3
• LazyJSON

I know LazyJSON is really really good. But it is also a package needs a lot of work to get it into a proper state. From my understanding it was more of a concept package. I'm definitely in favor of speeding things up, and converting to JSON3.jl should be pretty straight forward.

[ExpandingMan]

JSON3 is definitely fast and it is already pretty widely used. It is also created and maintained by the incredibly prolific Jacob Quinn, so it is likely to remain very well maintained unless he is trampled to death in a freak llama stampede or something.

I do agree that perhaps we should think carefully about what we can lay on top of #384 (though I think JSON3 would be good to have in any case). The AWS API is very... well, a polite way to put it would be pedantic, but it's also not very consistent. There are about a trillion specific dict formats which they treat like full-blown types. It's all very nested and Java-y. While it doesn't seem like it would be any fun at all to try to generate them, some more restructuring of the library might be nice. For example, if params could be given as keyword arguments instead of in the dict. Sometimes the separation between main arguments and params in the AWS.jl don't make much sense either, but I suppose this is probably something we inherit from the AWS REST API.

[mattBrzezinski]

JSON3 is definitely fast and it is already pretty widely used. It is also created and maintained by the incredibly prolific Jacob Quinn, so it is likely to remain very well maintained unless he is trampled to death in a freak llama stampede or something.

Yeah that's a good point. Even if LazyJSON were faster, does anyone really want to maintain it? Unless it's some how substantially faster, using JSON3 seems like the correct move here.

[mattBrzezinski]

if params could be given as keyword arguments instead of in the dict.

Yes! I think this is a really wanted change and will make everyone's lives better. It shouldn't be too challenging to convert this into some form of kwargs.

[ExpandingMan]

Yeah... I was going to bring it up before but I don't think there's any chance the interface to this package is ever going to be "nice". I think the general lousiness and Javaness of AWS's API dooms us to needing tons of wrapper packages 😨

Anyway, would still be nice.

[mattBrzezinski]

Yeah... I was going to bring it up before but I don't think there's any chance the interface to this package is ever going to be "nice". I think the general lousiness and Javaness of AWS's API dooms us to needing tons of wrapper packages 😨

Anyway, would still be nice.

Yeah... I think AWS.jl will always end up being the barebones format where you'll need to (some degree) jump around Dicts.

Building out packages such as AWSS3.jl to have these convenience functions that hide away those parts is do-able. But I don't see a way that you can auto-generate that stuff? Unless it's already pre-written in another language and we convert into Julia (or use something underlying, but I don't think there's a well-formed AWS C library?).

[ExpandingMan]

Yeah, there's really no way to do it. I think the only possible solutions are:

1. Build a super sophisticated AWS.jl that consumes AWS API spec and docs and is somehow magically so good that it handles the horrible inconsistency thereof and somehow miraculously results in a package that's nice to use.
2. Write a bunch of secondary wrapper packages as needed.

I think we'll probably be going with 2 lol

[omus]

One gotcha with switching to JSON3 is that the default parsing behaviour when parsing JSON and XML will have very different output unless we make a StructTypes style XMLDict.

[ExpandingMan]

Ugh, I keep forgetting how so much of their API relies on XML. Yuck.

My vote would be to still go to JSON3 and just do whatever we can to mangle the XML stuff into a similar output (e.g. use StructTypes) because none of the available XML parsers are nearly as nice as JSON3, in my opinion.

Maybe we can berate Jacob Quinn into writing an XML parser too 😆

[omus]

As of AWS.jl 1.63 (#384) you can control how AWS.jl parses responses when using the feature use_response_type (e.g. @service S3 use_response_type=true). You just need to define your own read function which can handle the same MIME-types as defined here:

AWS.jl/src/utilities/response.jl

Lines 51 to 63 in 64c7a72

	function _read(io::IO, ::MIME"application/xml")
	xml = parse_xml(read(io, String))
	root = XMLDict.root(xml.x) # Drop XML declaration
	return xml_dict(root, LittleDict{Union{String,Symbol},Any})
	end
	function _read(io::IO, ::MIME"application/json")
	# Note: Using JSON instead of JSON3 since it does not support OrderedDict/LittleDict
	return JSON.parse(io; dicttype=LittleDict{String,Any})
	end
	_read(io::IO, ::MIME"text/plain") = read(io, String)
	_read(io::IO, ::MIME) = read(io)

and then do parse(my_read_function, aws_response).

We can still internally switch over to JSON3 but we'll need to deal with parsing XML responses in a similar way to JSON3.

[omus]

I think we should probably switch to JSON3 as the default for AWS.jl@2. We still need to deal with the XML bit but we could probably do something inefficient to start with that returns a JSON3 like structure.