Predicable object iteration order

[gibson042]

Spun off from #47 (reply in thread)

JavaScript learned the hard way that "implementations are not required to return values in any specific order" is not really a tenable position for a language with multiple implementations, because in practice if a sufficiently popular implementation does provide results in a predictable order then users come to rely upon that behavior (cf. Hyrum's Law) and less popular implementations are forced to follow suit, resulting in a de facto predictable order that eventually gets added to the specification anyway. I am only aware of two ways around this: either specify a predictable order up front, or specify that order is intended to be unpredictable in any particular implementation (such that an implementation which sorted entries or preserved source order would be nonconforming, which is basically impossible to ensure with conformance tests).

I absolutely understand the desire to let the use of predictable vs. unpredictable ordering be unspecified and therefore support maximum convenience for building an implementation, but my experience is that it just doesn't work out in the long run.

Accordingly, keys and values and a potential items should all be specified to return results in a particular order. Using source order is possible and would be acceptable, but I'm more partial to specifying a static algorithm by which keys must be sorted (e.g., lexicographically by Unicode code point) because that minimizes potential security vulnerabilities derived from leveraging information about insertion order.

[springcomp]

Thanks a lot for that.

I'm completely aware that Hyrum's Law will kick in. That's why I suggested standardizing an undocumented behaviour.

By any chance, would you feel like drafting a JEP here ?

[gibson042]

The more I think about this, the more I think that this should be left for the application to define. Forcing an order could be problematic to the underlying data structure types.

Understandable, but what mechanism would prevent the ecosystem from de facto requiring consistency with a predictable ordering exhibited by popular implementations? If the prevailing convention is e.g. source order and people start to rely upon that, then bugs will be filed against implementations with unpredictable ordering and they'll capitulate despite technically conforming with the spec. Implementation freedom with respect to observable behavior is a hazard in general, but for this case in particular there is already precedent that can hopefully be learned from (cf. https://github.com/tc39/proposal-for-in-order ).

[springcomp]

… (e.g., observe at https://jmespath.site/ how the non-I-JSON `"\uD800\uDEAD"` is round-trippable).

I would be interested to understand more about this observation.

Seems to me, that "\uD800\uDEAD" is a valid I-JSON encoding of an otherwise invalid Unicode string (it is explicitly called as valid in section 2.1 of that spec).

What do you mean by round-trippable ? That it can be converted to an in-memory representation of a string and serialized back to the same JSON escaped string ?

[springcomp]

The more I think about this, the more I think that this should be left for the application to define. Forcing an order could be problematic to the underlying data structure types.

@innovate-invent I understand your point and would tend to agree when requiring a specific sort order.
Would you still feel that way if we merely required that source order be preserved ?

[innovate-invent]

That depends on the definition of "source order". I generally use JMESPath to manipulate arbitrary data structures that do not necessarily originate from a document, or even JSON. For example, a 'object' can be represented (in Python) by a dict, a arbitrary class type, or any other associative data type. A 'array' can be represented as a native list, generator, or any class type with the __getitem__ function accepting a number. These arbitrary types can have their own defined order that they iterate over their sub-elements. If the underlying object is the source, then "source order" would be acceptable, otherwise it really needs to be left to the application to define.

It is easy to forget that JMESPath is a library, and certain allowances need to be made to not over-constrain the importing application.

[gibson042]

… (e.g., observe at https://jmespath.site/ how the non-I-JSON `"\uD800\uDEAD"` is round-trippable).

I would be interested to understand more about this observation.

Seems to me, that "\uD800\uDEAD" is a valid I-JSON encoding of an otherwise invalid Unicode string (it is explicitly called as valid in section 2.1 of that spec).

I apologize for the mistake. "\uD800\uDEAD" is absolutely valid (as is its unescaped analog, "𐊭"). I meant to use "\uDEAD", which does encode an unpaired surrogate and therefore is not valid I-JSON.

What do you mean by round-trippable ? That it can be converted to an in-memory representation of a string and serialized back to the same JSON escaped string ?

Basically, yes. Not necessarily the same JSON string, but one that represents exactly the same data (in this case, "\uDEAD" → "\udead") and is therefore treated equivalently if reused as input.