refactor(autonomi): deterministic archive

[b-zee]

This ensures the same bunch of files/directories will yield the same (serialized) archive (assuming the files haven't changed). Furthermore, this will mean that uploading the same files will not require uploading a new version of the archive (which would differ slightly from the old).

[happybeing]

Hi @b-zee , hope you had a good break.

I'm curious about this change as I wondered about it for my website metadata which also has a hashmap and considered making it deterministic with a btreemap.

Is it that you don't update the time unless the content changes, that this saves chunks?

[b-zee]

Hey Mark! Thanks for asking, I had a good break, hope you've had a nice time as well!

Is it that you don't update the time unless the content changes, that this saves chunks?

This takes the metadata from the filesystem. So unless that changes, you'll end up with the same Archive. That will translate to the same bytes that we'll put on the network, and thus no requirement to put/pay again. (Given that with these changes the Archive will have the deterministically serialized (b-tree) map.) The uploaded metadata was always set to the current time ('now'), so that always translated into a unique Archive.

[maqi]

restore the output balance check during the second client upload within the memory_check CI test.
To ensure there is no extra cost to be paid during repeated uploads.