Sync stable-cadence branch with master #2455
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is to allow CCF codec to use new StreamEncoder.Close() function provided by latest fxamacker/cbor.
- added NewMeteredUFix64FromUint64 to create metered UFix64 from uint64. - added NewMeteredFix64FromInt64 to create metered Fix64 from int64.
Cadence Compact Format (CCF) is a data format designed for compact, efficient, and deterministic encoding of Cadence external values. CCF obsoletes JSON-Cadence Data Interchange Format (JCDIF) for use cases that do not require JSON. Unlike JCDIF, CCF Specifications explicitly defines requirements for - well-formed encodings - valid encodings - deterministic encodings CCF is hybrid data format. CCF-based messages can be: - fully self-describing or - partially self-describing. Both CCF modes are more compact than JSON-based messages. CCF-based protocols can send Cadence metadata just once for all messages of that type. Malformed data can be detected without Cadence metadata and without creating Cadence objects. CCF codec implements CCF Specification (RC1), which is temporarily at github.com/fxamacker/ccf_draft. The CCF specs will be hosted under github.com/onflow after it is updated, cleaned up, and reformatted.
CCF obsoletes JSON-Cadence Data Interchange Format for use cases
that do not require JSON. Given this, preliminary comparisons are
described here for the CCF codec implementing CCF Specifications (RC1).
PRELIMINARY COMPARISONS
Comparisons used 48,309 events from a single mainnet transaction.
There were 9 event types. To simplify benchmark code, the first event's
value in each of the 9 event types was used.
CCF's partially self-describing mode (aka "detached" mode) would be
even smaller than this (e.g. maybe less than 1/4 the size of JSON when
Flow eventually supports detached mode).
SIZE COMPARISON
Encoding | Num Events | Encoded size | Comments
-------- | ---------- | ------------ | --------
JSON | 48,309 | 13,858,836 | JSON-Cadence Data Interchange
CCF | 48,309 | 6,159,931 | CCF in fully self-describing mode
CCF SPEED AND MEMORY COMPARISONS
This isn't apples to apples comparison. JSON data isn't sorted, etc.
- CCF encoder sorts data to encode event data deterministically.
- CCF decoder also verifies event data is sorted, well-formed, and valid.
- CCF encoding is less than 1/2 size of JSON-Cadence Data Interchange.
ENCODER COMPARISON
48k_events_encode_json.log │ 48k_events_encode_ccf.log
sec/op │ sec/op vs base
89.84m ± 17% 69.28m ± 3% -22.88%
48k_events_encode_json.log │ 48k_events_encode_ccf.log
B/op │ B/op vs base
32.45Mi ± 0% 25.82Mi ± 0% -20.45%
48k_events_encode_json.log │ 48k_events_encode_ccf.log
allocs/op │ allocs/op vs base
756.6k ± 0% 370.4k ± 0% -51.05%
DECODER COMPARISON
48k_events_decode_json.log │ 48k_events_decode_ccf.log
sec/op │ sec/op vs base
646.2m ± 8% 158.3m ± 5% -75.50%
48k_events_decode_json.log │ 48k_events_decode_ccf.log
B/op │ B/op vs base
234.97Mi ± 0% 56.16Mi ± 0% -76.10%
48k_events_decode_json.log │ 48k_events_decode_ccf.log
allocs/op │ allocs/op vs base
4.746M ± 0% 1.288M ± 0% -72.86%
Benchmarked using Go 1.19.6, linux_amd64, i5-13600k. Results are
subject to change because CCF codec reference implementation in Go
has not yet been reviewed or merged into onflow/cadence yet.
Added comment stating that cadence.String and cadence.Character
must be valid UTF-8 and it is the application's responsibility
to provide the CCF encoder with valid UTF-8 strings.
"Valid CCF Encoding Requirements" in CCF Specification states:
"Encoders are not required to check for invalid input items
(e.g. invalid UTF-8 strings, duplicate dictionary keys, etc.)
Applications MUST NOT provide invalid items to encoders."
Co-authored-by: Bastian Müller <[email protected]>
Updated Go comments containing CCF specs in CDDL notation. Specifically, changed CDDL notation representing "one or more" to "zero or more" for: - composite-value.fields - composite-type.fields - composite-type-value.fields See CCF specification PRs: - fxamacker/ccf_draft#66 - fxamacker/ccf_draft#67
Updated Go comments containing CCF specs in CDDL notation. Specifically, changed CDDL notation representing "one or more" to "zero or more" for: - restricted-type.restrictions - restricted-type-value.restrictions See CCF specification PRs: - fxamacker/ccf_draft#66 - fxamacker/ccf_draft#67
CCF specs were updated to allow "zero or more" sortable items rather than "one or more". Given this, update CCF codec to check for zero items before sorting. Relevant changes to CCF specification include: - fxamacker/ccf_draft#66 - fxamacker/ccf_draft#67 Co-authored-by: Bastian Müller <[email protected]>
- Return Cadence UnexpectedError for implemention errors. - Rethrow Go runtime errors and Cadence internal errors in: - CCF Encoder.Encode() - CCF Decoder.Decode()
Co-authored-by: Bastian Müller <[email protected]>
Thanks @SupunS for the suggestion!
Co-authored-by: Bastian Müller <[email protected]>
Explained why tids isn't passed down and other related aspects.
Cadence Benchstat comparisonThis branch with compared with the base branch onflow:feature/stable-cadence commit be303b7 Collapsed results for better readability
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
SupunS
force-pushed
the
supun/sync-stable-cadence
branch
2 times, most recently
from
6e89c5d to
adca90b
Compare
|
There were quite a few conflicts, mostly between the #2401, and the purity changes. In addition to conflicts, had to do the following behavioral changes:
|
SupunS
force-pushed
the
supun/sync-stable-cadence
branch
from
adca90b to
46b6996
Compare
SupunS
commented
Apr 20, 2023
Codecov Report
@@ Coverage Diff @@
## feature/stable-cadence #2455 +/- ##
==========================================================
- Coverage 78.75% 78.48% -0.28%
==========================================================
Files 317 326 +9
Lines 69296 72949 +3653
==========================================================
+ Hits 54574 57253 +2679
- Misses 12919 13615 +696
- Partials 1803 2081 +278
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 4 files with indirect coverage changes Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
SupunS
requested review from
turbolent,
dsainati1 and
joshuahannan
as code owners
dsainati1
approved these changes
Apr 21, 2023
6 tasks
turbolent
reviewed
Apr 22, 2023
turbolent
reviewed
Apr 22, 2023
|
Would like to get this merged soon since I'm going to be reworking the entitlement stack. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Sync stable-cadence branch with master
masterbranchFiles changedin the Github PR explorer