limit record size of manifest #2114
Conversation
api/python/quilt3/packages.py
Outdated
|
|
||
| def dumps(obj): | ||
| data = json_encode(obj).encode() | ||
| if len(data) > MAX_MANIFEST_RECORD_SIZE: |
There was a problem hiding this comment.
Maybe we should set this limit on the size of user metadata, but then we need to be sure that we have enough space for our own data.
There was a problem hiding this comment.
I think you are doing it right: we should measure the TOTAL size of the row because that's what S3 select will care about.
sir-sigurd
force-pushed
the
manifest-row-size-limit
branch
from
2bc227f
to
4d9de18
Compare
Codecov Report
@@ Coverage Diff @@
## master #2114 +/- ##
==========================================
+ Coverage 46.26% 46.33% +0.07%
==========================================
Files 430 430
Lines 20581 20608 +27
Branches 2418 2418
==========================================
+ Hits 9521 9548 +27
Misses 10098 10098
Partials 962 962
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
api/python/quilt3/packages.py
Outdated
|
|
||
| def dumps(obj): | ||
| data = json_encode(obj).encode() | ||
| if len(data) > MAX_MANIFEST_RECORD_SIZE: |
There was a problem hiding this comment.
I think you are doing it right: we should measure the TOTAL size of the row because that's what S3 select will care about.
sir-sigurd
force-pushed
the
manifest-row-size-limit
branch
6 times, most recently
from
359b8af
to
65945d9
Compare
sir-sigurd
force-pushed
the
manifest-row-size-limit
branch
from
65945d9
to
881586d
Compare
Co-authored-by: Aneesh Karve <[email protected]>
Co-authored-by: Aneesh Karve <[email protected]>
TODO
build.pyfor new docstrings