Catalog 1.58 docs #4336
Catalog 1.58 docs #4336
Conversation
Co-authored-by: Dr. Ernie Prabhakar <[email protected]>
Co-authored-by: Dr. Ernie Prabhakar <[email protected]> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Sergey Fedoseev <[email protected]>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #4336 +/- ##
========================================
Coverage 39.07% 39.07%
========================================
Files 787 787
Lines 34813 34813
Branches 5525 5525
========================================
Hits 13604 13604
- Misses 20026 20666 +640
+ Partials 1183 543 -640
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
PR Summary
This PR introduces comprehensive documentation for the new Packaging Engine feature and updates Tabulator documentation with version 1.58 enhancements, focusing on error handling and configuration options.
- Added new
/docs/Catalog/Packaging.mddetailing automated package creation from S3 data via Admin GUI, SQS queue, and EventBridge rules - Updated
/docs/advanced-features/tabulator.mdwith newcontinue_on_errorconfiguration and$issuecolumn functionality - Added Packaging Engine link to
/docs/SUMMARY.mdunder Catalog User section - Fixed undefined
rule_namevariable in EventBridge example code - Corrected typo in ORCID website link ("ORDiD" to "ORCID")
3 file(s) reviewed, 4 comment(s)
Edit PR Review Bot Settings | Greptile
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
| The Quilt Packaging Engine in the Quilt Platform allows administrators and | ||
| developers to automate the process of creating Quilt packages from data stored | ||
| in Amazon S3. It serves as a key component in Quilt's SDMS (Scientific Data | ||
| Management System) strategy, enabling automated data ingestion and |
There was a problem hiding this comment.
Is this relevant to end users? This is important to us. Maybe we could rephrase as "It serves as a key compoent of Quilt's functionality as a Scientific Data Management System" ...
| developers to automate the process of creating Quilt packages from data stored | ||
| in Amazon S3. It serves as a key component in Quilt's SDMS (Scientific Data | ||
| Management System) strategy, enabling automated data ingestion and | ||
| standardization. It consists of: |
There was a problem hiding this comment.
Suggest "currently consists of" implying that there are features to come down the line.
|
|
||
| 1. Admin Settings GUI to enable package creation based on notifications from: | ||
| 1. AWS Health Omics | ||
| 2. Nextflow workflows using `nf-prov`'s WRROC ([Workflow Run |
|
|
||
| When enabled, this will create a package when indexing any folder containing an | ||
| `ro-crate-manifest.json`. Indexing happens when the bucket is added to the | ||
| stack, or when a folder is written to a bucket already in the stack. |
There was a problem hiding this comment.
This is confusing when on its own, I think we can drive clarity by linking to Quilt's Indexing documentation
|
|
||
| ## SQS Message Processing | ||
|
|
||
| The primary interface to the Packaging Engine is through an SQS queue in the |
There was a problem hiding this comment.
In order for this to make sense we need an overview of the components of the packaging engine. Right now you're assuming that the reader understands what the packaging engine is, and why the SQS queue is used. An overview of how the packaging engine is composed might be helpful (in a section above).
| ### Example: Event-Driven Packaging (EDP) | ||
|
|
||
| [Event-Driven Packaging](../advanced-features/event-driven-packaging.md) is a | ||
| high-end add-on to Quilt that coalesces multiple S3 uploads into a single |
There was a problem hiding this comment.
what is a high-end add-on?
| When enabled, this will create a package when indexing any folder containing an | ||
| `ro-crate-manifest.json`. Indexing happens when the bucket is added to the | ||
| stack, or when a folder is written to a bucket already in the stack. |
There was a problem hiding this comment.
we don't create packages for existing ro-crate-manifest.json when bucket is added to the stack
| @@ -0,0 +1,226 @@ | |||
| # Packaging Engine | |||
There was a problem hiding this comment.
let's add a note that it requires 1.58
Description
Tabulator and Packager changes
TODO