-
Notifications
You must be signed in to change notification settings - Fork 160
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement payload offload support for Azure Blob #3136
Labels
enhancement
New feature or request
Comments
punktilious
added a commit
that referenced
this issue
Dec 21, 2021
Signed-off-by: Robin Arnold <[email protected]>
punktilious
added a commit
that referenced
this issue
Feb 21, 2022
Signed-off-by: Robin Arnold <[email protected]>
punktilious
added a commit
that referenced
this issue
Feb 21, 2022
Signed-off-by: Robin Arnold <[email protected]>
punktilious
added a commit
that referenced
this issue
Feb 22, 2022
Signed-off-by: Robin Arnold <[email protected]>
punktilious
added a commit
that referenced
this issue
Feb 23, 2022
Signed-off-by: Robin Arnold <[email protected]>
punktilious
added a commit
that referenced
this issue
Feb 23, 2022
Signed-off-by: Robin Arnold <[email protected]>
punktilious
added a commit
that referenced
this issue
Feb 25, 2022
Signed-off-by: Robin Arnold <[email protected]>
punktilious
added a commit
that referenced
this issue
Mar 1, 2022
Signed-off-by: Robin Arnold <[email protected]>
punktilious
added a commit
that referenced
this issue
Mar 1, 2022
Signed-off-by: Robin Arnold <[email protected]>
punktilious
added a commit
that referenced
this issue
Mar 1, 2022
Signed-off-by: Robin Arnold <[email protected]>
Was running some concurrent create/delete and this seems to have caused a issue with Azure.Blob offload active. |
punktilious
added a commit
that referenced
this issue
Mar 29, 2022
…ad is enabled Signed-off-by: Robin Arnold <[email protected]>
punktilious
added a commit
that referenced
this issue
Mar 30, 2022
Signed-off-by: Robin Arnold <[email protected]>
punktilious
added a commit
that referenced
this issue
Mar 30, 2022
…s code Signed-off-by: Robin Arnold <[email protected]>
lmsurpre
pushed a commit
that referenced
this issue
Mar 30, 2022
…g is enabled (#3531) * issue #3136 handle multiple transactions for batch bundles when offload is enabled Signed-off-by: Robin Arnold <[email protected]> * issue #3136 added system test for concurrent batch bundle processing Signed-off-by: Robin Arnold <[email protected]> * issue #3136 addressed review comments and translate transaction status code Signed-off-by: Robin Arnold <[email protected]>
Tested many scenarios with Azure.blob offload, and currently I have not found any new problems. Closing issue |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
Relational database systems are not best suited to storing large objects. This can lead to increase runtime costs as well as scalability limitations.
Describe the solution you'd like
Following the groundwork done in #1869 and #2900, implement payload persistence offloading for Azure Blob.
The solution must provide a mechanism for cleaning orphaned records which may remain after a failed transaction.
The solution should consider using asynchronous patterns for storing and fetching the payload data. This will be important when processing large bundles to maintain acceptable response times.
Describe alternatives you've considered
Accept the limitations and cost of keeping the payload within the RDBMS.
Acceptance Criteria
GIVEN a server configured with payload persistence for Azure Blob enabled
AND a new resource is ingested
WHEN the transaction is successful
THEN the resource can be read back
AND the payload can be found in the offload storage
GIVEN a server configured with payload persistence for Azure Blob enabled
AND a new resource is ingested
WHEN the transaction is rolled back
THEN the payload can not found in the offload storage
GIVEN a server configured with payload persistence for Azure Blob enabled
AND a new resource is ingested
WHEN the transaction fails and rollback is not possible (e.g. network partition)
THEN the payload is removed from the offload storage after the cleanup reconciliation process is run
Additional context
Design discussion to be had regarding how, where and when to run the transaction cleanup process.
The payload persistence mechanism is not tied to a Liberty Datasource so Liberty's transaction recovery mechanism doesn't apply. The discussion is really about whether the cleanup process should be a separate standalone job, or a background thread started by the FHIRServletContext.
Work for this feature should consider #2900 which implements the same offloading mechanism using Cassandra.
The text was updated successfully, but these errors were encountered: