Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement payload offload support for Azure Blob #3136

Closed
punktilious opened this issue Dec 20, 2021 · 2 comments
Closed

Implement payload offload support for Azure Blob #3136

punktilious opened this issue Dec 20, 2021 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@punktilious
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
Relational database systems are not best suited to storing large objects. This can lead to increase runtime costs as well as scalability limitations.

Describe the solution you'd like
Following the groundwork done in #1869 and #2900, implement payload persistence offloading for Azure Blob.

The solution must provide a mechanism for cleaning orphaned records which may remain after a failed transaction.

The solution should consider using asynchronous patterns for storing and fetching the payload data. This will be important when processing large bundles to maintain acceptable response times.

Describe alternatives you've considered
Accept the limitations and cost of keeping the payload within the RDBMS.

Acceptance Criteria

GIVEN a server configured with payload persistence for Azure Blob enabled
AND a new resource is ingested
WHEN the transaction is successful
THEN the resource can be read back
AND the payload can be found in the offload storage

GIVEN a server configured with payload persistence for Azure Blob enabled
AND a new resource is ingested
WHEN the transaction is rolled back
THEN the payload can not found in the offload storage

GIVEN a server configured with payload persistence for Azure Blob enabled
AND a new resource is ingested
WHEN the transaction fails and rollback is not possible (e.g. network partition)
THEN the payload is removed from the offload storage after the cleanup reconciliation process is run

Additional context
Design discussion to be had regarding how, where and when to run the transaction cleanup process.

The payload persistence mechanism is not tied to a Liberty Datasource so Liberty's transaction recovery mechanism doesn't apply. The discussion is really about whether the cleanup process should be a separate standalone job, or a background thread started by the FHIRServletContext.

Work for this feature should consider #2900 which implements the same offloading mechanism using Cassandra.

@prb112 prb112 added the enhancement New feature or request label Dec 20, 2021
punktilious added a commit that referenced this issue Dec 21, 2021
@punktilious punktilious self-assigned this Feb 9, 2022
punktilious added a commit that referenced this issue Feb 21, 2022
punktilious added a commit that referenced this issue Feb 21, 2022
punktilious added a commit that referenced this issue Feb 22, 2022
punktilious added a commit that referenced this issue Feb 23, 2022
punktilious added a commit that referenced this issue Feb 23, 2022
punktilious added a commit that referenced this issue Feb 25, 2022
punktilious added a commit that referenced this issue Mar 1, 2022
punktilious added a commit that referenced this issue Mar 1, 2022
punktilious added a commit that referenced this issue Mar 1, 2022
@d0roppe
Copy link
Collaborator

d0roppe commented Mar 28, 2022

Was running some concurrent create/delete and this seems to have caused a issue with Azure.Blob offload active.

punktilious added a commit that referenced this issue Mar 29, 2022
punktilious added a commit that referenced this issue Mar 30, 2022
punktilious added a commit that referenced this issue Mar 30, 2022
lmsurpre pushed a commit that referenced this issue Mar 30, 2022
…g is enabled (#3531)

* issue #3136 handle multiple transactions for batch bundles when offload is enabled

Signed-off-by: Robin Arnold <[email protected]>

* issue #3136 added system test for concurrent batch bundle processing

Signed-off-by: Robin Arnold <[email protected]>

* issue #3136 addressed review comments and translate transaction status code

Signed-off-by: Robin Arnold <[email protected]>
@d0roppe
Copy link
Collaborator

d0roppe commented Apr 2, 2022

Tested many scenarios with Azure.blob offload, and currently I have not found any new problems. Closing issue

@d0roppe d0roppe closed this as completed Apr 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants