docs: ADR for mediator-message-storage [skip ci] #530

mineme0110 · 2023-05-18T15:39:17Z

Overview

Fixes ATL-4215

Checklist

My PR contains...

No code changes (changes to documentation, CI, metadata, etc.)
Bug fixes (non-breaking change which fixes an issue)
Improvements (misc. changes to existing features)
Features (non-breaking change which adds functionality)

My changes...

are breaking changes
are not breaking changes
If yes to above: I have updated the documentation accordingly

Documentation

My changes do not require a change to the project documentation
My changes require a change to the project documentation
If yes to above: I have updated the documentation accordingly

Tests

My changes can not or do not need to be tested
My changes can and should be tested by unit and/or integration tests
If yes to above: I have added tests to cover my changes
If yes to above: I have taken care to cover edge cases in my tests

github-actions · 2023-05-18T15:40:30Z

E2E Test Results

10 files ±0 10 suites ±0 5m 24s ⏱️ -17s
19 tests ±0 19 ✔️ ±0 0 💤 ±0 0 ❌ ±0
30 runs ±0 30 ✔️ ±0 0 💤 ±0 0 ❌ ±0

Results for commit 836c5c8. ± Comparison against base commit 3b9437c.

♻️ This comment has been updated with latest results.

github-actions · 2023-05-18T15:44:35Z

Atala PRISM Test Results

  49 files ±0   49 suites ±0 35s ⏱️ +2s
370 tests ±0 370 ✔️ ±0 0 💤 ±0 0 ❌ ±0
371 runs ±0 371 ✔️ ±0 0 💤 ±0 0 ❌ ±0

Results for commit 836c5c8. ± Comparison against base commit 3b9437c.

♻️ This comment has been updated with latest results.

github-actions · 2023-05-18T15:44:37Z

Atala PRISM Code Coverage

File	Coverage
All files	`22%`	✅

Minimum allowed coverage is 12%

Generated by 🐒 cobertura-action against 836c5c8

…m-building-blocks into ATL-4215

ghost · 2023-05-18T18:19:14Z

docs/decisions/20230515-mediator-message-storage.md

@@ -0,0 +1,38 @@
+# Mediator message storage 


The title usually depicts the decision being made - in this - if the outcome is to be mongodb - the title could be: use-mongo-db-for-message-storage.md

This way, just the filenames tell part of the story - this should be part of the guidelines but having read them - it doens't make this clear - I'll move them to the handbook and update them

ghost · 2023-05-18T18:23:06Z

docs/decisions/20230515-mediator-message-storage.md

+MongoDB typically stores data within documents using JSON (JavaScript Object Notation) or BSON (Binary JavaScript Object Notation), which simplifies the handling of big data and complex data streams.
+document database supports a rapid, iterative cycle of development the way that a document database turns data into code.
+MongoDB is faster at inserting and for queries that use nested references instead of joins
+In Mediator the data which we send or receive is json message and to process json message we can avoid the unnecessary serialization and deserialization by having it stored in document database.


Woudn't using postgres with a single text/string field accomplish the same thing? What are the read and write paths we ne need to support - sequential or random access [get by ID or search]

The cost [congnitive load, someone to install it, someone to write migrations and manage it, something to sort it out when it goes wrong] can be high even when using a tool that has automations for such operations. We still need to understand how to run it locally for development and for self hosted

ghost · 2023-05-18T18:24:25Z

docs/decisions/20230515-mediator-message-storage.md

+Mediator messages store in simple and strigh forward write there is no transactional workflow involved so we don't gain much by using relational db like postgres.
+
+Below are the 2 options which we can use to reduce infrastructure management 
+MongoDB Atlas. Fully managed MongoDB in the cloud which can reduce the infrastructure management https://www.mongodb.com/atlas/database


How do these reduce the overhead? Is it worth the cost over using and existing DB that's already running and using jsonblobs?

It reduces the overhead of not worrying about the backup and scalability and cluster maangement
Have read at this
https://medium.com/@nparsons08/mongodb-atlas-technical-overview-benefits-9e4cff27a75e#:~:text=MongoDB%20Atlas%20provides%20high%20throughput,compared%20to%20traditional%20database%20software.

ghost · 2023-05-18T18:26:59Z

docs/decisions/20230515-mediator-message-storage.md

+## Refrences used 
+https://www.plesk.com/blog/various/mongodb-vs-postgresql/
+https://www.dbvis.com/thetable/json-vs-jsonb-in-postgresql-a-complete-comparison/
+https://severalnines.com/blog/overview-json-capabilities-within-postgresql/


This link paints a very good picture of the JSON support - my personal view from the information provided would be that it would be better to start off by using a simple postgres database with a single column for the json data with a key which we can do gets on and then if we find it's too slow or has other problems - introduce the complexity of another db

For the Mediator pickup protocol, we need to query the JSON based on key etc another field, IN Postgresql so we won't be able to index the field in the JSON blob, If the database grows that won't be the ideal solution to use Postgres, We have seen with PrismAgent, The message we stored to have the same problem.

…epo subject to the Developer Certificate of Origin (DCO), Version 1.1. cf49ac5 docs: Update status of the ADR for multi-tenancy routing [skip ci] (#547) fe5920b docs: ADR for mediator-message-storage [skip ci] (#530) 6452ec3 docs: ADR for multi-tenancy routing [skip ci] (#523) f3e705c test(pollux) : Added test for service (#507) b271ac0 test(pollux): Unit test added for JDBCRepo (#494) 948f2fc feat(prism-agent): Update the connect pollux and mercury to fix to use nimbus-jose-jwt 10.0.0-preview (#467) 8444057 feat(connect): Updated the mercury version for jwt (#466) 67e3622 feat(mercury): Added the resolver and update the nimbus libray to pre… (#465) adb7000 feat(prism-agent): Update pollux lib to 0.43.0 includes the nimbus-jo… (#460) 8c0070f feat(pollux): Update the nimbus-jose-jwt library with a patch 10.0.0-preview (#457) ee93880 feat(prism-agent): VerificationOptions are configurable for PrismAgent (#449) 301022a feat(prism-agent): Added connectionId to the request Presentation and deleted the readme files as they are moved here https://github.com/input-output-hk/atala-prism-building-blocks/tree/main/docs/docusaurus (#432) 8f52b12 feat(mercury): Code cleanup removed the Presentation Attachment and added to the Pollux (#365) 55569ed feat(prism-agent): Added new state PresentationVerificationFailed (#412) 94031f8 feat(pollux): Added new state PresentationVerificationFailed (#400) bff8d21 feat(connect): Updated version for mercury (#398) faac0a7 feat(pollux): Updated version for mercury (#399) 403f4c2 feat(mercury): ReportProblem Protocol updated with test and updated t… (#396) 409b673 feat(prsim-agent): pollux version updated (#392) 2765845 feat(pollux): Moving the check when accepting the credential to crea… (#391) 35704f4 feat(pollux): Added the Issued credetinal check at the ApiEndpoint (#387) 4f0b261 feat(prism-agent): Added challenge and domain and to protect agains r… (#364) 5349a0e feat(pollux): Added New Erorr (#363) abc007f feat(pollux) : Added challenge and domain (#361) da288bf feat(mercury) : Added challenge domain and presentation definition (#322) cf2b2d3 fix(mercury) : fix the test for attachment (#312) cc7c533 feat(prismagent): prism did creation and use updated protocol state (#291) 859c29e feat(pullux): dummy commit for release (#293) 174fe52 feat(pollux): added column credentionals_to_use and new state present… (#288) (#290) 25a1df1 feat:(pollux) added column credentionals_to_use and new state presentation generated (#289) 7c7f051 feat:(pollux) added column credentionals_to_use and new state present… (#288) ae74e20 feat(prism-agnet): Verifiable Credential (#281) a87dc17 feat(pollux): Added JWT verfication (#280) 7629eab feat(connect): updated the version for mercury (#246) a2af672 feat(pollux): updated the version for mercury (#245) b704abc feat(pollux): Added new state (#234) cd416bf feat(pollux): Added a new column to the table to store the issued credential faf3e6f added thid (#183) f7cba3b feat(prism-agent): implement Connect flow (#130) 7f0f412 feat(pollux): Verify PresentProof Protocol (#149) 96b0fbc feat(connect): removed the unwanted dependencies (#145) e4d4a41 feat(connect): Added Connect Lib (#138) 402248b feat(mercury): ATL-2287 Connection Protocol (#140) 46ef902 addedd the connect to lint (#143) b37b762 feat(mercury): [WIP] ATL-1741-Present-Proof-Protocol (#98) 21fe5bc feat(mercury): ATL-1740 issue credential (#64) bb4dfe5 ATL-1738-Aries-RFC-0023-DIDExchange-Protocol (#29) f160a51 ATL-1886-Fix issue (#54) c0fb7aa ATL-1893-DidPeerResolver (#36) 57ed209 Aries RFC 0434: Out-of-Band Protocol 1.1 (#22) Signed-off-by: Shailesh Patil <[email protected]>

mineme0110 added 2 commits May 18, 2023 16:36

docs: ADR for mediator-message-storage [skip ci]

271a340

docs: ADR for mediator-message-storage [skip ci]

3903663

mineme0110 requested a review from antonbaliasnikov as a code owner May 18, 2023 15:39

mineme0110 requested a review from a user May 18, 2023 15:39

mineme0110 requested review from yshyn-iohk and BS-IO as code owners May 18, 2023 15:39

atala-dev added the Type: Documentation label May 18, 2023

Merge branch 'main' into ATL-4215

660f165

yshyn-iohk approved these changes May 22, 2023

View reviewed changes

mineme0110 added 2 commits May 22, 2023 22:20

Updated the ADR for Storage requrements

1a568a3

Merge branch 'ATL-4215' of iohk.github.com:input-output-hk/atala-pris…

836c5c8

…m-building-blocks into ATL-4215

bvoiturier approved these changes May 23, 2023

View reviewed changes

mineme0110 merged commit fe5920b into main May 24, 2023

mineme0110 deleted the ATL-4215 branch May 24, 2023 11:57

ghost reviewed May 24, 2023

View reviewed changes

mineme0110 mentioned this pull request Aug 17, 2023

I, Shailesh Patil hereby sign-off-by all of my past commits to this r… #632

Merged

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: ADR for mediator-message-storage [skip ci] #530

docs: ADR for mediator-message-storage [skip ci] #530

mineme0110 commented May 18, 2023 •

edited

Loading

github-actions bot commented May 18, 2023 •

edited

Loading

github-actions bot commented May 18, 2023 •

edited

Loading

github-actions bot commented May 18, 2023 •

edited

Loading

ghost May 18, 2023

ghost May 18, 2023

ghost May 18, 2023

mineme0110 May 24, 2023 •

edited

Loading

ghost May 18, 2023

mineme0110 May 24, 2023

docs: ADR for mediator-message-storage [skip ci] #530

docs: ADR for mediator-message-storage [skip ci] #530

Conversation

mineme0110 commented May 18, 2023 • edited Loading

Overview

Checklist

My PR contains...

My changes...

Documentation

Tests

github-actions bot commented May 18, 2023 • edited Loading

E2E Test Results

github-actions bot commented May 18, 2023 • edited Loading

Atala PRISM Test Results

github-actions bot commented May 18, 2023 • edited Loading

ghost May 18, 2023

Choose a reason for hiding this comment

ghost May 18, 2023

Choose a reason for hiding this comment

ghost May 18, 2023

Choose a reason for hiding this comment

mineme0110 May 24, 2023 • edited Loading

Choose a reason for hiding this comment

ghost May 18, 2023

Choose a reason for hiding this comment

mineme0110 May 24, 2023

Choose a reason for hiding this comment

mineme0110 commented May 18, 2023 •

edited

Loading

github-actions bot commented May 18, 2023 •

edited

Loading

github-actions bot commented May 18, 2023 •

edited

Loading

github-actions bot commented May 18, 2023 •

edited

Loading

mineme0110 May 24, 2023 •

edited

Loading