docs: Proposal for FederatedCatalog Distribution and TargetNodeDirectory #1555

bmg13 · 2024-09-16T13:17:10Z

WHAT

Decision Record for new proposal for FederatedCatalog distribution (with Tractus-X) and the TargetNodeDirectory (TND).

The TND initial proposal was defined in this PR and continue in the current one.

Related with #1585

[edit @paullatzelsperger]
Closes eclipse-tractusx/sig-release#736

github-actions · 2024-09-24T02:00:40Z

This pull request is stale because it has been open for 7 days with no activity.

paullatzelsperger

Generally, before going into details I want to clarify a few things. A decision record should document a Decision (not options) and have the following structure:

Decision: outline in a succinct and concise manner, in one or two sentences, what was decided.
Rationale: outline the thought process how that decision was reached, maybe mention one or two alternatives and why they were not chosen. Motivate your decision.
Approach: how the decision can be implemented. This should also outline possible limitations.

Specifically, the FC is not a "standalone service". What you probably mean is a separate runtime (docker image, runnable JAR file) and thus a separate deployment in the Tx-EDC Helm Chart, like the data plane. It is extremely important to use precise wording

docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md

ndr-brt · 2024-09-26T12:56:47Z

docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md

+The Federated Catalog aims at providing the aggregation of catalogs from multiple participants in a centralized point to reduce latency.
+
+Choosing a solution that decouples from Control Plane (like the one used for the Data Plane) and able to be scalable will future-proof the Federated Catalog as a feature and embraces wider usage. 
+Like so, having a Federated Catalog service, scalable on its own based on user needs, and not on Control Plane, allows the best resource management in the long term. It also enables adding features to the Federated Catalog without the need to change the Control Plane logic (assuming no API usage change).


the first part is already described by the previous sentence, the second one is not 100% correct, because it is actually possible to add features to the control plane without change its "logic" (depending on what do you mean with "logic", but generally speaking that's what the EDC modular architecture is about)

This was bad phrasing on my side, I was aiming at mentioned the goal of having independent updates. However, in the previous paragraph was already mentioned the decoupling this approach promotes so it (like the scalability) can be seen as repetitive information.
Removed in here.

in any case also the "indipendent updates" is not something we want to achieve with this decision, because we will release the FC together with CP and DP, otherwise we would put it in a separate chart.

I see. I wanted to highlight, as an example, that if a change in the Federated Catalog Cache or similar should be transparent for other instances (like CP or DP). And it would be, I believe, but updating to a new version when all are in tractus-edc charts then "independent update" is not accurate.
Thanks!

docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md

bmg13 · 2024-09-30T13:59:32Z

Considering this suggestion, the Decision Record proposals for the Distribution and the TargetNodeDirectory were merged in one since they related within same scope.

docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md

… into dr_proposal_federated_catalog_distribution

ndr-brt

please add some more details on the approach part, it is still too vague, it would be good to see paths, methods, request/response body structure and so on.

docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md

github-actions · 2024-11-07T01:59:36Z

This pull request is stale because it has been open for 7 days with no activity.

bmg13 · 2024-11-07T14:00:15Z

sorry for reping, but @jimmarino @ndr-brt @paullatzelsperger @wolf4ood can someone look at this pr, please?

jimmarino · 2024-11-07T14:30:45Z

sorry for reping, but @jimmarino @ndr-brt @paullatzelsperger @wolf4ood can someone look at this pr, please?

Can you resolve the outstanding comments and we can re-review?

… into dr_proposal_federated_catalog_distribution

bmg13 · 2024-11-08T17:09:21Z

sorry for reping, but @jimmarino @ndr-brt @paullatzelsperger @wolf4ood can someone look at this pr, please?

Can you resolve the outstanding comments and we can re-review?

Sorry about it, all seems addressed now, thank you!

jimmarino · 2024-11-11T12:54:55Z

sorry for reping, but @jimmarino @ndr-brt @paullatzelsperger @wolf4ood can someone look at this pr, please?

Can you resolve the outstanding comments and we can re-review?

Sorry about it, all seems addressed now, thank you!

The comments are not resolved in GitHub, which makes it difficult for reviewers to track what was applied. Please do that before requesting a re-review.

bmg13 · 2024-11-13T09:51:00Z

sorry for reping, but @jimmarino @ndr-brt @paullatzelsperger @wolf4ood can someone look at this pr, please?

Can you resolve the outstanding comments and we can re-review?

Sorry about it, all seems addressed now, thank you!

The comments are not resolved in GitHub, which makes it difficult for reviewers to track what was applied. Please do that before requesting a re-review.

I usually leave the creator of the comment to resolve it. But I understand, resolved it in the meantime all that are resolved, left a couple open since is ongoing discussion

docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md

paullatzelsperger · 2024-11-13T10:15:39Z

docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md

+
+Since the Federated Catalog will be a standalone runtime, the Tractus-X EDC Connector Helm charts will be updated to include the Federated Catalog as a separated deployment. The update will include the creation of a specific `deployment-federatedcatalog.yaml`, similar [to this one](https://github.com/eclipse-tractusx/tractusx-edc/blob/a263bf71a110245657131509d4b37d058a1d220d/charts/tractusx-connector-azure-vault/templates/deployment-dataplane.yaml#L47) (for `ingress` and `hpa` as well), for different scenarios (InMemory, PostreSQL, etc.). This results in added configuration complexity.
+
+To enable the Federated Catalog flow, please [see this table](https://github.com/eclipse-tractusx/tractusx-edc/blob/75bdacbad43e2cad352204ea28a359c6aac7adea/docs/development/management-domains/README.md#enable-and-configure-the-crawler-subsystem).


spurious. how we configure an FCC is not relevant for this Decision Record.

I understand, kept the first paragraph since believe it to add relevant context here, but the enabling config was removed.
Removed in here.

docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md

paullatzelsperger · 2024-11-13T10:19:36Z

docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md

+
+Some limitations of this TargetNodeDirectory solution are:
+- Each partner must have the DID's beforehand. If a new Partner is registered and an existing partner would want their catalog, the DID (or BPN) of the new partner must be obtained first and added through the new extension API;
+- Deal with the overhead an additional persistence store;


what does this mean? how is this a limitation? If anything, it means that we'll need an in-memory and an SQL variant of this store.

You're right, this will be a limitation if a SQL store is used, but InMem or SQL options are provided as other cases and this is not a valid limitation here.
Removed in here.

paullatzelsperger · 2024-11-13T10:23:44Z

docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md

+    {
+        "bpn": "BPNL000000000002",
+        "connectorEndpoint": [
+            "https://connector2/api/v1/dsp",


i would caution against using the BPN as primary key. Moving away from BPNs and toward DIDs fully is currently being discussed and this would cause problems with the data model later on. In addition, conflating business keys (such as BPNs) and database keys is somewhat of an antipattern.
There always is the possibility to wrap the TargetNode in another class and use that class as database entity.

paullatzelsperger · 2024-11-13T10:26:59Z

Please lets be very specific with the wording:

the "Federated Catalog" is simply a unified list of catalogs
the "Federated Catalog Cache" is what contains the federated catalog.

With that said, IMO the new runtime should be called "Federated Catalog Cache", not "Federated Catalog". There may also be "Federated Catalog Servers" in the future (-> Management Domains).

… into dr_proposal_federated_catalog_distribution

bmg13 · 2024-11-13T15:06:34Z

Please lets be very specific with the wording:
* the "Federated Catalog" is simply a unified list of catalogs

* the "Federated Catalog Cache" is what contains the federated catalog.
With that said, IMO the new runtime should be called "Federated Catalog Cache", not "Federated Catalog". There may also be "Federated Catalog Servers" in the future (-> Management Domains).

Thank you for that, it makes sense.
Changed in here.

github-actions · 2024-11-26T02:04:49Z

This pull request is stale because it has been open for 7 days with no activity.

github-actions · 2024-12-03T02:07:00Z

This pull request was closed because it has been inactive for 7 days since being marked as stale.

bmg13 · 2024-12-09T11:22:37Z

@jimmarino @paullatzelsperger @ndr-brt can this PR be reopened, please?

sonarqubecloud · 2024-12-12T06:22:46Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

ndr-brt

I already approved, but to be honest I lost all the further discussion. My suggestion is to close this and re-open it as a fresh new PR, that will remove a lot of noise from the previous reviews (95 comments are A LOT! :) )

bmg13 · 2024-12-16T11:14:41Z

I already approved, but to be honest I lost all the further discussion. My suggestion is to close this and re-open it as a fresh new PR, that will remove a lot of noise from the previous reviews (95 comments are A LOT! :) )

Will follow your suggestion and create a new one (cleaner) referencing this one so people are aware :)

Edit: Created the new PR #1718

Doc: Proposal for FederatedCatalog distribution.

8b7775f

bmg13 marked this pull request as ready for review September 16, 2024 13:27

github-actions bot added the stale label Sep 24, 2024

paullatzelsperger requested changes Sep 24, 2024

View reviewed changes

paullatzelsperger added documentation Improvements or additions to documentation and removed stale labels Sep 24, 2024

Restructure based on PR.

40f9a94

bmg13 requested a review from paullatzelsperger September 24, 2024 14:05

ndr-brt self-requested a review September 26, 2024 10:46

ndr-brt reviewed Sep 26, 2024

View reviewed changes

Changes from PR.

a452459

bmg13 requested a review from ndr-brt September 26, 2024 14:26

Small doc improvements

960a5f9

paullatzelsperger reviewed Sep 30, 2024

View reviewed changes

ndr-brt mentioned this pull request Sep 30, 2024

docs: Proposal for a TargetNodeDirectory in Tractus-X. #1556

Closed

bmg13 added 2 commits September 30, 2024 12:12

Changes from PR.

48b1900

Merge both Distribution and TargetNodeDirectory decision proposals.

4247446

bmg13 changed the title ~~docs: Proposal for FederatedCatalog distribution~~ docs: Proposal for FederatedCatalog Distribution and TargetNodeDirectory Sep 30, 2024

bmg13 requested a review from paullatzelsperger September 30, 2024 14:02

ndr-brt reviewed Oct 7, 2024

View reviewed changes

docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md Outdated Show resolved Hide resolved

docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md Outdated Show resolved Hide resolved

bmg13 added 2 commits October 7, 2024 15:01

Merge branch 'main' of https://github.com/eclipse-tractusx/tractusx-edc…

c50c2c9

… into dr_proposal_federated_catalog_distribution

Changes from PR.

b851b92

bmg13 requested a review from ndr-brt October 7, 2024 14:58

ndr-brt reviewed Oct 7, 2024

View reviewed changes

docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md Outdated Show resolved Hide resolved

docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md Show resolved Hide resolved

Changes from PR.

b4b00b0

bmg13 requested a review from ndr-brt October 7, 2024 17:39

Update retrieve path.

473e800

ndr-brt approved these changes Oct 8, 2024

View reviewed changes

bmg13 requested a review from ndr-brt October 29, 2024 14:23

github-actions bot removed the stale label Oct 30, 2024

github-actions bot added the stale label Nov 7, 2024

github-actions bot removed the stale label Nov 8, 2024

bmg13 added 2 commits November 8, 2024 16:57

Merge branch 'main' of https://github.com/eclipse-tractusx/tractusx-edc…

59f6746

… into dr_proposal_federated_catalog_distribution

Changes from PR.

b082df1

paullatzelsperger reviewed Nov 13, 2024

View reviewed changes

bmg13 added 2 commits November 13, 2024 14:45

Merge branch 'main' of https://github.com/eclipse-tractusx/tractusx-edc…

a7e1bb4

… into dr_proposal_federated_catalog_distribution

Changes from PR.

8a3f859

bmg13 requested a review from paullatzelsperger November 13, 2024 15:37

github-actions bot added the stale label Nov 26, 2024

github-actions bot closed this Dec 3, 2024

paullatzelsperger reopened this Dec 12, 2024

ndr-brt reviewed Dec 12, 2024

View reviewed changes

github-actions bot removed the stale label Dec 13, 2024

bmg13 closed this Dec 16, 2024

bmg13 mentioned this pull request Dec 16, 2024

docs: Proposal for FederatedCatalog Distribution and TargetNodeDirectory #1718

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: Proposal for FederatedCatalog Distribution and TargetNodeDirectory #1555

docs: Proposal for FederatedCatalog Distribution and TargetNodeDirectory #1555

bmg13 commented Sep 16, 2024 •

edited by paullatzelsperger

Loading

github-actions bot commented Sep 24, 2024

paullatzelsperger left a comment •

edited

Loading

ndr-brt Sep 26, 2024

bmg13 Sep 26, 2024

ndr-brt Sep 27, 2024

bmg13 Sep 27, 2024

bmg13 commented Sep 30, 2024

ndr-brt left a comment

github-actions bot commented Nov 7, 2024

bmg13 commented Nov 7, 2024

jimmarino commented Nov 7, 2024

bmg13 commented Nov 8, 2024

jimmarino commented Nov 11, 2024

bmg13 commented Nov 13, 2024

paullatzelsperger Nov 13, 2024

bmg13 Nov 13, 2024

paullatzelsperger Nov 13, 2024

bmg13 Nov 13, 2024

paullatzelsperger Nov 13, 2024

paullatzelsperger commented Nov 13, 2024

bmg13 commented Nov 13, 2024

github-actions bot commented Nov 26, 2024

github-actions bot commented Dec 3, 2024

bmg13 commented Dec 9, 2024

sonarqubecloud bot commented Dec 12, 2024

ndr-brt left a comment

bmg13 commented Dec 16, 2024 •

edited

Loading


		Since the Federated Catalog will be a standalone runtime, the Tractus-X EDC Connector Helm charts will be updated to include the Federated Catalog as a separated deployment. The update will include the creation of a specific `deployment-federatedcatalog.yaml`, similar [to this one](https://github.com/eclipse-tractusx/tractusx-edc/blob/a263bf71a110245657131509d4b37d058a1d220d/charts/tractusx-connector-azure-vault/templates/deployment-dataplane.yaml#L47) (for `ingress` and `hpa` as well), for different scenarios (InMemory, PostreSQL, etc.). This results in added configuration complexity.

		To enable the Federated Catalog flow, please [see this table](https://github.com/eclipse-tractusx/tractusx-edc/blob/75bdacbad43e2cad352204ea28a359c6aac7adea/docs/development/management-domains/README.md#enable-and-configure-the-crawler-subsystem).

docs: Proposal for FederatedCatalog Distribution and TargetNodeDirectory #1555

docs: Proposal for FederatedCatalog Distribution and TargetNodeDirectory #1555

Conversation

bmg13 commented Sep 16, 2024 • edited by paullatzelsperger Loading

WHAT

github-actions bot commented Sep 24, 2024

paullatzelsperger left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bmg13 commented Sep 30, 2024

ndr-brt left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 7, 2024

bmg13 commented Nov 7, 2024

jimmarino commented Nov 7, 2024

bmg13 commented Nov 8, 2024

jimmarino commented Nov 11, 2024

bmg13 commented Nov 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paullatzelsperger commented Nov 13, 2024

bmg13 commented Nov 13, 2024

github-actions bot commented Nov 26, 2024

github-actions bot commented Dec 3, 2024

bmg13 commented Dec 9, 2024

sonarqubecloud bot commented Dec 12, 2024

Quality Gate passed

ndr-brt left a comment

Choose a reason for hiding this comment

bmg13 commented Dec 16, 2024 • edited Loading

bmg13 commented Sep 16, 2024 •

edited by paullatzelsperger

Loading

paullatzelsperger left a comment •

edited

Loading

bmg13 commented Dec 16, 2024 •

edited

Loading