Backend worker calls unknown enpoint /api/v1/validation for Clearing House Notarization Check #819
Assignees
Labels
Comments
|
@gomezbc which component are you using for clearinghouse? it seems like the version or component doesn't have the endpoint. but it's needed from the portal backend to be able to proceed the process successfully |
|
@Phil91 I'm using Which version needs the back-end? |
|
@gomezbc I recommend that you join our matrix chat https://matrix.to/#/#tractusx-portal:matrix.eclipse.org , all people that can support with the portal are part of it and the communication is usually faster. to come back to your question, which version of the backend are you using? than we can clarify which clearinghouse version is needed |
|
@Phil91 thanks for the link to the matrix chat. I'm using version 2.0.0 of the portal helm chart, which uses v2.0.0 of the backend |
|
@gomezbc I'll look into it and come back to you asap |
|
@Phil91 Thanks a lot! Appreciate it. Looking forward to your update! 😃 |
|
@gomezbc sorry that it took so long, currently we can't support the validate clearinghouse step with foss applications, we are working on a toggle to disable the calls to the clearinghouse and self description, but that will be available in the future. for now we could solve your issue by running a custom sql script on the database to skip the clearinghouse and self description process. WITH applications AS (
SELECT distinct ca.id as Id, ca.checklist_process_id as ChecklistId
FROM portal.company_applications as ca
JOIN portal.application_checklist as ac ON ca.id = ac.application_id
where ca.application_status_id = 7 and ac.application_checklist_entry_type_id = 6 and ac.application_checklist_entry_status_id = 4
),
updated AS (
UPDATE portal.application_checklist
SET application_checklist_entry_status_id = 3
WHERE application_id IN (SELECT Id FROM applications)
RETURNING *
)
INSERT INTO process_steps (id, process_step_type_id, process_step_status_id, date_created, date_last_changed, process_id, message)
SELECT gen_random_uuid(), 12, 1, now(), NULL, a.ChecklistId, NULL
FROM applications a; |
|
@Phil91 Thanks for the update and for the workaround with the custom SQL script. Awesome work! 👏 |
|
Hey guys, im trying to achieve a similar setup as @gomezbc but I'm stuck in an earlier stage of the company onboarding process:
I used the same chart and the same values file (just updated the mail server values) as mentioned in the first post of this issue ( #819 (comment) ) but the process worker shows the following error message: Additionally, the logs of the bpdm services (cleaning-service, gate, pool) show error messages which indicate the services are not able to communicate with each other due to a wrong client/security setup from the values file: Cleaning Service: Gate: Pool: I tried many different client/security settings for the bpdm when running along with the umbrella services and finally ended up disabling the authentication complete by using the "no-auth" spring profile. At least, this resolved the "401 unauthorized" Problem but i was still stuck in the "BPN Creation" Step. @gomezbc Did I miss something from your description in order to reach your stage of the onboarding? @Phil91 Does a helm chart exist which includes umbrella and bpdm services and is tested regarding version compatibility of the services? I tried many different variations of services / versions / content of the values files but not luck in onboarding a company Thanks for your support. |
|
Hi @ipa-big , to achieve BPDM integration i had to modify the centralidp keycloak configuration by adding the keyclaok clients BPDM components needed. Because BPDM chart uses the following keycloak config https://github.com/eclipse-tractusx/bpdm/blob/main/charts/bpdm/templates/keycloakRealm.tpl , as a workaround i merged the centralidp and BPDM keycloak config using This is the output json: CX-Central-realm.json How to replace the original centralidp with the new configuration:
eval $(minikube docker-env)
docker build -t tractusx/umbrella-init-container:0.1.0-init .Check authentication eventsYou can check if the client authentication works by checking the events in centralidp: I hope it can help you. If that's not the case, i think that there is another issue with the creation of a company business partner which i am trying to replicate, as soon as i find the issue i will reply this post. |
|
Perfect. The Authentication of the BPDM Services works. Unfortunately, the error in the onboarding process remains: Portal-Frontend: Process Worker Gate: I assume the second error regarding the POST request to /v6/golden-record-tasks/state/search comes from a misconfiguration of service urls in the values file or a missing/misconfigured ingress rule. But the first one i have no clue where this is coming from |
|
I have just finished debugging the chart values and i have realized that the orchestrator url is not configured. That's the reason for the connection refused error. These are the correct values for bpdm: bpdm:
enabled: true
postgres:
enabled: true
fullnameOverride: &bpdm-postgres-host bpdm-postgres
auth:
database: bpdm
username: bpdm
# Change for production
password: bpdm
keycloak:
enabled: false
bpdm-gate:
enabled: true
postgres:
enabled: false
fullnameOverride: bpdm-postgres
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
hosts:
- host: partners-gate.tx.test
paths:
- path: /
pathType: Prefix
springProfiles:
- auth
- pool-auth
applicationConfig:
bpdm:
client:
pool:
base-url: http://partners-pool.tx.test/
registration:
client-id: sa-cl7-cx-5
orchestrator:
base-url: http://partners-orchestrator.tx.test
security:
auth-server-url: http://centralidp.tx.test/auth
realm: CX-Central
client-id: Cl16-CX-BPDMGate
datasource:
host: *bpdm-postgres-host
applicationSecrets:
bpdm:
client:
orchestrator:
registration:
client-secret: &gateOrchSecret gate_orch_client_secret
pool:
registration:
client-secret: &gatePoolSecret bWSck103qNJ0jZ1LVtG9mUAlcL7R5RLg
bpdm-pool:
enabled: true
postgres:
enabled: false
fullnameOverride: bpdm-postgres
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
hosts:
- host: partners-pool.tx.test
paths:
- path: /
pathType: Prefix
springProfiles:
- auth
applicationConfig:
bpdm:
security:
auth-server-url: http://centralidp.tx.test/auth
realm: CX-Central
client-id: Cl7-CX-BPDM
client:
orchestrator:
base-url: http://partners-orchestrator.tx.test/
datasource:
host: *bpdm-postgres-host
applicationSecrets:
bpdm:
client:
orchestrator:
registration:
client-secret: &poolOrchSecret pool_orch_client_secret
bpdm-cleaning-service-dummy:
enabled: true
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
hosts:
- host: partners-cleaning.tx.test
paths:
- path: /
pathType: Prefix
springProfiles:
- auth
applicationConfig:
bpdm:
client:
orchestrator:
base-url: http://partners-orchestrator.tx.test/
provider:
issuer-uri: http://centralidp.tx.test/auth/realms/CX-Central
datasource:
host: *bpdm-postgres-host
applicationSecrets:
bpdm:
client:
orchestrator:
registration:
client-secret: &dummyOrchSecret dummy_orch_client_secret
bpdm-orchestrator:
enabled: true
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
hosts:
- host: partners-orchestrator.tx.test
paths:
- path: /
pathType: Prefix
springProfiles:
- auth
applicationConfig:
bpdm:
security:
auth-server-url: http://centralidp.tx.test/auth
datasource:
host: *bpdm-postgres-hostWarning Make sure to add the new hostnames to your etc/hosts file. |
|
Perfect, the errors in the logs of the bpdm services are gone now. Only the request from worker to gate (POST /v6/sharing-state/ready) which results in a status code 400 is there and I'm still stuck in the bpn creation process. I'm wondering why you are not facing this issue. |
|
Try running these request to get the progress status: TOKEN=$(curl -s -X POST 'http://centralidp.tx.test/auth/realms/CX-Central/protocol/openid-connect/token' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-d 'scope=openid' \
-d 'grant_type=client_credentials' \
-d 'client_id=sa-cl7-cx-5' \
-d 'client_secret=bWSck103qNJ0jZ1LVtG9mUAlcL7R5RLg' | jq -r '.access_token')
curl -X GET 'http://partners-gate.tx.test/v6/sharing-state' \
-H "Authorization: Bearer $TOKEN" | jq{
"totalElements": 1,
"totalPages": 1,
"page": 0,
"contentSize": 1,
"content": [
{
"externalId": "6487fd87-3ed9-45eb-bc3b-17b9ffaaf30d",
"sharingStateType": "Error",
"sharingErrorCode": "SharingProcessError",
"sharingErrorMessage": "Alternative Address has no country",
"sharingProcessStarted": "2024-07-07T17:54:31.078627",
"taskId": "4bcd4947-c876-4c6e-883a-2f09b6fcc19e"
}
]
}If your output is like this, it's because the process worker doesn't set |
@gomezbc this issue should be solved with the 5.0.2 patch from BPDM team, see #801 |
Hi @nicoprow, as discussed, could you please provide the bpdm configuration which doesn't require a change to the centralidp CX-Central realm? |
|
According to the comment of @evegufy
i updated the version of the BPDM from 5.0.1 to 5.02. After Inviting a new Company, Inserting the company information in the registration frontend and approving the information i receive the following when I do
So it shows the error "Alternative Address has no country". But I'm pretty sure that this was not the case when I used version 5.0.1 of BPDM. So it looks like this error came up with version 5.0.2. The Process Worker still shows the same error as before: Gate log shows: |
|
Hi @ipa-big ,I have also updated to 5.0.2. Although I continue getting 400 error, the BPNL is successfully created (you may have to wait a little to be created). Check it running this script TOKEN=$(curl -s -X POST 'http://centralidp.tx.test/auth/realms/CX-Central/protocol/openid-connect/token' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-d 'scope=openid' \
-d 'grant_type=client_credentials' \
-d 'client_id=sa-cl7-cx-5' \
-d 'client_secret=bWSck103qNJ0jZ1LVtG9mUAlcL7R5RLg' | jq -r '.access_token')
curl -X GET 'http://partners-pool.tx.test/v6/legal-entities' \
-H "Authorization: Bearer $TOKEN"| jqIf you get a response with the BPNL, you can use the following SQL script to assign manually the BPN continue to the next step. Host: umbrella-portal-backend-postgresql Port: 5432 User: postgres Password: dbpasswordportal Launch a Important Replace WITH applications AS (
SELECT distinct ca.id as Id, ca.checklist_process_id as ChecklistId, ca.company_id as CompanyId
FROM portal.company_applications as ca
JOIN portal.application_checklist as ac ON ca.id = ac.application_id
where ca.application_status_id = 7 and ac.application_checklist_entry_type_id = 2 and ac.application_checklist_entry_status_id = 4
),
updated_company AS (
UPDATE portal.companies
SET business_partner_number = 'YOUR_BPNL'
WHERE id IN (SELECT CompanyId FROM applications)
RETURNING *
)
UPDATE portal.application_checklist
SET date_last_changed = now(), application_checklist_entry_status_id = 3, comment = null
WHERE application_id IN (SELECT Id FROM applications) AND application_checklist_entry_type_id = 2;
-- Create the process that continues with the registration progress
WITH applications AS (
SELECT distinct ca.id as Id, ca.checklist_process_id as ChecklistId, ca.company_id as CompanyId
FROM portal.company_applications as ca
JOIN portal.application_checklist as ac ON ca.id = ac.application_id
where ca.application_status_id = 7 and ac.application_checklist_entry_type_id = 3 and ac.application_checklist_entry_status_id = 1
)
INSERT INTO portal.process_steps (id, process_step_type_id, process_step_status_id, date_created, date_last_changed, process_id, message)
SELECT gen_random_uuid(), 5, 1, now(), NULL, a.ChecklistId, NULL
FROM applications a; |
|
Ah ok. Now the "BPN Creation" and the "Identity Wallet Creation" Step are in "approved" state. I had the problem that
always returned an empty array What worked for me:
Now the result looks like this: In this state i tried @Phil91's sql query (#819 (comment)) but it did not change anything (clearing house and self description are still in to-do state). I will try to add the clearing house service to my setup in order to make the last two steps, too |
|
For the @Phil91 sql query to work, the clearing house step has to fail first, so that You can achieve that by setting WITH applications AS (
SELECT distinct ca.id as Id, ca.checklist_process_id as ChecklistId
FROM portal.company_applications as ca
JOIN portal.application_checklist as ac ON ca.id = ac.application_id
where ca.application_status_id = 7 and ac.application_checklist_entry_type_id = 6 and ac.application_checklist_entry_status_id = 1
),
updated AS (
UPDATE portal.application_checklist
SET application_checklist_entry_status_id = 3
WHERE application_id IN (SELECT Id FROM applications)
RETURNING *
)
INSERT INTO portal.process_steps (id, process_step_type_id, process_step_status_id, date_created, date_last_changed, process_id, message)
SELECT gen_random_uuid(), 12, 1, now(), NULL, a.ChecklistId, NULL
FROM applications a; |
Hi @nicoprow could you please provide an update? |
|
Right now I'm trying different combinations of Portal and BPDM versions in order to get a successful "BPN Creation" step when onboarding a company. For now I've tried:
Unfortunately, no combination works. E.g. Portal 2.0.0 / BPDM 4.0.4 gets stuck here:
Is there a compatibility matrix for Portal/BPDM or has the onboarding of companies never been working at all? |
|
When I use BDPM Chart Version 4.0.4 it looks like no BPN gets created at all. When I use Version 5.0.2 i get the same result as you describe in #830. Maybe the new Chart version fixes some of the issues: https://github.com/eclipse-tractusx/bpdm/releases/tag/bpdm-5.1.0 |
|
Short Update from current state of work. Finally, I do have a setup which completes the first three stages of the checklist (manually approve, BPN creation, Wallet creation) without any additional, manual actions from my side. I used the hint from here #830 (comment) to overcome the "Bad Request" response from the BPDM. So no additional request to the BPDM and no SQL queries are required anymore. Now I tried to use the toggle feature (mentioned here: #793 and here #819 (comment)) for connecting to clearing house and sd factory to also complete the last two stages of the onboarding checklist. Unfortunately, the last two stages are still failing:
Process Worker log shows I've checked out version "portal-2.1.0-RC1" of the portal repo and used the chart in my overall chart like this: In order to disable communication between portal and clearinghouse/sd-factory, i modified all tag information (changed to v2.1.0-RC1) and added the "clearinghouseConnectDisabled" property in portal.backend.processworker like this: With this setup I'm still stuck in the clearing house step. When I apply the sql query mentioned by @Phil91 (see here #819 (comment)) then "clearing house" and "self description" are in approved state but when I log in with the company user I do get the following message from the portal:
Is there still a bug in the toggle feature or did i miss something in setting up the portal and the bpdm? |
|
Ok, i don't know why, but after rechecking this messages does not show up anymore:
After that I create a tech-user for the onboarded company and try to register a connector via
but this fails because the value of "self_description_document_id" (in the db of the portal-backend) of the new onboarded company is null (I guess because I skipped the clearing house and sd factory step during registration). After adding a value for the self description I'm finally able to register a connector. |
Hi @ipa-big yes, the versions aren't compatible Portal 2.0.0 is R24.05 and BPDM 4.0.x is R24.03 you did some awesome progress! just mentioning that we have a weekly open meeting around such issues, you're very welcome to join https://eclipse-tractusx.github.io/community/open-meetings#Umbrella%20Helm%20Chart%20Sync |
This was referenced Jul 18, 2024
I created a working setup for BPDM inside the Umbrella chart with a new Hotfix 5.0.3: eclipse-tractusx/tractus-x-umbrella#108 that you can use for reference. The BPDM golden record process works as expected when using the BPDM APIs directly. I also put in some comments for why the current 5.0.2 deployment eclipse-tractusx/tractus-x-umbrella#106 may not work and added some corrections. However, I experienced that at least the Portal's Partner Network contains outdated endpoint urls for the Pool and is unable to query business partners there. For the registration process I'm unsure if there is similar issues. |
@nicoprow Great that you created the working setup! Just quickly clarfying on what you mentioned regarding outdated endpoints in the portal: I assume you tested with the version of the portal which is currently still in the main branch of the umbrella repo, which is the portal version 1.8.0 for the release 24.03, that one is expected to be not compatible with the BPDM version for 24.05. |
Merged
2 tasks
|
Hi @gomezbc I'll close the issue now and I created eclipse-tractusx/tractus-x-umbrella#115 to follow up on more proper documentation. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm trying to set up a local data-space to test company boarding process using the portal.
Current Behavior
After the company has uploaded their data and, Data Validation, BPN Creation and Identity Wallet Creation have been approved. In the clearing house step, the process tries to call the following endpoint of the Clearing House
http://compliance.tx.test/api/v1/validationand fails with 404 status code.The documentation shows that the call has to be made to the notarization component. But this service doesn't have such endpoint. Neither gaia-x registry nor compliance services. Which are the mandatory components.
Expected Behavior
I don't know which should be the expected behavior.
Steps To Reproduce
I am using the following helm charts version:
Values used are present inside this zip: values.zip
For the GXDCH im using
v1And, i'm using the following domain names pointing to a local minikube cluster:
The text was updated successfully, but these errors were encountered: