Update helm-test post install when upgrading EDC from 0.7.1 to 0.7.7 #159

matbmoser · 2024-11-06T15:30:32Z

Task

We need to update the test and post install.

Relates to #134 and #153

evegufy · 2024-11-06T18:38:25Z

it's not about some test that is failing, it's about a failing post install job which is supposed to be importing/seeding the test data https://github.com/eclipse-tractusx/tractus-x-umbrella/tree/umbrella-2.0.2/charts/tx-data-provider/resources

kss682 · 2024-11-06T18:46:54Z

  File "/opt/scripts/upload.py", line 253, in create_registry_asset
    print_response(response_)
Policy default-policy already exists. Skipping creation.
Create policy test on EDC http://umbrella-dataprovider-edc-controlplane:8081/management/v2/policydefinitions
<Response [409]>
[{"message":"Policy with ID test already exists","type":"ObjectConflict","path":null,"invalidValue":null}]
http://umbrella-dataprovider-edc-controlplane:8084
http://umbrella-dataprovider-edc-controlplane:8081
Query Catalog for registry asset http://umbrella-dataprovider-edc-controlplane:8081/management/v2/catalog/request
<Response [500]>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 Internal Server Error</title>
</head>
<body><h2>HTTP ERROR 500 Internal Server Error</h2>
<table>
<tr><th>URI:</th><td>/management/v2/catalog/request</td></tr>
<tr><th>STATUS:</th><td>500</td></tr>
<tr><th>MESSAGE:</th><td>Internal Server Error</td></tr>
<tr><th>SERVLET:</th><td>EDC-management</td></tr>
</table>

</body>
</html>

  File "/opt/scripts/upload.py", line 168, in print_response
    raise Exception("Failed to call service")
Exception: Failed to call service

The error on the post install job

kss682 · 2024-11-06T18:54:34Z

For the catalog request payload.
The only difference I could see is, missing key "@type".
catalog doc 0.7.3
code

evegufy · 2024-11-06T20:25:28Z

@kss682 hadn't seen that log yet, I get the helm test point in the context of the post install job now.
I noticed another thing, not sure if relevant:

For some reason, I can't find the catalog request endpoint in 0.7.3 and once it's available again - from 0.7.4 onwards - it's v3, not v2 (as in 0.7.1) anymore ... 0.7.7

Maybe it's worth checking the versions of all endpoints here

kss682 · 2024-11-07T13:16:13Z

@evegufy @matbmoser

Query Catalog for registry asset http://umbrella-dataprovider-edc-controlplane:8081/management/v3/catalog/request
<Response [500]>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 Internal Server Error</title>
</head>
<body><h2>HTTP ERROR 500 Internal Server Error</h2>
<table>
<tr><th>URI:</th><td>/management/v3/catalog/request</td></tr>
<tr><th>STATUS:</th><td>500</td></tr>
<tr><th>MESSAGE:</th><td>Internal Server Error</td></tr>
<tr><th>SERVLET:</th><td>EDC-management</td></tr>
</table>

</body>

The issue persists even after changing it to v3.
It seems like the issue is because of the this breaking change eclipse-edc/Connector#4185

evegufy · 2024-11-07T14:02:52Z

Hi @kss682 thank you for testing v3, I was just guessing.

Do you need support for upgrading the post install job for seeding the test data to the breaking change?

If so, I'm not so deep into EDC stuff. @matbmoser can you or @saudkhan116 support here? If not, there are also some other committers we could ask, I think.

kss682 · 2024-11-07T14:52:02Z

@evegufy I was reading into the commit, not sure if this would be the only change.
"edc:operandLeft": "https://w3id.org/edc/v0.0.1/ns/type" to "edc:operandLeft": {"@id": "https://w3id.org/edc/v0.0.1/ns/type"} in here.
All such occurrences would need to be changed.

evegufy · 2024-11-08T08:34:35Z

@kss682 ok good, will you test that change? Once you tested it and it still doesn't work, we'd ask for support from someone.

kss682 · 2024-11-08T13:06:55Z

@evegufy Hello, I did try testing with the changes but the error persists.
shall post the same in the chat room.

evegufy · 2024-11-08T13:30:32Z

Hi @kss682 as we have this issue already, please post the error here as comment.
I'll write a quick message in the matrix chat thread that all should join the discussion here.

kss682 · 2024-11-08T13:39:39Z

Summary: We upgraded the tractusx-edc to 0.7.7 and tried testing the umbrella helm chart, we could see that the post install test job fails post-install-test script.
From the discussions, it was mentioned that this could be due to a breaking change in this PR. Hence we modified the "edc:operandLeft": "https://w3id.org/edc/v0.0.1/ns/type" to "edc:operandLeft": {"@id": "https://w3id.org/edc/v0.0.1/ns/type"} in here.
But the errors persists as before.
controlplane logs

Similar issue : eclipse-edc/Connector#4160

ndr-brt · 2024-11-08T14:10:29Z

Summary: We upgraded the tractusx-edc to 0.7.7 and tried testing the umbrella helm chart, we could see that the post install test job fails post-install-test script. From the discussions, it was mentioned that this could be due to a breaking change in this PR. Hence we modified the "edc:operandLeft": "https://w3id.org/edc/v0.0.1/ns/type" to "edc:operandLeft": {"@id": "https://w3id.org/edc/v0.0.1/ns/type"} in here. But the errors persists as before.
Similar issue : eclipse-edc/Connector#4160

@kss682 could you paste the actual json used for the call and copy and paste the logs in a triple tick block instead of screenshots?

kss682 · 2024-11-08T14:16:36Z

Payload:

        payload_ = {
            "@context": edc_context(),
            "@type": "CatalogRequest",
            "edc:protocol": "dataspace-protocol-http",
            "edc:counterPartyAddress": f"{edc_url_}/api/v1/dsp",
            "edc:counterPartyId": f"{edc_bpn_}",
            "edc:querySpec": {
                "edc:filterExpression": {
                    "@type": "edc:Criterion",
                    "edc:operandLeft": {"@id": "https://w3id.org/edc/v0.0.1/ns/type"},
                    "edc:operator": {"@id": "odrl:eq"},
                    "edc:operandRight": "data.core.digitalTwinRegistry"
                }
            }
        }

Error logs:

SEVERE 2024-11-08T13:05:12.292763896 JerseyExtension: Unexpected exception caught
java.lang.NullPointerException: Cannot invoke "jakarta.json.JsonString.getString()" because the return value of "org.eclipse.parsson.JsonObjectBuilderImpl$JsonObjectImpl.getJsonString(String)" is null
	at org.eclipse.parsson.JsonObjectBuilderImpl$JsonObjectImpl.getString(JsonObjectBuilderImpl.java:249)
	at org.eclipse.edc.validator.jsonobject.validators.MandatoryValue.lambda$validate$2(MandatoryValue.java:43)
	at java.base/java.util.Optional.map(Unknown Source)
	at org.eclipse.edc.validator.jsonobject.validators.MandatoryValue.validate(MandatoryValue.java:43)
	at org.eclipse.edc.validator.jsonobject.validators.MandatoryValue.validate(MandatoryValue.java:31)
	at org.eclipse.edc.validator.jsonobject.JsonObjectValidator.lambda$validate$0(JsonObjectValidator.java:60)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
	at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)

jakarta.ws.rs.InternalServerErrorException: Failed to compact JsonObject: When compacting an IRI would result in an IRI which could be confused with a compact IRI [code=IRI_CONFUSED_WITH_PREFIX].
	at org.eclipse.edc.web.jersey.providers.jsonld.JerseyJsonLdInterceptor.lambda$compact$2(JerseyJsonLdInterceptor.java:87)
	at org.eclipse.edc.spi.result.AbstractResult.orElseThrow(AbstractResult.java:119)
	at org.eclipse.edc.web.jersey.providers.jsonld.JerseyJsonLdInterceptor.compact(JerseyJsonLdInterceptor.java:87)
	at org.eclipse.edc.web.jersey.providers.jsonld.JerseyJsonLdInterceptor.aroundWriteTo(JerseyJsonLdInterceptor.java:79)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
	at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:85)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
	at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:61)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)

ndr-brt · 2024-11-08T14:18:15Z

Payload:

        payload_ = {
            "@context": edc_context(),
            "@type": "CatalogRequest",
            "edc:protocol": "dataspace-protocol-http",
            "edc:counterPartyAddress": f"{edc_url_}/api/v1/dsp",
            "edc:counterPartyId": f"{edc_bpn_}",
            "edc:querySpec": {
                "edc:filterExpression": {
                    "@type": "edc:Criterion",
                    "edc:operandLeft": {"@id": "https://w3id.org/edc/v0.0.1/ns/type"},
                    "edc:operator": {"@id": "odrl:eq"},
                    "edc:operandRight": "data.core.digitalTwinRegistry"
                }
            }
        }

@kss682
I'd need the actual payload in json, not python code

kss682 · 2024-11-08T15:31:23Z

@ndr-brt Unfortunately some other post-install job is failing in our azure env that I can't generate the exact payload.
But here is the payload without the BPN ID and provider dsp url.

{
  "@context": {
    "dct": "http://purl.org/dc/terms/",
    "tx": "https://w3id.org/tractusx/v0.0.1/ns/",
    "edc": "https://w3id.org/edc/v0.0.1/ns/",
    "dcat": "https://www.w3.org/ns/dcat/",
    "odrl": "http://www.w3.org/ns/odrl/2/",
    "dspace": "https://w3id.org/dspace/v0.8/",
    "cx-common": "https://w3id.org/catenax/ontology/common#"
  },
  "@type": "CatalogRequest",
  "edc:protocol": "dataspace-protocol-http",
  "edc:counterPartyAddress": "/api/v1/dsp",
  "edc:counterPartyId": "",
  "edc:querySpec": {
    "edc:filterExpression": {
      "@type": "edc:Criterion",
      "edc:operandLeft": {
        "@id": "https://w3id.org/edc/v0.0.1/ns/type"
      },
      "edc:operator": {
        "@id": "odrl:eq"
      },
      "edc:operandRight": "data.core.digitalTwinRegistry"
    }
  }
}

wolf4ood · 2024-11-11T08:46:37Z

do you know why the edc:counterPartyId is empty?
if you remove the filter does it actually works the catalog request?

ndr-brt · 2024-11-11T10:54:11Z

@kss682 I see different issues:

counterPartyAddress should be a proper URL, and not a path
counterPartyId is empty
the exception itself is caused by:
there's a misuse of the edc Criterion attributes: they are all "values" and not "ids"
ODRL operator is not supported as filter, an EDC operator must be used

shaped like this, it likely didn't work correctly in 0.7.1 as well
a working one would be (please fill in correct values for counter party id and address)

{
  "@context": {
    "dct": "http://purl.org/dc/terms/",
    "tx": "https://w3id.org/tractusx/v0.0.1/ns/",
    "edc": "https://w3id.org/edc/v0.0.1/ns/",
    "dcat": "https://www.w3.org/ns/dcat/",
    "odrl": "http://www.w3.org/ns/odrl/2/",
    "dspace": "https://w3id.org/dspace/v0.8/",
    "cx-common": "https://w3id.org/catenax/ontology/common#"
  },
  "@type": "CatalogRequest",
  "edc:protocol": "dataspace-protocol-http",
  "edc:counterPartyAddress": "<counter party url>",
  "edc:counterPartyId": "<counter party id>",
  "edc:querySpec": {
    "edc:filterExpression": {
      "@type": "edc:Criterion",
      "edc:operandLeft": "https://w3id.org/edc/v0.0.1/ns/type",
      "edc:operator": "=",
      "edc:operandRight": "data.core.digitalTwinRegistry"
    }
  }
}

kss682 · 2024-11-11T14:54:00Z

Hello @ndr-brt , @wolf4ood , @evegufy
I have been trying to set this up locally on a minikube cluster (data-exchange values file)and as there were few issues with the azure env.

The current issue is with the wallet. (Not sure if this occurring because the previous one got resolved)
Error on control plane

DEBUG 2024-11-11T14:36:03.498133907 DIM Token expired, need to refresh.
DEBUG 2024-11-11T14:36:03.498880696 [Hashicorp Vault] Failed to resolve secret 'edc-wallet-secret': [Failed to get secret with status 404]
SEVERE 2024-11-11T14:36:03.499069479 JerseyExtension: Unexpected exception caught
org.eclipse.edc.spi.EdcException: Failed to fetch client secret from the vault with alias: edc-wallet-secret
 at org.eclipse.tractusx.edc.identity.mapper.BdrsClientImpl.lambda$resolve$0(BdrsClientImpl.java:101)
 at org.eclipse.edc.spi.result.AbstractResult.orElseThrow(AbstractResult.java:119)
 at org.eclipse.tractusx.edc.identity.mapper.BdrsClientImpl.resolve(BdrsClientImpl.java:101)
 at org.eclipse.tractusx.edc.identity.mapper.BdrsClientAudienceMapper.resolve(BdrsClientAudienceMapper.java:42)
 at org.eclipse.edc.protocol.dsp.http.dispatcher.DspHttpRemoteMessageDispatcherImpl.dispatch(DspHttpRemoteMessageDispatcherImpl.java:120)
 at org.eclipse.edc.connector.core.message.RemoteMessageDispatcherRegistryImpl.dispatch(RemoteMessageDispatcherRegistryImpl.java:48)
 at org.eclipse.edc.connector.controlplane.services.catalog.CatalogServiceImpl.requestCatalog(CatalogServiceImpl.java:44)
 at org.eclipse.edc.connector.controlplane.api.management.catalog.BaseCatalogApiController.requestCatalog(BaseCatalogApiController.java:53)
 at org.eclipse.edc.connector.controlplane.api.management.catalog.v3.CatalogApiV3Controller.requestCatalogV3(CatalogApiV3Controller.java:43)
 at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
 at java.base/java.lang.reflect.Method.invoke(Unknown Source)
 at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)

To reproduce the error:
Follow the steps in here umbrella and run it for values-adopter-data-exchange, after changing the edc version in /tractusx-data-provider to 0.7.7. Then querying the below request gives the error pasted above.

curl --location 'http://dataprovider-controlplane.tx.test/management/v3/catalog/request' \
--header 'Content-Type: application/json' \
--header 'X-Api-Key: TEST2' \
--data-raw '{
  "@context": {
    "dct": "http://purl.org/dc/terms/",
    "tx": "https://w3id.org/tractusx/v0.0.1/ns/",
    "edc": "https://w3id.org/edc/v0.0.1/ns/",
    "dcat": "https://www.w3.org/ns/dcat/",
    "odrl": "http://www.w3.org/ns/odrl/2/",
    "dspace": "https://w3id.org/dspace/v0.8/",
    "cx-common": "https://w3id.org/catenax/ontology/common#"
  },
  "@type": "CatalogRequest",
  "edc:protocol": "dataspace-protocol-http",
  "edc:counterPartyAddress": "https://dataprovider-controlplane.tx.test/api/v1/dsp",
  "edc:counterPartyId": "BPNL00000003AYRE",
  "edc:querySpec": {
    "edc:filterExpression": {
      "@type": "edc:Criterion",
      "edc:operandLeft": "https://w3id.org/edc/v0.0.1/ns/type",
      "edc:operator": "=",
      "edc:operandRight": "data.core.digitalTwinRegistry"
    }
  }
}'

But querying the ssi-dim-wallet manually works.

curl -X 'POST' \
  'https://ssi-dim-wallet-stub.tx.test/oauth/token' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'client_id=BPNL00000003AYRE&client_secret=edc-wallet-secret&grant_type='

wolf4ood · 2024-11-11T16:42:21Z

@kss682

I think the vault is a separated issue probably in your setup. Anyway I applied

the change similar to your PR + a fix in the policy definition due eclipse-edc/Connector#4185

The changeset is here main...wolf4ood:tractus-x-umbrella:chore/upgrade_tx_edc_0_7_7

And I run the helm lint/test also which is green on my fork

https://github.com/wolf4ood/tractus-x-umbrella/actions/runs/11781562495

Let me know if this helps

Thanks

evegufy · 2024-11-13T06:59:24Z

@kss682 regarding Failed to resolve secret 'edc-wallet-secret', please make sure to have executed a helm dependency updatebefore the install of the umbrellla, could be that you're missing the this change in the tx-data-provider locally when installing.

could you also please integrate the change from @wolf4ood and test it?

kss682 · 2024-11-13T15:06:39Z

@evegufy making the changes to the policy definition as per @wolf4ood suggestion does resolve the issue and the checks are passing.

* Uprade EDC to 0.7.3 #134 * fix policy definition json #159 * Bump edc to 0.7.7 #134 * chore: change tx-data-provider version --------- Co-authored-by: Evelyn Gurschler <[email protected]>

matbmoser added this to Umbrella Helm Chart Nov 6, 2024

matbmoser converted this from a draft issue Nov 6, 2024

matbmoser assigned kss682 Nov 6, 2024

evegufy changed the title ~~Update helm-test post install~~ Update helm-test post install when upgrading EDC from 0.7.1 to 0.7.6 Nov 8, 2024

evegufy changed the title ~~Update helm-test post install when upgrading EDC from 0.7.1 to 0.7.6~~ Update helm-test post install when upgrading EDC from 0.7.1 to 0.7.7 Nov 8, 2024

evegufy closed this as completed Nov 13, 2024

github-project-automation bot moved this from In Progress to Done in Umbrella Helm Chart Nov 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update helm-test post install when upgrading EDC from 0.7.1 to 0.7.7 #159

Update helm-test post install when upgrading EDC from 0.7.1 to 0.7.7 #159

matbmoser commented Nov 6, 2024 •

edited by evegufy

Loading

evegufy commented Nov 6, 2024 •

edited

Loading

kss682 commented Nov 6, 2024

kss682 commented Nov 6, 2024

evegufy commented Nov 6, 2024 •

edited

Loading

kss682 commented Nov 7, 2024 •

edited

Loading

evegufy commented Nov 7, 2024 •

edited

Loading

kss682 commented Nov 7, 2024

evegufy commented Nov 8, 2024

kss682 commented Nov 8, 2024

evegufy commented Nov 8, 2024

kss682 commented Nov 8, 2024 •

edited

Loading

ndr-brt commented Nov 8, 2024

kss682 commented Nov 8, 2024

ndr-brt commented Nov 8, 2024 •

edited

Loading

kss682 commented Nov 8, 2024

wolf4ood commented Nov 11, 2024 •

edited

Loading

ndr-brt commented Nov 11, 2024

kss682 commented Nov 11, 2024

wolf4ood commented Nov 11, 2024

evegufy commented Nov 13, 2024

kss682 commented Nov 13, 2024

Update helm-test post install when upgrading EDC from 0.7.1 to 0.7.7 #159

Update helm-test post install when upgrading EDC from 0.7.1 to 0.7.7 #159

Comments

matbmoser commented Nov 6, 2024 • edited by evegufy Loading

Task

evegufy commented Nov 6, 2024 • edited Loading

kss682 commented Nov 6, 2024

kss682 commented Nov 6, 2024

evegufy commented Nov 6, 2024 • edited Loading

kss682 commented Nov 7, 2024 • edited Loading

evegufy commented Nov 7, 2024 • edited Loading

kss682 commented Nov 7, 2024

evegufy commented Nov 8, 2024

kss682 commented Nov 8, 2024

evegufy commented Nov 8, 2024

kss682 commented Nov 8, 2024 • edited Loading

ndr-brt commented Nov 8, 2024

kss682 commented Nov 8, 2024

ndr-brt commented Nov 8, 2024 • edited Loading

kss682 commented Nov 8, 2024

wolf4ood commented Nov 11, 2024 • edited Loading

ndr-brt commented Nov 11, 2024

kss682 commented Nov 11, 2024

wolf4ood commented Nov 11, 2024

evegufy commented Nov 13, 2024

kss682 commented Nov 13, 2024

matbmoser commented Nov 6, 2024 •

edited by evegufy

Loading

evegufy commented Nov 6, 2024 •

edited

Loading

evegufy commented Nov 6, 2024 •

edited

Loading

kss682 commented Nov 7, 2024 •

edited

Loading

evegufy commented Nov 7, 2024 •

edited

Loading

kss682 commented Nov 8, 2024 •

edited

Loading

ndr-brt commented Nov 8, 2024 •

edited

Loading

wolf4ood commented Nov 11, 2024 •

edited

Loading