Migration to EDC 0.7.2 (#7)

* Migrating to EDC 0.7.2 (Digital-Ecosystems#77) * feat: migrating to edc 0.7.3 * feat: migrating to edc 0.7.2 * feat: migrating to edc 0.7.2 * feat: migrating to edc 0.7.2 * feat: migrating to edc 0.7.2 * feat: migrating to edc 0.7.2 * Fixing connector-persistente to EDC 0.7.2 (Digital-Ecosystems#78) * feat: migrating to edc 0.7.3 * feat: migrating to edc 0.7.2 * feat: migrating to edc 0.7.2 * feat: migrating to edc 0.7.2 * feat: migrating to edc 0.7.2 * feat: migrating to edc 0.7.2 * feat: migrating to edc 0.7.2 * S3 Endpoint Regions Validation (Digital-Ecosystems#79) * feat: s3 endpoint regions validation * chore: fixing assets doc * fix: launchers config * fix: regions endpoint signature changes (Digital-Ecosystems#81) Co-authored-by: Glaucio Jannotti <[email protected]> --------- Co-authored-by: Glaucio Jannotti <[email protected]> Co-authored-by: Glaucio Jannotti <[email protected]>
ionos-cloud · Dec 5, 2024 · cb4e6b8 · cb4e6b8
1 parent 0366e9c
commit cb4e6b8
Show file tree

Hide file tree

Showing 87 changed files with 970 additions and 1,507 deletions.
diff --git a/.gitignore b/.gitignore
@@ -51,6 +51,7 @@ launchers/demo-e2e/edc-config.properties
 *.hprof
 
 **/vault-keys.json
+**/vault-tokens.json
 runtime_settings.properties
 generated_backend.tf
 

diff --git a/README.md b/README.md
@@ -59,15 +59,15 @@ The credentials can be found/configured in one of the following:
 
 It is required to configure those parameters:
 
-| Parameter name                      | Description                                                                                                                                        | Mandatory                                            |
-|-------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------|
-| `edc.ionos.access.key`              | IONOS Access Key Id to access S3                                                                                                                   | Yes if the context is accessing file                 |
-| `edc.ionos.secret.access.key`       | IONOS Secret Access Key to access S3                                                                                                               | Yes if the context is accessing file                 |
-| `edc.ionos.token`                   | IONOS token to allow S3 provisioning                                                                                                               | Yes if the context is provisioning access for others |
-| `edc.ionos.endpoint.region`         | IONOS S3 endpoint region. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/endpoints) for further information.      | No, the default value is de                          |
-| `edc.ionos.max.files`               | Maximum number of files retrieved by list files function.                                                                                          | No, the default value is 5,000 files                 |
-| `edc.ionos.key.validation.attempts` | Maximum number of attemps to validate a temporary key after its creation.                                                                          | No, the default values is 10 attempts                |
-| `edc.ionos.key.validation.delay`    | Time to wait (in milisseconds) before each key validation attempt. In each new attempt the delay is multiplied by the attempt number.              | No, the default value is 3,000 (3 seconds)           |
+| Parameter name                      | Description                                                                                                                                   | Mandatory                                            |
+|-------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------|
+| `edc.ionos.access.key`              | IONOS Access Key Id to access S3                                                                                                              | Yes if the context is accessing file                 |
+| `edc.ionos.secret.access.key`       | IONOS Secret Access Key to access S3                                                                                                          | Yes if the context is accessing file                 |
+| `edc.ionos.token`                   | IONOS token to allow S3 provisioning                                                                                                          | Yes if the context is provisioning access for others |
+| `edc.ionos.endpoint.region`         | IONOS S3 endpoint region. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/endpoints) for further information. | No, the default value is "de"                        |
+| `edc.ionos.max.files`               | Maximum number of files retrieved by list files function.                                                                                     | No, the default value is 5,000 files                 |
+| `edc.ionos.key.validation.attempts` | Maximum number of attemps to validate a temporary key after its creation.                                                                     | No, the default values is 10 attempts                |
+| `edc.ionos.key.validation.delay`    | Time to wait (in milisseconds) before each key validation attempt. In each new attempt the delay is multiplied by the attempt number.         | No, the default value is 3,000 (3 seconds)           |
 
 To create the token please take a look at the following [documentation](./ionos_token.md).
 

diff --git a/assets.md b/assets.md
@@ -11,21 +11,21 @@ The asset registration aims to specify which file/folder we want to share. We ca
 
 | Parameter        | Description                                                                                                                                                                                                                                                                                                                                 | Mandatory              |
 |------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------|
-| `region`        | IONOS S3 endpoint region. Refer to  [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/s3-endpoints)  for further information.                                                                                                                                                                                           | no, default value = de |
-| `bucketName`     | IONOS S3 bucket name.   Refer to  [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/concepts/buckets) for further information.                                                                                                                                                                                                                                                                      | yes                    |
-| `blobName`       | File name or path to folder                                                                                                                                                                                                                                                                                                                 | yes                    |
-| `filterIncludes` | `filterIncludes` use regular expression that will be used to select the file name pattern from the asset's blobName that will be copied during the transfer <br/> * do not consider the blobName in the expression, but the path from it. example: blobName = folder1, filterIncludes=file1.csv, the file foloder1/file1.csv will be copied | no                     |
-| `filterExcludes` | `filterExcludes` use regular expression that will be used to select the file name pattern from the asset's blobName that will NOT be copied during the transfer <br/>                                                                                                                                                                       | no                     |
+| `region`         | IONOS S3 endpoint region. Refer to  [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/s3-endpoints)  for further information.| no, default value = de |
+| `bucketName`     | IONOS S3 bucket name.   Refer to  [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/concepts/buckets) for further information.| yes                    |
+| `blobName`       | File name or path to folder| yes                    |
+| `filterIncludes` | `filterIncludes` use regular expression that will be used to select the file name pattern from the asset's blobName that will be copied during the transfer <br/> * do not consider the blobName in the expression, but the path from it. example: blobName = folder1, filterIncludes=file1.csv, the file foloder1/file1.csv will be copied| no                     |
+| `filterExcludes` | `filterExcludes` use regular expression that will be used to select the file name pattern from the asset's blobName that will NOT be copied during the transfer <br/>| no                     |
 
 Note:  if `filterIncludes` and  `filterExcludes` parameters are satisfied, the files to be copied will be selected using the `filterIncludes` and after that selected list, the files that have the pattern defined in the `filterExcludes` will be ignored.
 
 
 ## Example
 
 ```json
-"dataAddress": {
+"dataAddress":{
   "type": "IonosS3", //from EDC
-  "region": "de,
+  "region": "de",
   "bucketName": "mybucket",
   "blobName": "folder1/",
   "filterIncludes": "file1.csv",
@@ -40,17 +40,17 @@ The transfer of assets aims to transfer the files/folders from one connector to
 ### Requirements
 
 
-| Parameter    | Description                                                                                                                                        | Mandatory              |
-|--------------|----------------------------------------------------------------------------------------------------------------------------------------------------|------------------------|
-| `region`     | IONOS S3 endpoint region. Refer to  [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/s3-endpoints)  for further information. | no, default value = de |
-| `bucketName` | IONOS S3 bucket name.   Refer to  [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/concepts/buckets) for further information.                                                                             | yes                    |
-| `path`       | Path of destination where the file/folder will be placed. </br>  *if the path not filled, the file will be placed in the root of the bucket.       | no                     |
+| Parameter     | Description                                                                                                                                  | Mandatory              |
+|---------------|----------------------------------------------------------------------------------------------------------------------------------------------|------------------------|
+| `region`      | IONOS S3 endpoint region. Refer to  [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/s3-endpoints)  for further information.| no, default value = de |
+| `bucketName`  | IONOS S3 bucket name.   Refer to  [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/concepts/buckets) for further information| yes                    |
+| `path`        | Path of destination where the file/folder will be placed. </br>  *if the path not filled, the file will be placed in the root of the bucket. | no                     |
 
 
 ## Example
 
 ```json
-"dataDestination": {
+"dataDestination":{
   "type": "IonosS3", //from EDC
   "region": "de",
   "bucketName": "mybucket",

diff --git a/deployment/README.md b/deployment/README.md
@@ -86,13 +86,13 @@ In case you want to configure this Connector without Hashicorp Vault, you need t
 
 ```yaml
   ionos:
-    region: <IONOS-S3-ENDPOINT-REGION>
+    region: <YOUR-S3-ENDPOINT-REGION>
     accessKey: <YOUR-KEY>
     secretKey: <YOUR-SECRET-KEY>
     token: <IONOS-TOKEN>
 ```
 
-They should be the same as the ones set in the environment variables. The **ionos.endpoint** is set to the default S3 location, but it can be changed to any other location.
+They should be the same as the ones set in the environment variables. The **ionos.region** is set to the default S3 endpoint region, but it can be changed to any other location.
 
 
 If you don't want the Connector to be externally accessible, you need to set the following parameters in the helm [values.yaml](deployment/helm/edc-ionos-s3/values.yaml):
@@ -110,21 +110,19 @@ This will allocate a public IP address to the Connector. You can then access it
 
 All commands paths are relative to the current directory where this readme is located.
 
-### 1. Install the EDC Ionos S3 services
+### 1. Deploy the services
 
-To install the services run the script ```deploy-services.sh``` in ```terraform``` directory.
+To deploy the services run the script ```deploy-services.sh``` in ```terraform``` directory.
 
 ```sh
 cd terraform
 ./deploy-services.sh
 ```
+### 2. Undeploy the services
 
-### 2. Vault keys
-After the services are installed you will have ```vault-keys.json``` file containing the vault keys in ```terraform``` directory.
-
-### 3. Destroy the services
+To undeploy the services run the script ```undeploy-services.sh``` in ```terraform``` directory.
 
 ```sh
 cd terraform
-./destroy-services.sh
+./undeploy-services.sh
 ```
diff --git a/deployment/helm/edc-ionos-s3/templates/configmap.yaml b/deployment/helm/edc-ionos-s3/templates/configmap.yaml
@@ -4,6 +4,7 @@ metadata:
   name: {{ include "edc-ionos-s3.fullname" . }}-config
 data:
   config.properties: |
+    edc.participant.id={{ .Values.edc.participant.id }}
     web.http.port={{ .Values.web.http.port }}
     web.http.path={{ .Values.web.http.path }}
     web.http.management.port={{ .Values.web.http.management.port }}
@@ -14,20 +15,19 @@ data:
     web.http.public.path={{ .Values.web.http.public.path }}
     web.http.control.port={{ .Values.web.http.control.port }}
     web.http.control.path={{ .Values.web.http.control.path }}
+    edc.dsp.callback.address={{ .Values.edc.dsp.callback.address }}
+    edc.dataplane.token.validation.endpoint={{ .Values.edc.dataplane.token.validation.endpoint }}
+    edc.dataplane.api.public.baseurl={{ .Values.edc.dataplane.api.public.baseurl }}
     edc.api.auth.key={{ .Values.edc.api.auth.key }}
-    edc.participant.id={{ .Values.edc.participant.id }}
-    edc.ionos.access.key={{ .Values.edc.ionos.accessKey }}
-    edc.ionos.secret.key={{ .Values.edc.ionos.secretKey }}
-    edc.ionos.endpoint.region={{ .Values.edc.ionos.endpoint.region }}
-    edc.ionos.token={{ .Values.edc.ionos.token }}
+    edc.transfer.proxy.token.signer.privatekey.alias={{ .Values.edc.vault.certificates.privateKey.alias }}
+    edc.transfer.proxy.token.verifier.publickey.alias={{ .Values.edc.vault.certificates.publicKey.alias }}
     edc.vault.hashicorp.url={{ .Values.edc.vault.hashicorp.url }}
     edc.vault.hashicorp.token={{ .Values.edc.vault.hashicorp.token }}
     edc.vault.hashicorp.timeout.seconds={{ .Values.edc.vault.hashicorp.timeout.seconds }}
-    edc.ids.id={{ .Values.edc.ids.id }}
-    edc.dsp.callback.address={{ .Values.edc.dsp.callback.address }}:{{ .Values.web.http.protocol.port }}{{ .Values.web.http.protocol.path }}
-    edc.receiver.http.endpoint={{ .Values.edc.receiver.http.endpoint }}/receiver/{{ .Values.edc.ids.id }}/callback
-    edc.public.key.alias={{ .Values.edc.public.key.alias }}
-    edc.dataplane.token.validation.endpoint={{ .Values.edc.dataplane.token.validation.endpoint }}:{{ .Values.web.http.control.port }}{{ .Values.web.http.control.path }}/token
+    edc.ionos.access.key={{ .Values.edc.ionos.accessKey }}
+    edc.ionos.secret.key={{ .Values.edc.ionos.secretKey }}
+    edc.ionos.endpoint.region={{ .Values.edc.ionos.region }}
+    edc.ionos.token={{ .Values.edc.ionos.token }}
 
     {{- if eq .Values.edc.persistenceType "PostgreSQLaaS" }}
     edc.datasource.asset.name=asset

diff --git a/deployment/helm/edc-ionos-s3/values.yaml b/deployment/helm/edc-ionos-s3/values.yaml
@@ -113,34 +113,31 @@ edc:
     auth:
       key: password
   vault:
-    clientid: company1
-    tenantid: 1
-    certificate: /resources/
+    certificates:
+      publicKey:
+        alias: edc.connector.public.key
+      privateKey:
+        alias: edc.connector.private.key
     hashicorp:
       url: http://vault:8200
       token:
       timeout:
         seconds: 30
-  ids:
-    id: urn:connector:provider
   ionos:
-    endpoint: s3-eu-central-1.ionoscloud.com
+    region: de
     accessKey: notnull
     secretKey: notnull
     token: notnull
   dsp:
     callback:
-      address: http://localhost
-  receiver:
-    http:
-      endpoint: http://localhost:4000
-  public:
-    key:
-      alias: alias
+      address: http://localhost:8281/protocol
   dataplane:
+    api:
+      public:
+        baseurl: http://localhost:8282/public
     token:
       validation:
-        endpoint: http://localhost
+        endpoint: http://localhost:8283/control/token
   persistenceType: PostgreSQLaaS # 'PostgreSQLaaS', 'PostgreSQL' or 'None'
   postgresql: # Only used if persistenceType is 'PostgreSQLaaS' or 'PostgreSQL'
     host: postgresql

diff --git a/deployment/terraform/clean-state.sh b/deployment/terraform/clean-state.sh
@@ -0,0 +1,42 @@
+#!/bin/bash
+
+# remove terraform state
+rm -rf ./configure-public-address/.terraform
+rm -f ./configure-public-address/terraform.tfstate
+rm -f ./configure-public-address/.terraform.lock.hcl
+rm -f ./configure-public-address/terraform.tfstate.backup
+
+rm -rf ./ionos-s3-deploy/.terraform
+rm -f ./ionos-s3-deploy/terraform.tfstate
+rm -f ./ionos-s3-deploy/.terraform.lock.hcl
+rm -f ./ionos-s3-deploy/terraform.tfstate.backup
+
+rm -rf ./vault-init/.terraform
+rm -f ./vault-init/terraform.tfstate
+rm -f ./vault-init/.terraform.lock.hcl
+rm -f ./vault-init/terraform.tfstate.backup
+
+rm -rf ./vault-deploy/.terraform
+rm -f ./vault-deploy/terraform.tfstate
+rm -f ./vault-deploy/.terraform.lock.hcl
+rm -f ./vault-deploy/terraform.tfstate.backup
+
+rm -rf ./ionos-postgresqlaas/.terraform
+rm -f ./ionos-postgresqlaas/terraform.tfstate
+rm -f ./ionos-postgresqlaas/.terraform.lock.hcl
+rm -f ./ionos-postgresqlaas/terraform.tfstate.backup
+
+rm -rf ./postgresql-deploy/.terraform
+rm -f ./postgresql-deploy/terraform.tfstate
+rm -f ./postgresql-deploy/.terraform.lock.hcl
+rm -f ./postgresql-deploy/terraform.tfstate.backup
+
+rm -rf ./db-scripts/.terraform
+rm -f ./db-scripts/terraform.tfstate
+rm -f ./db-scripts/.terraform.lock.hcl
+rm -f ./db-scripts/terraform.tfstate.backup
+
+rm -f vault-init/vault-keys.json
+rm -f vault-init/vault-tokens.json
+
+echo "Terraform state cleanup complete"
diff --git a/deployment/terraform/configure-public-address/public-addresses.sh b/deployment/terraform/configure-public-address/public-addresses.sh
@@ -12,9 +12,9 @@ fi
 # Change public address in the config.properties in the configmap
 kubectl --kubeconfig=$TF_VAR_kubeconfig -n $TF_VAR_namespace get configmap edc-ionos-s3-config -o yaml | sed "s/edc.dsp.callback.address=.*/edc.dsp.callback.address=http:\/\/$CONNECTOR_ADDRESS:8281\/protocol/g" | kubectl --kubeconfig=$TF_VAR_kubeconfig apply -f -
 
-kubectl --kubeconfig=$TF_VAR_kubeconfig -n $TF_VAR_namespace get configmap edc-ionos-s3-config -o yaml | sed "s/edc.receiver.http.endpoint=.*/edc.receiver.http.endpoint=http:\/\/$CONNECTOR_ADDRESS:4000\/receiver\/urn:connector:provider\/callback/g" | kubectl --kubeconfig=$TF_VAR_kubeconfig apply -f -
-
 kubectl --kubeconfig=$TF_VAR_kubeconfig -n $TF_VAR_namespace get configmap edc-ionos-s3-config -o yaml | sed "s/edc.dataplane.token.validation.endpoint=.*/edc.dataplane.token.validation.endpoint=http:\/\/$CONNECTOR_ADDRESS:8283\/control\/token/g" | kubectl --kubeconfig=$TF_VAR_kubeconfig apply -f -
 
+kubectl --kubeconfig=$TF_VAR_kubeconfig -n $TF_VAR_namespace get configmap edc-ionos-s3-config -o yaml | sed "s/edc.dataplane.api.public.baseurl=.*/edc.dataplane.api.public.baseurl=http:\/\/$CONNECTOR_ADDRESS:8282\/public/g" | kubectl --kubeconfig=$TF_VAR_kubeconfig apply -f -
+
 # Restart the pods
 kubectl --kubeconfig=$TF_VAR_kubeconfig -n $TF_VAR_namespace delete pod -l app.kubernetes.io/name=edc-ionos-s3