Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chore/edc 0.7.0 infra #351

Merged
merged 21 commits into from
May 21, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
5c72cbd
chore(edc): version increase wip
tom-rm-meyer-ISST Apr 23, 2024
14632d0
refactor(bdrs): move to infrastructure docker
tom-rm-meyer-ISST Apr 25, 2024
e30b7a6
refactor(docker-compose-edc): refactor to get to old state
tom-rm-meyer-ISST Apr 29, 2024
652e876
refactor(local): cleanup bdrs and so on
tom-rm-meyer-ISST Apr 29, 2024
ed119a2
feat(iam-mock): added iam-mock to infrastructure
tom-rm-meyer-ISST Apr 29, 2024
0041b97
feat(iam-mock): added non-root user to dockerfile
tom-rm-meyer-ISST Apr 30, 2024
7ce0213
chore: updated frontend license file
tom-rm-meyer-ISST Apr 30, 2024
85b3a3a
refactor(local): made dtr and edc of customer / supplier share one db…
tom-rm-meyer-ISST Apr 30, 2024
6f5db7f
refactor(local): made dtr and edc of customer / supplier share one db…
tom-rm-meyer-ISST May 3, 2024
19bdf5c
feat(iam-mock): added sts edr refresh workflow mock
tom-rm-meyer-ISST May 6, 2024
057a2f0
ci(.tractusx): exclude iam-mock Docker from qGate check as not distri…
tom-rm-meyer-ISST May 7, 2024
fdfaf8f
refactor(local deployment): removed comments and made mock-util only …
tom-rm-meyer-ISST May 14, 2024
4e56c84
refactor(generate-keys.sh): generate file seed-bdrs.sh
tom-rm-meyer-ISST May 14, 2024
60fa49f
chore(mock-util-service): updated dependencies
tom-rm-meyer-ISST May 14, 2024
14106c6
Merge branch 'main' into chore/edc-0.7.0-infra
tom-rm-meyer-ISST May 15, 2024
60c191d
chore: updated frontend dependencies
tom-rm-meyer-ISST May 15, 2024
815fd42
fix(iam-mock): corrected claims in credential flow
tom-rm-meyer-ISST May 16, 2024
b268dd7
fix(local/docker-compose.yaml): corrected version of dtr-supplier (me…
tom-rm-meyer-ISST May 16, 2024
7f42181
chore(tractus-x-edc/docker-compose.yaml): bump version to 0.7.1
tom-rm-meyer-ISST May 16, 2024
daae340
Merge branch 'main' into chore/edc-0.7.0-infra
tom-rm-meyer-ISST May 17, 2024
ec0b456
chore: remove unused edc compose and notes during migration
tom-rm-meyer-ISST May 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .tractusx
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,4 @@ repositories:
skipReleaseChecks:
alignedBaseImage:
- "local/vault/Dockerfile"
- "local/iam-mock/Dockerfile"
55 changes: 43 additions & 12 deletions docs/DEVELOPMENT.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
## Setup development database

For local development a postgresql database is needed. The file local/docker-compose-dev-postgres.yaml provides a
For local development a postgresql database is needed. The file local/docker-compose-dev-postgres.yaml provides a
postgres that can be started freshly for development:

```shell
Expand All @@ -16,19 +16,23 @@ docker compose -f docker-compose-dev-postgres.yaml down
_NOTE: For testing purposes HyperSql is still used but excluded for spring run._

## Keeping dependencies-files up to date

### Backend

Navigate to the `./backend` folder and run:
Navigate to the `./backend` folder and run:

```
mvn org.eclipse.dash:license-tool-plugin:license-check
cp DEPENDENCIES ../DEPENDENCIES_BACKEND
```
The first line runs the maven license tool with the parameters specified in the

The first line runs the maven license tool with the parameters specified in the
`./backend/pom.xml` and produces a DEPENDENCIES file in the .`/backend` folder.
Then this file gets copied to the PURIS-project root folder under the name `DEPENDENCIES_BACKEND`.
Both files should be updated prior to any pull request.
Then this file gets copied to the PURIS-project root folder under the name `DEPENDENCIES_BACKEND`.
Both files should be updated prior to any pull request.

### Frontend

```
# move to a persistent folder. Could also be ~/jars.
mv org.eclipse.dash.licenses-1.0.2.jar ~/coding/org.eclipse.dash.licenses-1.0.2.jar
Expand All @@ -42,22 +46,43 @@ cd frontend
eclipseDashTool package-lock.json -project automotive.tractusx -summary ../DEPENDENCIES_FRONTEND
```

### mock-util-service

Temprorary also the mock-util-service needs to be kept up to date. First add the alias same as for frontend to your
`.bashrc` and then use the following command:

```shell
cd local/iam-mock
cat requirements.txt | grep -v \# \
| sed -E -e 's|([^= ]+)==([^= ]+)|pypi/pypi/-/\1/\2|' -e 's| ||g' \
| sort | uniq \
| eclipseDashTool -summary DEPENDENCIES -
```

Note: Dash action provided by eclipse-tractusx/sig-infra does not provide to opportunity for python.

## Frontend container building workaround to use environment variables for vue

### The mechanism for docker is the following:

- `.env` has vite variables
- `.env.dockerbuild` has the vite variable that maps on an environment variable (`VITE_BACKEND_BASE_URL=$BACKEND_BASE_URL`)
- `.env.dockerbuild` has the vite variable that maps on an environment
variable (`VITE_BACKEND_BASE_URL=$BACKEND_BASE_URL`)
- `src/config.json` has the environment variable names and the environment variable to substring in a json format.

### When building the container:

1. `.env.dockerbuild` is used
2. vite / vue builds the application into a dest folder, that will be served by nginx

> Result for the .env: <br> VITE_BACKEND_BASE_URL won't write a variable value BUT a placeholder into the built files ($BACKEND_BASE_URL)
> Result for the .env: <br> VITE_BACKEND_BASE_URL won't write a variable value BUT a placeholder into the built files (
> $BACKEND_BASE_URL)

### When building the container, there is a "start-nginx.sh" file that does the following

1. Collects the environment variables (set for the docker container / set via helm as `BACKEND_BASE_URL`)
2. Looks-up the "to replace string" from `config.json` (e.g., for `BACKEND_BASE_URL`, it will search for `$BACKEND_BASE-URL` in the built files)
2. Looks-up the "to replace string" from `config.json` (e.g., for `BACKEND_BASE_URL`, it will search
for `$BACKEND_BASE-URL` in the built files)
3. Does the replacement in the built files
4. Starts nginx

Expand All @@ -66,26 +91,30 @@ eclipseDashTool package-lock.json -project automotive.tractusx -summary ../DEPEN
When changing the helm charts due to changes of e.g. environment variables, one should test locally whether the changes
work.

First thing one should check is whether the templates may be resolved / substituted correctly and if your changes are
First thing one should check is whether the templates may be resolved / substituted correctly and if your changes are
defaulted correctly:

```shell
cd charts/puris
helm template .
>> no error is thrown, chart is resolved, changes are done correctly
```

Now build your images as explained in the respective install.mds.

- [backend](../backend/INSTALL.md)
- [frontend](../frontend/INSTALL.md)

Now you need to update your Chart.yml and values.yml:

- Chart.yml: change `appVersion` to your build tag (e.g., `dev`)
- values.yml: for both frontend and backend change the image
- `repository` should be set to the image name used during docker build (e.g., `puris-backend`, `puris-frontend`)
- `pullPolicy` should be set to `Never`
- `repository` should be set to the image name used during docker build (e.g., `puris-backend`, `puris-frontend`)
- `pullPolicy` should be set to `Never`

Now depending on your runtime environment you need to load the images into it (we assume, you built puris-backend:dev
Now depending on your runtime environment you need to load the images into it (we assume, you built puris-backend:dev
and (puris-frontend:dev):

```shell
# minikube
minikube image load puris-backend:dev
Expand All @@ -96,6 +125,7 @@ docker image ls | grep puris-backend
minikube image ls --format table | grep puris-frontend
docker image ls | grep puris-frontend
```

```shell
# kind
kind load puris-backend:dev
Expand All @@ -108,6 +138,7 @@ docker image ls | grep puris-backend
docker exec -it {container-id} crictl images | grep puris-frontend
docker image ls | grep puris-frontend
```

**ATTENTION: MAKE SURE THAT THE IMAGE ID IN YOUR KUBERNETES ENVIRONMENT IS THE SAME AS IN YOUR LOCAL DOCKER.**

Else you can delete images as follows:
Expand Down
1 change: 1 addition & 0 deletions local/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@
*.keys
*.secret
.env
seed-bdrs.sh
72 changes: 50 additions & 22 deletions local/INSTALL.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,28 +2,34 @@

## Additional information for Windows users

If you want to use the local setup on a Windows machine, it seems advisable to use the Windows subsystem for Linux (WSL).
An installation guide can be found [here](https://learn.microsoft.com/en-us/windows/wsl/install).
If you want to use the local setup on a Windows machine, it seems advisable to use the Windows subsystem for Linux (
WSL).
An installation guide can be found [here](https://learn.microsoft.com/en-us/windows/wsl/install).

Also see this [additional information about using Docker in combination with WSL2](https://docs.docker.com/desktop/wsl/)

## Initial Setup
In case you had any previous installations of this project on your machine, it is advisable to remove them via the script

In case you had any previous installations of this project on your machine, it is advisable to remove them via the
script
(see below in the Notes on debugging section).

Run the following script to generate the necessary keys. It will also create an .env file in the ./local folder.
Make sure to have `openssl` and `jq` installed in your shell. openssl is pre-installed on most operating systems. jq can be
Run the following script to generate the necessary keys. It will also create an .env file in the ./local folder.
Make sure to have `openssl` and `jq` installed in your shell. openssl is pre-installed on most operating systems. jq can
be
installed via the usual installation repositories, see [here](https://jqlang.github.io/jq/download/)

```shell
cd local
sh generate-keys.sh
```

## Build
If you are doing a fresh install and everytime you edited the code of the PURIS frontend or backend you have to create a
new build of docker images for the PURIS frontend/backend.

For creating a docker image of the frontend, navigate your shell to the frontend folder and run
If you are doing a fresh install and everytime you edited the code of the PURIS frontend or backend you have to create a
new build of docker images for the PURIS frontend/backend.

For creating a docker image of the frontend, navigate your shell to the frontend folder and run

```
docker build -t puris-frontend:dev .
Expand All @@ -37,50 +43,72 @@ docker build -t puris-backend:dev .

Please see the INSTALL.md documents in the [frontend](../frontend/INSTALL.md) and [backend](../backend/INSTALL.md)

The default image tag is 'dev'. Remember to also adjust the tag in the docker-compose.yaml if you want to use different
tags.
The default image tag is 'dev'. Remember to also adjust the tag in the docker-compose.yaml if you want to use different
tags.

## Start
First start the infrastructure by navigating your shell to the local folder and running

First start the infrastructure by navigating your shell to the local folder and running

```shell
docker compose -f docker-compose-infrastructure.yaml up
```
After the MIW container has finished booting, use this script (also in the local folder) to initialise two wallets for customer and supplier:

Note: sh init-wallets.sh is temporarily not needed
After the MIW container has finished booting, use this script (also in the local folder) to initialise two wallets for
customer and supplier:

```shell
sh init-wallets.sh
```
Then start the PURIS demonstrator containers via:

After starting the central infrastructure, initialize the bdrs-service. To do so, just run the script `seed-bdrs.sh`
created during the run of script `generate-keys.sh`.

```shell
sh seed-brds.sh
```

Then start the PURIS demonstrator containers via:

```shell
docker compose up
```
Wait for the startup and visit http://localhost:3000/ for the customer's frontend or http://localhost:3001/ for the supplier side.

Whenever you have edited the source code in the frontend or backend and you want to test these changes, we recommend that you
stop all the containers, which are not part of the infrastructure, by deleting the volumes, i.e. run
Wait for the startup and visit http://localhost:3000/ for the customer's frontend or http://localhost:3001/ for the
supplier side.

Whenever you have edited the source code in the frontend or backend and you want to test these changes, we recommend
that you
stop all the containers, which are not part of the infrastructure, by deleting the volumes, i.e. run

```
docker compose down -v
```

Then, build a new docker image for the respective component (frontend and/or backend) as described above and then restart via
Then, build a new docker image for the respective component (frontend and/or backend) as described above and then
restart via

```shell
docker compose up
```

In general, it is not necessary to restart the infrastructure, if you had to rebuild frontend or backend.
However, in rare cases there may be issues with the MIW. If this
happens, you should use the cleanup script as mentioned in the debugging section below and then repeat the above-mentioned
steps beginning with the Initial Setup section.
In general, it is not necessary to restart the infrastructure, if you had to rebuild frontend or backend.
However, in rare cases there may be issues with the MIW. If this
happens, you should use the cleanup script as mentioned in the debugging section below and then repeat the
above-mentioned
steps beginning with the Initial Setup section.

## Notes on debugging

### Vault & Certs

When having problems with the certs or the vault, one may need to delete the vault container.
The following script stops all infrastructure containers as well as the PURIS demonstrator containers:
The following script stops all infrastructure containers as well as the PURIS demonstrator containers:

```shell
cd local
sh cleanup.sh
```

Then start your containers again with the aforementioned commands.
9 changes: 9 additions & 0 deletions local/bdrs/application.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
edc.api.auth.key=${EDC_API_PW}
web.http.management.port=8581
web.http.management.path=/api/management
web.http.directory.port=8582
web.http.directory.path=/api/directory
# looking up DIDs should not use https
edc.iam.did.web.use.https=false
edc.iam.trusted-issuer.issuer.id=did:web:mock-util-service/trusted-issuer
_level=DEBUG
3 changes: 3 additions & 0 deletions local/cleanup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@
docker compose down -v
docker compose -f docker-compose-infrastructure.yaml down -v
docker image rm local-vault
docker image rm local-mock-util-service
rm .env
rm seed-bdrs.sh
rm ./vault/secrets -r
rm ./iam-mock/keys -r
echo "Deleted .env and vault/secrets"
29 changes: 26 additions & 3 deletions local/docker-compose-infrastructure.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,17 +31,17 @@
networks:
- miw-net

postgres:

Check warning on line 34 in local/docker-compose-infrastructure.yaml

View workflow job for this annotation

GitHub Actions / Analyze

[MEDIUM] Healthcheck Not Set

Check containers periodically to see if they are running properly.
image: postgres:15.4-alpine
container_name: postgres-miw
environment:
POSTGRES_DB: edc
POSTGRES_DB: miw
POSTGRES_USER: ${PG_USER}
POSTGRES_PASSWORD: ${PG_PW}
volumes:
# volumes:
# use docker-compose down --volumes to kill db volume
# only then, changes to the script are executed!
- ./postgres/init-db.sql:/docker-entrypoint-initdb.d/init-db.sql
# - ./postgres/init-db.sql:/docker-entrypoint-initdb.d/init-db.sql
ports:
- "127.0.0.1:5432:5432"
networks:
Expand Down Expand Up @@ -81,6 +81,29 @@
- ./vault/secrets:${VAULT_SECRETS_DIR}
networks:
- miw-net

mock-util-service:

Check warning

Code scanning / KICS

Healthcheck Not Set Warning

Healthcheck is not defined.

Check warning

Code scanning / KICS

Memory Not Limited Warning

'deploy' is not defined

Check warning

Code scanning / KICS

Security Opt Not Set Warning

Docker compose file does not have 'security_opt' attribute

Check warning

Code scanning / KICS

Container Capabilities Unrestricted Warning

Docker compose file doesn't have 'cap_drop' attribute. Make sure your container only has necessary capabilities.

Check notice

Code scanning / KICS

Cpus Not Limited Note

'deploy' is not defined
build: ./iam-mock
container_name: mock-util-service
ports:

Check warning

Code scanning / KICS

Privileged Ports Mapped In Container Warning

Docker compose file has 'ports' attribute set to privileged ports (<1024).
- "127.0.0.1:8888:80"
networks:
- miw-net

bdrs:

Check warning on line 93 in local/docker-compose-infrastructure.yaml

View workflow job for this annotation

GitHub Actions / Analyze

[MEDIUM] Healthcheck Not Set

Check containers periodically to see if they are running properly.

Check warning

Code scanning / KICS

Healthcheck Not Set Warning

Healthcheck is not defined.

Check warning

Code scanning / KICS

Memory Not Limited Warning

'deploy' is not defined

Check warning

Code scanning / KICS

Security Opt Not Set Warning

Docker compose file does not have 'security_opt' attribute

Check warning

Code scanning / KICS

Container Capabilities Unrestricted Warning

Docker compose file doesn't have 'cap_drop' attribute. Make sure your container only has necessary capabilities.

Check notice

Code scanning / KICS

Cpus Not Limited Note

'deploy' is not defined
image: tractusx/bdrs-server-memory:0.0.4
container_name: bdrs
env_file:
- ./bdrs/application.properties
environment:
WEB_HTTP_PORT: 8580
WEB_HTTP_PATH: /api
ports:
- "127.0.0.1:8580:8580"
- "127.0.0.1:8581:8581"
- "127.0.0.1:8582:8582"
networks:
- miw-net
networks:
miw-net:
name: miw-net
Loading
Loading