feat: upgrade ZIO http client to improve performance #850

ghost · 2024-01-18T16:07:26Z

Overview

Previous to this PR, when using ZIO HTTP RC2 - we saw a very long delay when making HTTP requests (>2 seconds for each request) - we identified that this was not only an issue with the version of ZIO HTTP we were using but also how it was managing and shuting down the ZIO HTTP stack - essentially every request was starting and stopping the full stack.

After, with upgrading to ZIO HTTP RC4 and refactoring the ZIO layer composition for scope and client, we now see requests completing within 100 ms as the full stack doens't shutdown with every request

Also included in this PR are several improvements to the performance testing framework for error handling when polling background jobs and for managing exceptions when deserialising JSON - previously, any exception thrown was not counted as a failure in the test execution and dropped - now - metrics are available for when the background job does not complete a state transition within the given number of polls

Fixes ATL-6318

Checklist

My PR contains...

No code changes (changes to documentation, CI, metadata, etc.)
Bug fixes (non-breaking change which fixes an issue)
Improvements (misc. changes to existing features)
Features (non-breaking change which adds functionality)

My changes...

are breaking changes
are not breaking changes
If yes to above: I have updated the documentation accordingly

Documentation

My changes do not require a change to the project documentation
My changes require a change to the project documentation
If yes to above: I have updated the documentation accordingly

Tests

My changes can not or do not need to be tested
My changes can and should be tested by unit and/or integration tests
If yes to above: I have added tests to cover my changes
If yes to above: I have taken care to cover edge cases in my tests

infrastructure/single-tenant-testing-stack/docker-compose.yml

+    ports:
+      - "9995:9187"
+    environment:
+      - DATA_SOURCE_NAME=postgresql://postgres:postgres@holder-db:5432/postgres?sslmode=disable


infrastructure/single-tenant-testing-stack/docker-compose.yml

+    ports:
+    - "9996:9187"
+    environment:
+      - DATA_SOURCE_NAME=postgresql://postgres:postgres@issuer-db:5432/postgres?sslmode=disable


github-actions · 2024-01-18T16:08:56Z

Unit Test Results

85 files ±0 85 suites ±0 19m 21s ⏱️ -41s
719 tests ±0 711 ✅ ±0 8 💤 ±0 0 ❌ ±0
725 runs ±0 717 ✅ ±0 8 💤 ±0 0 ❌ ±0

Results for commit 53ae046. ± Comparison against base commit 529a9e8.

♻️ This comment has been updated with latest results.

github-actions · 2024-01-18T16:18:39Z

Integration Test Results

12 files +1 12 suites +1 2s ⏱️ ±0s
25 tests +1 25 ✅ +1 0 💤 ±0 0 ❌ ±0
26 runs +1 26 ✅ +1 0 💤 ±0 0 ❌ ±0

Results for commit 53ae046. ± Comparison against base commit 529a9e8.

♻️ This comment has been updated with latest results.

Signed-off-by: David Poltorak <[email protected]>

…ance Signed-off-by: David Poltorak <[email protected]>

Signed-off-by: David Poltorak <[email protected]>

…requests Signed-off-by: Benjamin Voiturier <[email protected]>

…e) to agent's configuration file Signed-off-by: Benjamin Voiturier <[email protected]>

…f the active ZIO Scope Signed-off-by: Benjamin Voiturier <[email protected]>

Signed-off-by: David Poltorak <[email protected]>

...-agent/service/server/src/main/scala/io/iohk/atala/agent/notification/WebhookPublisher.scala

prism-agent/service/server/src/main/scala/io/iohk/atala/agent/server/Main.scala

prism-agent/service/server/src/test/scala/io/iohk/atala/ZioHttpTest.scala

pollux/lib/core/src/main/scala/io/iohk/atala/pollux/core/service/HttpURIDereferencerImpl.scala

Signed-off-by: David Poltorak <[email protected]>

…error case Signed-off-by: Benjamin Voiturier <[email protected]>

sonarqubecloud · 2024-01-25T11:20:00Z

Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

2 New issues
0 Security Hotspots
No data about Coverage
0.1% Duplication on New Code

See analysis details on SonarCloud

# [1.25.0](prism-agent-v1.24.0...prism-agent-v1.25.0) (2024-01-25) ### Features * **prism-agent:** add JWT auth support for agent-admin role ([#840](#840)) ([3ccd56e](3ccd56e)) * upgrade ZIO http client to improve performance ([#850](#850)) ([7aa9b4c](7aa9b4c)) Signed-off-by: Anton Baliasnikov <[email protected]>

Signed-off-by: David Poltorak <[email protected]> Signed-off-by: Benjamin Voiturier <[email protected]> Co-authored-by: Benjamin Voiturier <[email protected]> Signed-off-by: Shota Jolbordi <[email protected]>

# [1.25.0](prism-agent-v1.24.0...prism-agent-v1.25.0) (2024-01-25) ### Features * **prism-agent:** add JWT auth support for agent-admin role ([#840](#840)) ([3ccd56e](3ccd56e)) * upgrade ZIO http client to improve performance ([#850](#850)) ([7aa9b4c](7aa9b4c)) Signed-off-by: Anton Baliasnikov <[email protected]> Signed-off-by: Shota Jolbordi <[email protected]>

Signed-off-by: David Poltorak <[email protected]> Signed-off-by: Benjamin Voiturier <[email protected]> Co-authored-by: Benjamin Voiturier <[email protected]> Signed-off-by: Shota Jolbordi <[email protected]>

# [1.25.0](prism-agent-v1.24.0...prism-agent-v1.25.0) (2024-01-25) ### Features * **prism-agent:** add JWT auth support for agent-admin role ([#840](#840)) ([3ccd56e](3ccd56e)) * upgrade ZIO http client to improve performance ([#850](#850)) ([7aa9b4c](7aa9b4c)) Signed-off-by: Anton Baliasnikov <[email protected]> Signed-off-by: Shota Jolbordi <[email protected]>

Signed-off-by: David Poltorak <[email protected]> Signed-off-by: Benjamin Voiturier <[email protected]> Co-authored-by: Benjamin Voiturier <[email protected]> Signed-off-by: Shota Jolbordi <[email protected]>

# [1.25.0](prism-agent-v1.24.0...prism-agent-v1.25.0) (2024-01-25) ### Features * **prism-agent:** add JWT auth support for agent-admin role ([#840](#840)) ([3ccd56e](3ccd56e)) * upgrade ZIO http client to improve performance ([#850](#850)) ([7aa9b4c](7aa9b4c)) Signed-off-by: Anton Baliasnikov <[email protected]> Signed-off-by: Shota Jolbordi <[email protected]>

Signed-off-by: David Poltorak <[email protected]> Signed-off-by: Benjamin Voiturier <[email protected]> Co-authored-by: Benjamin Voiturier <[email protected]>

Signed-off-by: David Poltorak <[email protected]> Signed-off-by: Benjamin Voiturier <[email protected]> Co-authored-by: Benjamin Voiturier <[email protected]> Signed-off-by: Shota Jolbordi <[email protected]>

# [1.25.0](prism-agent-v1.24.0...prism-agent-v1.25.0) (2024-01-25) ### Features * **prism-agent:** add JWT auth support for agent-admin role ([#840](#840)) ([3ccd56e](3ccd56e)) * upgrade ZIO http client to improve performance ([#850](#850)) ([7aa9b4c](7aa9b4c)) Signed-off-by: Anton Baliasnikov <[email protected]> Signed-off-by: Shota Jolbordi <[email protected]>

Signed-off-by: David Poltorak <[email protected]> Signed-off-by: Benjamin Voiturier <[email protected]> Co-authored-by: Benjamin Voiturier <[email protected]> Signed-off-by: Shota Jolbordi <[email protected]>

Signed-off-by: David Poltorak <[email protected]> Signed-off-by: Benjamin Voiturier <[email protected]> Co-authored-by: Benjamin Voiturier <[email protected]>

Signed-off-by: David Poltorak <[email protected]> Signed-off-by: Benjamin Voiturier <[email protected]> Co-authored-by: Benjamin Voiturier <[email protected]> Signed-off-by: Shota Jolbordi <[email protected]>

# [1.25.0](prism-agent-v1.24.0...prism-agent-v1.25.0) (2024-01-25) ### Features * **prism-agent:** add JWT auth support for agent-admin role ([#840](#840)) ([3ccd56e](3ccd56e)) * upgrade ZIO http client to improve performance ([#850](#850)) ([7aa9b4c](7aa9b4c)) Signed-off-by: Anton Baliasnikov <[email protected]> Signed-off-by: Shota Jolbordi <[email protected]>

Signed-off-by: David Poltorak <[email protected]> Signed-off-by: Benjamin Voiturier <[email protected]> Co-authored-by: Benjamin Voiturier <[email protected]> Signed-off-by: Shota Jolbordi <[email protected]>

Signed-off-by: David Poltorak <[email protected]> Signed-off-by: Benjamin Voiturier <[email protected]> Co-authored-by: Benjamin Voiturier <[email protected]>

atala-dev added pollux prism-agent build infra mercury labels Jan 18, 2024

github-advanced-security bot found potential problems Jan 18, 2024

View reviewed changes

ghost changed the title ~~Feature/http client performance fix~~ feat(prism-agent): upgrade ZIO http client to improve performance Jan 18, 2024

ghost mentioned this pull request Jan 23, 2024

Add a security policy and vulnerability reporting process for the Open Enterprise Agent #836

Closed

atala-dev added the ci label Jan 24, 2024

ghost changed the title ~~feat(prism-agent): upgrade ZIO http client to improve performance~~ feat: upgrade ZIO http client to improve performance Jan 24, 2024

davepoltorak and others added 18 commits January 24, 2024 15:28

fix(prism-agent): metrics for background jobs and metric naming

7d0aba7

Signed-off-by: David Poltorak <[email protected]>

feat(prism-agent): add grafana dashboards, upgrade local perf testing

b6780d0

Signed-off-by: David Poltorak <[email protected]>

feat(prism-agent): add exception handling to k6 perf tests

a164361

Signed-off-by: David Poltorak <[email protected]>

feat(prism-agent): add integration test for metric endpoint

8bf64d2

Signed-off-by: David Poltorak <[email protected]>

feat(prism-agent): upgrade to ZIO HTTP RC4 to improve request perform…

3732c15

…ance Signed-off-by: David Poltorak <[email protected]>

chore(prism-agent): format docker-compose YAML file

d73e97d

Signed-off-by: David Poltorak <[email protected]>

chore(prism-agent): apply scalafmt to DidCommHttpServer

c8e5d6f

Signed-off-by: David Poltorak <[email protected]>

fix(prism-agent): validate 'content-type' header of incoming DIDComm …

2cce45d

…requests Signed-off-by: Benjamin Voiturier <[email protected]>

chore(prisma-gent): extract ZIO HTTP client params (timeouts + cp siz…

f9d6c3a

…e) to agent's configuration file Signed-off-by: Benjamin Voiturier <[email protected]>

fix(prism-agent): ensure response processing by HTTP client is part o…

c58d491

…f the active ZIO Scope Signed-off-by: Benjamin Voiturier <[email protected]>

feat(prism-agent): add security.md for security vuln reporting

91fec35

Signed-off-by: David Poltorak <[email protected]>

chore: update link to discord channel for security notices

2261fb5

Signed-off-by: David Poltorak <[email protected]>

fix(prism-agent): restore metric test integration test

8e38468

Signed-off-by: David Poltorak <[email protected]>

chore(prism-agent): shorten scenario name to meet linter rules

b19d7f3

Signed-off-by: David Poltorak <[email protected]>

chore(prism-agent): add linting hint for __ENV

4eb1574

Signed-off-by: David Poltorak <[email protected]>

chore(prism-agent): adjusted linting rulest to exclude ext k6chaijs

dced290

Signed-off-by: David Poltorak <[email protected]>

feat(prism-agent): remove unused imports and fix interpolation

b2c3d83

Signed-off-by: David Poltorak <[email protected]>

chore(prism-agent): restore schemaId in credential service

f5eef4d

Signed-off-by: David Poltorak <[email protected]>

chore: increase all acceptable group durations for smoke test

5125da9

Signed-off-by: David Poltorak <[email protected]>

shotexa approved these changes Jan 24, 2024

View reviewed changes

patlo-iog approved these changes Jan 25, 2024

View reviewed changes

pollux/lib/core/src/main/scala/io/iohk/atala/pollux/core/service/HttpURIDereferencerImpl.scala Outdated Show resolved Hide resolved

davepoltorak and others added 3 commits January 25, 2024 09:45

chore: remove commented out code to improve quality

1e7096c

Signed-off-by: David Poltorak <[email protected]>

chore: run scalafmt

05050e8

Signed-off-by: David Poltorak <[email protected]>

chore(prism-agent): improve perf when handling HTTP URI dereferencer …

53ae046

…error case Signed-off-by: Benjamin Voiturier <[email protected]>

bvoiturier force-pushed the feature/http-client-performance-fix branch from 980ad1d to 53ae046 Compare January 25, 2024 11:19

ghost merged commit 7aa9b4c into main Jan 25, 2024
11 checks passed

ghost deleted the feature/http-client-performance-fix branch January 25, 2024 12:44

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: upgrade ZIO http client to improve performance #850

feat: upgrade ZIO http client to improve performance #850

ghost commented Jan 18, 2024 •

edited by ghost

Loading

github-actions bot commented Jan 18, 2024 •

edited

Loading

github-actions bot commented Jan 18, 2024 •

edited

Loading

sonarqubecloud bot commented Jan 25, 2024

feat: upgrade ZIO http client to improve performance #850

feat: upgrade ZIO http client to improve performance #850

Conversation

ghost commented Jan 18, 2024 • edited by ghost Loading

Overview

Checklist

My PR contains...

My changes...

Documentation

Tests

github-actions bot commented Jan 18, 2024 • edited Loading

Unit Test Results

github-actions bot commented Jan 18, 2024 • edited Loading

Integration Test Results

sonarqubecloud bot commented Jan 25, 2024

Quality Gate passed

ghost commented Jan 18, 2024 •

edited by ghost

Loading

github-actions bot commented Jan 18, 2024 •

edited

Loading

github-actions bot commented Jan 18, 2024 •

edited

Loading