Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: fastapi schema-registry module with DI and E2E tests #995

Open
wants to merge 30 commits into
base: jjaakola-aiven-fastapi
Choose a base branch
from
Open

feat: fastapi schema-registry module with DI and E2E tests #995

wants to merge 30 commits into from

Conversation

nosahama
Copy link
Contributor

@nosahama nosahama commented Nov 27, 2024 •
edited
Loading

About this change - What it does

Some changes based off the work from the jjaakola-aiven-fastapi branch, majorly:

  • integrations tests which are failing need to be checked and resolved
    • some of the tests can be moved to the e2e folder
    • it'd be nice to stop spinning up all these kafka processes
    • it'd be nice to move this into the docker tests environment
    • i am not making any changes to the integration tests setup, so it's clear that we aren't changing logic that should verify the operability of our services
    • the integration tests fail now with issues spinning up kafka, so we can check that now or after we merge this work
  • unit tests now run within a docker compose environment
  • e2e tests were introduced, which also run within a docker compose environment
    • the plan is to extract some test out of the integration tests folder slowly into the e2e tests folder
    • e2e tests run against the karapace docker compose services: i.e. schema-registry/rest rather than local processes
  • karapace schema registry is now it's own module which houses all the FastAPI related things
  • karapace rest proxy still runs on rapu
  • we add dependency injection to the project
    • wiring the dependencies for both sr and rest
    • config is loaded on app startup
    • dependencies are injected into route handlers and other modules
    • modules with dependencies are wired on app startup
  • config is now based on pydantic-settings
    • settings are loaded via an .env file
    • base config is added as a yaml and packaged with the built python library
    • base config loads the reference to the .env file needed to populate the rest of the config
  • commits show progress
  • initial draft PR schema_registry: standalone module for SR with basic dependency injection #994 was rushed and contained more changes, this is a split out of that PR which will be closed at some point
  • we would need to check the coverage mechanism now with the new module and split

jjaakola-aiven and others added 28 commits November 13, 2024 13:09
@jjaakola-aiven
chore: pip-compile to upgrade requirements
8950b00
@jjaakola-aiven
chore: upgrade requirement versions
a43761a
@jjaakola-aiven
wip: Karapace SR runs on FastAPI
97f2f69
@jjaakola-aiven
fix: omit null fields in schema get response
5879107
@jjaakola-aiven
fix: healthcheck response type, interactive doc works
4b2bb13
@jjaakola-aiven
fix: tests/integration/test_schema.py passes
c2ddf18
@jjaakola-aiven
chore: remove trail from test_schema.py integration test
3222180
@jjaakola-aiven
fix: healthcheck corrected
3dc98c8
@jjaakola-aiven
chore: remove trail from test_schema_registry_mode.py
3f35858
@jjaakola-aiven
fix: corrected test_karapace.py test
af8a046
@jjaakola-aiven
fix: test_schema_reader.py integration test corrected
3855dae
@jjaakola-aiven
fix: added reference support for protobuf back
c677ee0
@jjaakola-aiven
fix: depencies integration test corrected
558a40c
@jjaakola-aiven
fix: corrected master coordinator test
cb449be
@jjaakola-aiven
fix: add missing root router file
197fe5e
@jjaakola-aiven
fix: add missing content type and errors files
5236889
@jjaakola-aiven
fix: added forwarding back
eac87b0
@jjaakola-aiven
fix: fixed remote client test
52c3f0f
@jjaakola-aiven
chore: added missing license headers
17c90f7
@jjaakola-aiven
fix: basic and no authentication integrated
1363d55
@jjaakola-aiven
fix: fixed schema response to have schemaType
46600b8
@jjaakola-aiven
fix: run Karapace as a module, __main__
4b09f9e
@jjaakola-aiven
fix: corrected backup tests
60a8086
@nosahama
feat: move schema-registry to own module with DI
eac4f40 
- we create a standalone module for SR related components

- we use DI to wire together the SR dependencies

- we move the routers to own folder

- we move the config initialization to DI and app startup
@nosahama
chore: drop manual dependencies in-favour of DI
1927a6b
@nosahama
chore: fix config and set defaults
279a310
@nosahama
feat: add karapace docker CLI
5524dcc 
- we add a docker compose service which would act like a CLI

- we add a dev Dockerfile

- we drop python3.9 from the tests matrix as pydantic-settings has issues with python3.9

- we add .env files for both registry and rest
@nosahama
feat: run unit-tests in the cli docker container
408bfd7 
- we replace use the karapace container to load the default config
@nosahama nosahama requested review from a team as code owners November 27, 2024 10:59
@nosahama nosahama marked this pull request as draft November 27, 2024 12:18
@nosahama nosahama marked this pull request as ready for review November 27, 2024 12:19
@nosahama nosahama force-pushed the nosahama/fastapi-sr-module-with-DI-e2e-tests branch 5 times, most recently from 74c54af to d336032 Compare November 27, 2024 12:39
nosahama
nosahama commented Nov 27, 2024
View reviewed changes
bin/smoke-test-registry.sh
@@ -6,6 +6,7 @@ for ((i = 0; i <= retries; i++)); do
response=$(
curl --silent --verbose --fail --request POST \
--header 'Content-Type: application/vnd.schemaregistry.v1+json' \
--header 'Authorization: Basic Og==' \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This are empty credentials, i.e. username="" | password="", this is needed as now on FastAPI we require the credentials to be at least present, even if it's empty, we can revise this later as it'd be now a breaking change, but shouldn't hurt too much.

nosahama
nosahama commented Nov 27, 2024
View reviewed changes
container/Dockerfile.dev
@@ -0,0 +1,62 @@
# Current versions of avro and zstandard don't yet have wheels for 3.11.
FROM python:3.10.11-bullseye AS builder
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to Dockerfile, we install dev dependencies and take out the healthcheck.

@nosahama
feat: e2e tests within docker
f52fe6c 
- we split out e2e tests into own folder

- e2e tests will run against the service in docker compose

- we start by splitting out the prometheus and kafka e2e tests which do not need local running services
@nosahama nosahama force-pushed the nosahama/fastapi-sr-module-with-DI-e2e-tests branch from d336032 to f52fe6c Compare November 27, 2024 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@nosahama @jjaakola-aiven