Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update conda-store-server image + use public auth_schema module for AuthenticationToken #2931

Merged
merged 6 commits into from
Feb 6, 2025

Conversation

soapy1
Copy link
Contributor

@soapy1 soapy1 commented Jan 31, 2025

In the latest release (2025.1.1) conda store moved the authentication schema to a public module. PR here:
conda-incubator/conda-store#1046.

This PR:

  • updates conda-store-server to 2025.2.1
  • updates keycloak auth to use the new (public) auth module

What does this implement/fix?

Put a x in the boxes that apply

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds a feature)
  • Breaking change (fix or feature that would cause existing features not to work as expected)
  • Documentation Update
  • Code style update (formatting, renaming)
  • Refactoring (no functional changes, no API changes)
  • Build related changes
  • Other (please describe):

Testing

  • Did you test the pull request locally?
  • Did you add new tests?

@soapy1 soapy1 force-pushed the update-conda-store-config branch from 0f97b7e to 5266076 Compare January 31, 2025 16:37
@soapy1 soapy1 force-pushed the update-conda-store-config branch from 3ff4ef5 to f9e08b4 Compare January 31, 2025 16:58
@marcelovilla marcelovilla mentioned this pull request Feb 3, 2025
10 tasks
Copy link
Member

@marcelovilla marcelovilla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @soapy1! We'll merge this after we've tested it on a couple deployments.

@viniciusdc
Copy link
Contributor

Hey @soapy1 I did another full test with this PR today, to also make sure the previous issue with the UI/DB was fixed. Basicaly my test consisted of 3 steps:

  • Deploying nebari with its latest available release 2024.12.1, and then I:
    • Created two namespaces (groups in keycloak) called custom1 and custom2 respectively to test the environment builds and also serve as a way to check the DB schema later on;
    • Granted user access to both, and create versy simple environemtns in both (python, numpy, pandas)
  • Manulay overrides the DB for the namespaces metadata_ collum to contain the null attrs instead of {}. Did that to only nebari-git, analyst, admin, global, developer namespaces -- to replicate a mix state similar to the original scenario when I first spoted this bug
  • Upgrade nebari with the changes in your branch here;

The deployment faill, due to a crash loop with the conda-store-server as seen bellow:

2025-02-04 19:28:42      INFO CondaStoreServer: 203: Running conda-store with store directory: /home/conda/
2025-02-04 19:28:42   WARNING CondaStoreServer: 214: Config option `access_token_url` not recognized by `KeyCloakAuthentication`.  Did you mean `access_scope`?
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade bf065abf375b -> 89637f546129, remove conda package build channel
Traceback (most recent call last):
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1964, in _exec_single_context
   self.dialect.do_execute(
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 942, in do_execute
   cursor.execute(statement, parameters)
psycopg2.errors.UniqueViolation: could not create unique index "_conda_package_build_uc"
DETAIL:  Key (package_id, subdir, build, build_number, sha256)=(9611, linux-64, py39h06a4308_0, 0, ef207471bfc4c2e04d01ec01c85d0a0588432822b4e50ab582751effa9fd84fc) is duplicated.
CONTEXT:  parallel worker


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/opt/conda/envs/conda-store-server/bin/conda-store-server", line 8, in <module>
   sys.exit(main())
            ^^^^^^
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/traitlets/config/application.py", line 1075, in launch_instance
   app.start()
 File "/opt/conda-store-server/conda_store_server/_internal/server/app.py", line 391, in start
   dbutil.upgrade(self.conda_store.config.database_url)
 File "/opt/conda-store-server/conda_store_server/_internal/dbutil.py", line 108, in upgrade
   command.upgrade(config=alembic_cfg, revision=revision)
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/alembic/command.py", line 406, in upgrade
   script.run_env()
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/alembic/script/base.py", line 586, in run_env
   util.load_python_file(self.dir, "env.py")
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
   module = load_module_py(module_id, path)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
   spec.loader.exec_module(module)  # type: ignore
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "<frozen importlib._bootstrap_external>", line 999, in exec_module
 File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
 File "/opt/conda-store-server/conda_store_server/_internal/alembic/env.py", line 92, in <module>
   run_migrations_online()
 File "/opt/conda-store-server/conda_store_server/_internal/alembic/env.py", line 86, in run_migrations_online
   context.run_migrations()
 File "<string>", line 8, in run_migrations
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/alembic/runtime/environment.py", line 946, in run_migrations
   self.get_context().run_migrations(**kw)
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/alembic/runtime/migration.py", line 623, in run_migrations
   step.migration_fn(**kw)
 File "/opt/conda-store-server/conda_store_server/_internal/alembic/versions/89637f546129_remove_conda_package_build_channel.py", line 101, in upgrade
   with op.batch_alter_table("conda_package_build") as batch_op:
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/opt/conda/envs/conda-store-server/lib/python3.12/contextlib.py", line 144, in __exit__
   next(self.gen)
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/alembic/operations/base.py", line 398, in batch_alter_table
   impl.flush()
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/alembic/operations/batch.py", line 116, in flush
   fn(*arg, **kw)
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/alembic/ddl/impl.py", line 387, in add_constraint
   self._exec(schema.AddConstraint(const))
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/alembic/ddl/impl.py", line 247, in _exec
   return conn.execute(construct, params)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1416, in execute
   return meth(
          ^^^^^
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/sqlalchemy/sql/ddl.py", line 180, in _execute_on_connection
   return connection._execute_ddl(
          ^^^^^^^^^^^^^^^^^^^^^^^^
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1527, in _execute_ddl
   ret = self._execute_context(
         ^^^^^^^^^^^^^^^^^^^^^^
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1843, in _execute_context
   return self._exec_single_context(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1983, in _exec_single_context
   self._handle_dbapi_exception(
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2352, in _handle_dbapi_exception
   raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1964, in _exec_single_context
   self.dialect.do_execute(
 File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 942, in do_execute
   cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) could not create unique index "_conda_package_build_uc"
DETAIL:  Key (package_id, subdir, build, build_number, sha256)=(9611, linux-64, py39h06a4308_0, 0, ef207471bfc4c2e04d01ec01c85d0a0588432822b4e50ab582751effa9fd84fc) is duplicated.
CONTEXT:  parallel worker

[SQL: ALTER TABLE conda_package_build ADD CONSTRAINT _conda_package_build_uc UNIQUE (package_id, subdir, build, build_number, sha256)]
(Background on this error at: https://sqlalche.me/e/20/gkpj)
Stream closed EOF for dev/nebari-conda-store-server-7b6f7ccf59-4zntv (conda-store-server)

Since this is related to conda-incubator/conda-store#1070, I then tested 2025.1.2-rc1, which works just fine without errors, and the UI can be seen again (even with the DB in that mix state):

image

image

@viniciusdc
Copy link
Contributor

I will do a full integration test with 2025.1.2 and if all works, maybe we can get that released as well?

@soapy1 soapy1 changed the title Use public auth_schema module for AuthenticationToken Update conda-store-server image + use public auth_schema module for AuthenticationToken Feb 4, 2025
@soapy1 soapy1 force-pushed the update-conda-store-config branch from 0f5206d to dbea89d Compare February 4, 2025 23:35
@viniciusdc
Copy link
Contributor

Manual run suceeded, also ran the conda-store-server user journey tests bellow:

❯ python -m pytest -m "user_journey"
================================================================= test session starts =================================================================
platform linux -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0
rootdir:.../conda-store/conda-store-server
configfile: pyproject.toml
plugins: anyio-4.8.0, alembic-0.11.1, playwright-0.7.0, cov-6.0.0, mock-3.14.0, base-url-2.1.0, celery-0.0.0a1
collected 316 items / 309 deselected / 7 selected                                                                                                     

tests/user_journeys/test_user_journeys.py .......                                                                                               [100%]

================================================================== warnings summary ===================================================================

tests/user_journeys/test_user_journeys.py::test_admin_user_can_create_environment[tests/user_journeys/test_data/simple_environment.yaml]
tests/user_journeys/test_user_journeys.py::test_admin_login_and_delete_shared_environment[tests/user_journeys/test_data/simple_environment.yaml]
tests/user_journeys/test_user_journeys.py::test_user_login_and_create_shared_environment[tests/user_journeys/test_data/simple_environment.yaml]
tests/user_journeys/test_user_journeys.py::test_admin_set_active_build
tests/user_journeys/test_user_journeys.py::test_failed_build_logs
tests/user_journeys/test_user_journeys.py::test_cancel_build
tests/user_journeys/test_user_journeys.py::test_get_lockfile
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================= 7 passed, 309 deselected, 41 warnings in 196.07s (0:03:16) ==============================================

@viniciusdc
Copy link
Contributor

Integration tests are failing due to #2933, working on a fix

@viniciusdc
Copy link
Contributor

Tested locally, works like a charm. Integration tests are falling due to a separate issue and are addressed as part of #2943 . I will disregard this to unblock the release candidate process. Amazing work @soapy1 !

@viniciusdc viniciusdc merged commit 5ea1874 into nebari-dev:main Feb 6, 2025
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done 💪🏾
Development

Successfully merging this pull request may close these issues.

3 participants