Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to initialize multi-node MAAS cluster simultaneously #315

Open
skatsaounis opened this issue Jan 28, 2025 · 0 comments
Open

Failed to initialize multi-node MAAS cluster simultaneously #315

skatsaounis opened this issue Jan 28, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@skatsaounis
Copy link
Collaborator

Steps to reproduce

juju add-machine -n 4 --constraints="cores=2 mem=4G virt-type=virtual-machine" [email protected]
juju deploy postgresql --channel 14/candidate --revision 553 --to 0 --config experimental_max_connections=250 --config plugin_audit_enable=false
juju deploy maas-region --channel 3.5/edge -n 3 --to 1,2,3
juju integrate postgresql maas-region

Expected behavior

The database is initialized and the MAAS region controllers are up and running, forming an HA cluster

Actual behavior

All 3 MAAS region controllers are running the database migrations simultaneously, thus, one of them succeeds and the other two fail to be initialized.

Versions

Operating system: Jammy 22.04

Juju CLI: 3.6.2-genericlinux-amd64

Juju agent: 3.6.2

Charm revision: 74

Log output

2025-01-28 14:59:08 INFO unit.maas-region/19.juju-log server.go:326 maas-db:70: MAAS database credentials received for user 'relation-70'
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60 Traceback (most recent call last):
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 82, in _execute
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     return self.cursor.execute(sql)
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60 psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "pg_class_relname_nsp_index"
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60 DETAIL:  Key (relname, relnamespace)=(auth_permission_id_seq, 2200) already exists.
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60 
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60 
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60 The above exception was the direct cause of the following exception:
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60 
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60 Traceback (most recent call last):
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/bin/maas-region", line 8, in <module>
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     sys.exit(run())
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/lib/python3.10/site-packages/maasserver/region_script.py", line 81, in run
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     run_django(is_snap, is_devenv)
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/lib/python3.10/site-packages/maasserver/region_script.py", line 70, in run_django
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     management.execute_from_command_line()
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     utility.execute()
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 413, in execute
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     self.fetch_command(subcommand).run_from_argv(self.argv)
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/core/management/base.py", line 354, in run_from_argv
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     self.execute(*args, **cmd_options)
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/core/management/base.py", line 398, in execute
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     output = self.handle(*args, **options)
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/lib/python3.10/site-packages/maasserver/management/commands/dbupgrade.py", line 250, in handle
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     call_command(
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 181, in call_command
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     return command.execute(*args, **defaults)
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/core/management/base.py", line 398, in execute
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     output = self.handle(*args, **options)
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/core/management/base.py", line 89, in wrapped
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     res = handle_func(*args, **kwargs)
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/core/management/commands/migrate.py", line 244, in handle
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     post_migrate_state = executor.migrate(
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 117, in migrate
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 227, in apply_migration
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     state = migration.apply(state, schema_editor)
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/db/migrations/migration.py", line 126, in apply
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/db/migrations/operations/models.py", line 92, in database_forwards
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     schema_editor.create_model(model)
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/db/backends/base/schema.py", line 345, in create_model
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     self.execute(sql, params or None)
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/db/backends/base/schema.py", line 145, in execute
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     cursor.execute(sql, params)
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 66, in execute
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     return executor(sql, params, many, context)
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 79, in _execute
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     with self.db.wrap_database_errors:
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/db/utils.py", line 90, in __exit__
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     raise dj_exc_value.with_traceback(traceback) from exc_value
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 82, in _execute
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     return self.cursor.execute(sql)
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60 django.db.utils.IntegrityError: duplicate key value violates unique constraint "pg_class_relname_nsp_index"
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60 DETAIL:  Key (relname, relnamespace)=(auth_permission_id_seq, 2200) already exists.
2025-01-28 14:59:12 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60 
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60 Traceback (most recent call last):
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/bin/maas", line 8, in <module>
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     sys.exit(main())
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/lib/python3.10/site-packages/maascli/__init__.py", line 39, in main
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     options.execute(options)
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/lib/python3.10/site-packages/maascli/snap.py", line 433, in __call__
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     raise exc
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/lib/python3.10/site-packages/maascli/snap.py", line 430, in __call__
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     self.handle(options)
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/lib/python3.10/site-packages/maascli/snap.py", line 698, in handle
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     self._finalize_init(mode)
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/lib/python3.10/site-packages/maascli/snap.py", line 712, in _finalize_init
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     perform_work(
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/lib/python3.10/site-packages/maascli/snap.py", line 363, in perform_work
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     return cmd(*args, **kwargs)
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/snap/maas/38185/lib/python3.10/site-packages/maascli/snap.py", line 338, in migrate_db
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     subprocess.check_call(
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60   File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60     raise CalledProcessError(retcode, cmd)
2025-01-28 14:59:13 WARNING unit.maas-region/19.maas-db-relation-changed logger.go:60 subprocess.CalledProcessError: Command '['/snap/maas/38185/bin/maas-region', 'dbupgrade']' returned non-zero exit status 1.
ubuntu@juju-76d202-36:~$ sudo maas status
Service          Startup   Current  Since
apiserver        enabled   active   today at 15:00 UTC
bind9            disabled  active   today at 15:00 UTC
http             disabled  active   today at 15:00 UTC
ntp              disabled  active   today at 15:00 UTC
proxy            disabled  active   today at 15:01 UTC
regiond          enabled   active   today at 15:00 UTC
syslog           disabled  active   today at 15:00 UTC
temporal         disabled  active   today at 15:00 UTC
temporal-worker  disabled  active   today at 15:00 UTC

ubuntu@juju-76d202-35:~$ sudo maas status
MAAS is not configured

ubuntu@juju-76d202-34:~$ sudo maas status
MAAS is not configured

Additional context

@skatsaounis skatsaounis added the bug Something isn't working label Jan 28, 2025
@skatsaounis skatsaounis changed the title Failed to initilaize many node MAAS cluster simultaneously Failed to initialize many node MAAS cluster simultaneously Jan 30, 2025
@skatsaounis skatsaounis changed the title Failed to initialize many node MAAS cluster simultaneously Failed to initialize multi-node MAAS cluster simultaneously Jan 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant