Can't DROP 'product_grade_'

[madchap]

Bug description
Pulling from dev, and doing a docker-compose up out of fresh checkout and docker image local build.

initializer_1   | Operations to perform:
initializer_1   |   Apply all migrations: admin, auditlog, auth, authtoken, contenttypes, custom_field, django_celery_results, dojo, sessions, sites, social_django, tagging, tastypie, watson
initializer_1   | Running migrations:
initializer_1   |   Applying admin.0003_logentry_add_action_flag_choices... OK
initializer_1   |   Applying auth.0009_alter_user_last_name_max_length... OK
initializer_1   |   Applying auth.0010_alter_group_name_max_length... OK
initializer_1   |   Applying auth.0011_update_proxy_permissions... OK
initializer_1   |   Applying custom_field.0003_customfield_mask... OK
initializer_1   |   Applying custom_field.0004_django2_upgrade... OK
initializer_1   |   Applying django_celery_results.0004_auto_20190516_0412... OK
initializer_1   |   Applying dojo.0006_django2_upgrade...Traceback (most recent call last):
initializer_1   |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
initializer_1   |     return self.cursor.execute(sql, params)
initializer_1   |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/mysql/base.py", line 71, in execute
initializer_1   |     return self.cursor.execute(query, args)
initializer_1   |   File "/usr/local/lib/python3.7/site-packages/MySQLdb/cursors.py", line 206, in execute
initializer_1   |     res = self._query(query)
initializer_1   |   File "/usr/local/lib/python3.7/site-packages/MySQLdb/cursors.py", line 312, in _query
initializer_1   |     db.query(q)
initializer_1   |   File "/usr/local/lib/python3.7/site-packages/MySQLdb/connections.py", line 224, in query
initializer_1   |     _mysql.connection.query(self, query)
initializer_1   | MySQLdb._exceptions.OperationalError: (1091, "Can't DROP 'product_grade_'; check that column/key exists")

Steps to reproduce
Steps to reproduce the behavior:

1. Clean clone from https, and git checkout dev
2. Remove all local images to force rebuild
3. setEnv.sh dev
4. docker-compose up
5. See error

Expected behavior
A clear and concise description of what you expected to happen.

Deployment method (select with an X)

• Kubernetes
• Docker
• setup.bash / legacy-setup.bash

[dr3dd589]

I had also faced this error. this error raise due to there is no column name product_grade_ in system_setting model to drop. I fixed this by commenting below code in 0006_django2_upgrade.py

        migrations.RemoveField(
            model_name='system_settings',
            name='product_grade_',
        ),

[madchap]

Thanks for the tip.

After that, I get a django.db.utils.OperationalError: (1060, "Duplicate column name 'product_grade_b'") ;-) this is again after wiping all out clean. Nevermind, the mysql volume was not. It passed after that, on to the next error...

[dr3dd589]

it means in your database has already column name product_grade_b. I think maybe then you should remove this one too in the same file:-

        migrations.AddField(
            model_name='system_settings',
            name='product_grade_b',
            field=models.IntegerField(default=80, help_text="Percentage score for a 'B' >=", verbose_name='Grade B'),
        ),

[madchap]

Yes. I was looking if there was a way in django to do a "if not exist", but nothing in the AddField() method anyways.

[madchap]

hehe and now I guess a collateral:

initializer_1   | Migrations for 'dojo':
initializer_1   |   dojo/db_migrations/0008_remove_system_settings_product_grade_.py
initializer_1   |     - Remove field product_grade_ from system_settings
initializer_1   | Traceback (most recent call last):
initializer_1   |   File "manage.py", line 10, in <module>
initializer_1   |     execute_from_command_line(sys.argv)
initializer_1   |   File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
initializer_1   |     utility.execute()
initializer_1   |   File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
initializer_1   |     self.fetch_command(subcommand).run_from_argv(self.argv)
initializer_1   |   File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv
initializer_1   |     self.execute(*args, **cmd_options)
initializer_1   |   File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute
initializer_1   |     output = self.handle(*args, **options)
initializer_1   |   File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 83, in wrapped
initializer_1   |     res = handle_func(*args, **kwargs)
initializer_1   |   File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/makemigrations.py", line 182, in handle
initializer_1   |     self.write_migration_files(changes)
initializer_1   |   File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/makemigrations.py", line 221, in write_migration_files
initializer_1   |     with open(writer.path, "w", encoding='utf-8') as fh:
initializer_1   | PermissionError: [Errno 13] Permission denied: '/app/dojo/db_migrations/0008_remove_system_settings_product_grade_.py'

[madchap]

After clearing again everything (docker container prune && docker volume prune) and having the RemoveField() removed, I still have the error above with 008 it goes further but bombs still with bunch of messages on both uwsgi and initializer.

initializer_1 | django.db.utils.IntegrityError: Problem installing fixture '/app/dojo/fixtures/system_settings.json': Could not load dojo.System_Settings(pk=1): (1364, "Field 'product_grade_' doesn't have a default value")

I am guessing the uwsgi error could be related:

uwsgi_1         | Traceback (most recent call last):
uwsgi_1         |   File "/usr/local/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
uwsgi_1         |     response = get_response(request)
uwsgi_1         |   File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
uwsgi_1         |     response = self.process_exception_by_middleware(e, request)
uwsgi_1         |   File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
uwsgi_1         |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
uwsgi_1         |   File "./dojo/user/views.py", line 138, in alertcount
uwsgi_1         |     count = Alerts.objects.filter(user_id=request.user).count()
uwsgi_1         |   File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
uwsgi_1         |     return getattr(self.get_queryset(), name)(*args, **kwargs)
uwsgi_1         |   File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 892, in filter
uwsgi_1         |     return self._filter_or_exclude(False, *args, **kwargs)
uwsgi_1         |   File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 910, in _filter_or_exclude
uwsgi_1         |     clone.query.add_q(Q(*args, **kwargs))
uwsgi_1         |   File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/query.py", line 1290, in add_q
uwsgi_1         |     clause, _ = self._add_q(q_object, self.used_aliases)
uwsgi_1         |   File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/query.py", line 1318, in _add_q
uwsgi_1         |     split_subq=split_subq, simple_col=simple_col,
uwsgi_1         |   File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/query.py", line 1224, in build_filter
uwsgi_1         |     self.check_related_objects(join_info.final_field, value, join_info.opts)
uwsgi_1         |   File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/query.py", line 1086, in check_related_objects
uwsgi_1         |     for v in value:
uwsgi_1         |   File "/usr/local/lib/python3.7/site-packages/django/utils/functional.py", line 257, in inner
uwsgi_1         |     return func(self._wrapped, *args)
uwsgi_1         | TypeError: 'AnonymousUser' object is not iterable

celeryworker_1  | Traceback (most recent call last):
celeryworker_1  |   File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 385, in trace_task
celeryworker_1  |     R = retval = fun(*args, **kwargs)
celeryworker_1  |   File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 648, in __protected_call__
celeryworker_1  |     return self.run(*args, **kwargs)
celeryworker_1  |   File "/app/dojo/tasks.py", line 281, in async_dupe_delete
celeryworker_1  |     system_settings = System_Settings.objects.get()
celeryworker_1  |   File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
celeryworker_1  |     return getattr(self.get_queryset(), name)(*args, **kwargs)
celeryworker_1  |   File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 408, in get
celeryworker_1  |     self.model._meta.object_name
celeryworker_1  | dojo.models.System_Settings.DoesNotExist: System_Settings matching query does not exist.

[madchap]

So it works OK from a clean slate. This means this does not work when trying to upgrade from a previous version, unless tweaking the migration file knowing what you're doing... :-/

[pratik0101]

can u plz tell the impact

[Maffooch]

I've been struggling to understand why this is happening as well. Notice when applying migrations, dojo 0001-0005 are not applied. That product_grade_ column is added in 0001 but then removed in 0006. I can only assume they are not applied because they already have been on a previous/current build. In that case, there should not be any issue with removing a column that already exists. Very aggravating.

[aaronweaver]

I believe you can do --fake to skip over it. One of the test cases I had thought about awhile ago is that we should have a database at a prior version in the repo and then attempt to apply the migration to that database. This check would have caught this issue.

There is more robust testing now in migrations but I think this last test would be worthwhile.

[alles-klar]

I think during the python3 migration all stings with a 'b' in front like b'Pacific/Truk' were replaced to 'Pacific/Truk' because with the 'b' in front there are not python3 compatible. If this was done automatically 'product_grade_b' results in product_grade_'.

I will provide a PR where 'product_grade_' is renamed in 'product_grade_b' again. Just like before the python3 migration.