Unable to serialize database with dumpdata

[madchap]

Bug description
On a python3 dev branch (from june 26), I wanted to perform a dumpdata prior to upgrading.
The following command

docker exec -ti dojo_upstream_afterpy3_migration_uwsgi_1 bash -c 'python manage.py dumpdata --indent=4 -e sessions -e admin --natural-foreign --natural-primary -e contenttypes -e auth.permission' > datadump_prior_to_update.json

launches, performs up to a certain point before failing with this error:

CommandError: Unable to serialize database: 'MultiSelectField' object has no attribute '_get_val_from_obj'
}Exception ignored in: <generator object cursor_iter at 0x7f6ba391f660>
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1515, in cursor_iter
    cursor.close()
  File "/usr/local/lib/python3.7/site-packages/MySQLdb/cursors.py", line 86, in close
    while self.nextset():
  File "/usr/local/lib/python3.7/site-packages/MySQLdb/cursors.py", line 136, in nextset
    nr = db.next_result()
MySQLdb._exceptions.OperationalError: (2006, '')

Expected behavior
The dumpdata works.

Deployment method (select with an X)

• Kubernetes
• Docker
• setup.bash / legacy-setup.bash

[madchap]

FWIW, dumping only the findings works fine (dumpdata dojo.finding).

Can anyone reproduce this issue? @dr3dd589 @valentijnscholten @aaronweaver

This feature is so important in my eyes to be able to feel safer...

[aaronweaver]

@dr3dd589 If you can take a look.

Just a quick search, this may be the solution.

value = self._get_val_from_obj(obj) to value = self.value_from_object(obj)

goinnn/django-multiselectfield#74

[dr3dd589]

hello, @madchap I have tested it on my updated python3 docker version of DD and also on setup.bash and it works fine, I'm not able to reproduce this error. Need some more info on how to reproduce this.
test link

[madchap]

Do you think there is a remote chance it could be connected to my upgrading from 2.x to 3.x branch through the import, that also caused #1315 ?

I will dump the mysql, and see how I can redact it to hand over to you. I am guessing this is the only way for you to truly test that. Anything else that could help you?

My exact git reflog (the 2 most recent lines was to incorporate work from @propersam for the API notes endpoint)

$ git reflog
5b394518 (HEAD -> dev) HEAD@{0}: am: Fix flake8 issues
1cc0c358 HEAD@{1}: am: Add Notes Endpoint to Api version2
44724d03 (origin/dev) HEAD@{2}: checkout: moving from master to dev
dbaaf5a0 (origin/master, origin/HEAD, master) HEAD@{3}: clone: from https://github.com/DefectDojo/django-DefectDojo.git

[dr3dd589]

yes, maybe it is some kind of issue on upgrading DD on your system. You can test it by doing a fresh install of DD.

[madchap]

I've spent some time this morning writing some proxySQL rules to anonymize my mysql dump, which looks ok.

To make sure I had the same experience on my local copy of DD, I imported it (went ok) and ran a dumpdata. Same error.

I could use some help to identify this issue. If you're willing @dr3dd589, I could send you a link to a private download of this anonimized mysql dump for you to look at? Even if the fresh installs work, it's not going to help me, since I cannot afford to lose the data in this particular instance.

Hopefully, this will help later down the road also for potential troubleshooting; I can start a wiki page with proxySQL basic rules and some guidance?

[madchap]

I found the culprit. When you associate a JIRA configuration to a product, then dumpdata fails. Started with fresh config, and added little by little...

[madchap]

@Maffooch maybe you know something looking at git blame, seems you pushed some stuff a few weeks ago?

[aaronweaver]

It should still be able to serialize it. I think it's a 2->3 bug.

[madchap]

It should still be able to serialize it. I think it's a 2->3 bug.

It is not. My test was only on the latest dev (python3) branch.

[aaronweaver]

Let me clarify, I think the the object relation is breaking as a result of the code upgrade from 2 to 3.

[madchap]

@dr3dd589 would you have some time to look into this soon? Cheers.

[madchap]

So I just fetched the latest dev locally, re-added jira conf and association, and I cannot reproduce what I could a week ago 🤔

Could #1371 as a collateral (even though it is not clear how this could have?). Or something else?

[madchap]

I can no longer reproduce on fresh instance. I am guessing I had left-over stuff from a previous db schema or something from previous tests... sigh

[madchap]

Culprit probably hides in there: 6508e7e (@Maffooch ).

Saw this as well: https://stackoverflow.com/questions/52057583/python-2-3-django-migration-causes-field-parameter-type-change leading to https://code.djangoproject.com/ticket/23226, which does sound something that is awfully happening to me here after the migration?

[madchap]

Tried to revert the notifications model of 0001_initial.py (since it has the MultiSelectField), makemigrations and migrate (https://gist.github.com/madchap/8e5f751b55ccca713666e3ad0a80ea89) but I am getting this error when migrating:

docker exec -ti madchap_ddojo_uwsgi_1 bash -c 'python manage.py migrate'             INSERT  ✔  12:27:59
Operations to perform:
  Apply all migrations: admin, auditlog, auth, authtoken, contenttypes, custom_field, django_celery_results, dojo, sessions, sites, social_django, tagging, tastypie, watson
Running migrations:
  Applying dojo.0010_auto_20190724_1017...Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/migrate.py", line 234, in handle
    fake_initial=fake_initial,
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/migration.py", line 124, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/operations/fields.py", line 249, in database_forwards
    schema_editor.alter_field(from_model, from_field, to_field)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/schema.py", line 535, in alter_field
    old_db_params, new_db_params, strict)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/schema.py", line 648, in _alter_field
    old_default = self.effective_default(old_field)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/schema.py", line 233, in effective_default
    return field.get_db_prep_save(self._effective_default(field), self.connection)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/fields/__init__.py", line 789, in get_db_prep_save
    return self.get_db_prep_value(value, connection=connection, prepared=False)
  File "/usr/local/lib/python3.7/site-packages/multiselectfield/db/fields.py", line 140, in get_db_prep_value
    value = self.get_prep_value(value)
  File "/usr/local/lib/python3.7/site-packages/multiselectfield/db/fields.py", line 136, in get_prep_value
    return '' if value is None else ",".join(value)
TypeError: sequence item 0: expected str instance, int found

[madchap]

My anonymized sql dump is here, can be simply loaded with mysqldump on a new instance of defectdojo: https://send.firefox.com/download/a5cb53aab79d472f/#rkST0cU3qbTP2u2hD5icjQ

You'd need to load up a user fixture to have a valid user then.

The first 3 persons willing to help wins ;-) valid 7 days.

I hope a good soul can help... Cheers.

[madchap]

Quickly tried the hack presented at goinnn/django-multiselectfield#74 and it works. I can dumpdata fine.

(why didn't I try it before, go figure)

However, I am definitely no django guy as you have surely noticed by now, so I do not know if there is anything you want to do with this, such as implementing a class as described at goinnn/django-multiselectfield#74 (comment) ?

[madchap]

Btw, I am still being hit by this on a yet newer instance -- of course bringing forward data that lived through the python 2 -> python 3 transition.

Eg. could this suggestion be implemented without hindering anything else? or as a flag?
goinnn/django-multiselectfield#74 (comment)

[dr3dd589]

but this is happening only when you upgrade from python2 to python3 and now default it is patched for Django 2.0. So I don't think it would be great to implement this manually for python2 when it is already fixed. it may raise another problem.

[madchap]

Yea... Sucks to be the only person in the world to have migrated from py2 to py3 with some "prod" data :-/

I am going to go ahead and close this.

Cheers.