pytest.mark.django_db doesn't play nice with setup_module()

[Suor]

Here is my test code:

import pytest
pytestmark = pytest.mark.django_db

from handy.db import do_sql, fetch_val

def setup_module(fetch_val):
    print 1
    do_sql('''
        create table test (
            id int primary key,
            tag int not null
        );

        insert into test values (1, 10), (2, 20);
    ''')
    print 2

def test_fetch_val():
    print 3
    assert fetch_val('select min(id) from test') == 1

And I get DatabaseError: relation "test" does not exist from test_fetch_val(). I also get

1
2
Creating test database for alias 'default'...
3

captured output. Looks like setup_module() is called before database is created.

[pelme]

The django_db mark is based on fixtures, I'm not sure how to communicate between fixtures and the setup_* methods.

Could you use a module scoped fixture instead to achieve the same thing? I.e. something like

pytestmark = pytest.mark.usefixture('test_table')

@pytest.fixture(scope='module')
def test_table(db):
    do_sql('create table ...')

def test_fetch_val():
    assert fetch_val('...') == 1

[Suor]

Just tried. Didn't work. I get Database access not allowed, use the "django_db" mark to enable now.
Also, test_table() doesn't run (I tried to raise exception there)

[pelme]

Sorry, it should be usefixtures, not usefixture:

http://pytest.org/latest/fixture.html#usefixtures

[Suor]

Now I get ScopeMismatchError: You tried to access the 'function' scoped funcarg 'db' with a 'module' scoped request object, involved factories.

Removing db parameter from parameter results in fixture failure, removing scope='module' make thinks work, but then test_table() is run for every test, which I don't want.

[Suor]

I also tried

pytestmark = pytest.mark.django_db

@pytest.fixture(scope='module')
def test_table():
    do_sql('''
        create table test (
            id int primary key,
            tag int not null
        );

        insert into test values (1, 10), (2, 20);
    ''')

def test_fetch_val(test_table):
    assert fetch_val('select min(id) from test') == 1

def test_fetch_val2(test_table):
    assert fetch_val('select min(id) from test') == 1

which almost works, but second test fails somehow with DatabaseError: relation "test" does not exist. A complete mystery to me.

[pelme]

Django and the django_db marker works the way that each test case runs in its own transaction, therefore using django_db with a module level fixture that inserts data into the database does not really make sense.

What are your use case here? How come you are creating a database table "by hand" in your tests, and not with Django, but still want to use Django's test database/test database cursor?

Couldn't you achieve what you are after by just constructing a plain database cursor?

(The last example avoids the ScopeMismatchError, but it fails in the test instead because the transaction is rolled back after the first test, hence rolling back the test table.)

[Suor]

I am trying to test a couple of low level db utilities, which use django cursor internally - https://github.com/Suor/handy/blob/master/handy/db.py#L40

And I find it will be far messier to define a model in models.py and then fixtures in some django format to test that. That would be 3 files in total to test pretty basic functionality.

I just want to execute some initialization code before running tests so that I have some data to play with.

[Suor]

Oh, I made it work! Wrapping initialization SQL into begin; ... commit; helped.

Many thanks for all the tips you provided.

[pelme]

That is a hack that looks a bit fragile to me. I would probably set up an extra app that is only used for tests with a simple model that you can then use to run those functions against. You could then easily just use Django's ORM to populate the data in your tests. It is slightly more typing and a couple of extra files, but then it should be safe for future breakages.

[pelme]

If anyone else finds this issue: doing database setup in setup_function/setup_class/setup_module is not really supported or possible in any good way since pytest-django's database setup is based on fixtures.

The solution is to use a fixture which properly requests the db fixture:
http://pytest-django.readthedocs.org/en/latest/helpers.html#db

[rinatio]

@pelme
With the db fixture it is possible to run such setup on function scope only, not on class, module or session. For example if we need to fill in some records into database before class tests executed.

==================================== ERRORS ====================================
___________ ERROR at setup of TestHistoryAlerts.test_one_point_data ____________
ScopeMismatch: You tried to access the 'function' scoped fixture 'db' with a 'session' scoped request object, involved factories
tests/conftest.py:140:  def ts(db)
../../../../.virtualenvs/app/lib/python2.7/site-packages/pytest_django/fixtures.py:158:  def db(request, _django_db_setup, _django_cursor_wrapper)

Is there any other way to handle it?

[pelme]

Yes, that is a limitation of the db fixture as it is currently implemented. In PR #258 there has been some work towards making it possible to create database state that is bound by class/module/session scope.

[slykar]

Even tough this issue is pretty old, I've found it useful to find the solution to a problem I was having with pytest and pytest-django. Since pytest 3.5.0 there's an issue when using django_db mark and module level fixtures.

Below was working before 3.5.0 (at least it wasn't causing any issues).

@pytest.fixture(scope='module')
def user():
    return User.objects.create(...)

@pytest.mark.django_db
def test_some_case(user):
    pass

After 3.5.0 it raises

Failed: Database access not allowed, use the "django_db" mark, or the "db" or "transactional_db" fixtures to enable it.

The solution was to remove scope='module'.

[mahadi]

+1 For me, just switching from some initial Djano unit tests, this is a major drawback, when initializing the database for my API test cases. Actually supporting session/module database fixtures should be a basic feature. Any suggestions how to solve this?

[matthewpull]

+1