MongoDB: error while impersonating user

[ftcmnc]

I have this

    if request.vars.user_id:
        auth.impersonate(request.vars.user_id)

that produces this error:

<type 'exceptions.OverflowError'> MongoDB can only handle up to 8-byte ints

Version

web2py™     Version 2.11.2-stable+timestamp.2015.05.30.16.33.24
Python      Python 2.7.6: /usr/local/bin/uwsgi (prefix: /usr)

Traceback

Traceback (most recent call last):
    File "/home/www/web2py/gluon/restricted.py", line 227, in restricted
        exec ccode in environment
    File "/home/www/web2py/applications/emprega/controllers/default.py", line 1212, in <module>
    File "/home/www/web2py/gluon/globals.py", line 412, in <lambda>
        self._caller = lambda f: f()
    File "/home/www/web2py/applications/emprega/controllers/default.py", line 506, in user
        auth.impersonate(request.vars.user_id)
    File "/home/www/web2py/gluon/tools.py", line 3475, in impersonate
        user_id):
    File "/home/www/web2py/gluon/tools.py", line 3776, in has_permission
        == record_id).select(permission.group_id)
    File "/home/www/web2py/gluon/packages/dal/pydal/objects.py", line 1986, in select
        return adapter.select(self.query,fields,attributes)
    File "/home/www/web2py/gluon/packages/dal/pydal/adapters/mongo.py", line 359, in select
        for record in mongo_list_dicts:
    File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 983, in next
        if len(self.__data) or self._refresh():
    File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 908, in _refresh
        self.__read_preference))
    File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 813, in __send_message
        **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 744, in _send_message_with_response
        exhaust)
    File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 755, in _reset_on_error
        return func(*args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/pymongo/server.py", line 85, in send_message_with_response
        set_slave_okay, sock_info.is_mongos)
    File "/usr/local/lib/python2.7/dist-packages/pymongo/message.py", line 107, in get_message
        self.ntoreturn, spec, self.fields, self.codec_options)
OverflowError: MongoDB can only handle up to 8-byte ints

It was working well on the previous version

[stephenrauch]

MongoDB object IDs are 12 bytes and will not fit into an 8 byte integer. Problem is in gluon/tools.py at line 2081:

db.define_table(
    settings.table_permission_name,
    Field('group_id', reference_table_group,
          label=self.messages.label_group_id),
    Field('name', default='default', length=512,
          label=self.messages.label_name,
          requires=is_not_empty),
    Field('table_name', length=512,
          label=self.messages.label_table_name),
    Field('record_id', 'integer', default=0,            # field type is wrong for MongoDB
          label=self.messages.label_record_id,
          requires=IS_INT_IN_RANGE(0, 10 ** 9)),

Specifically 'record_id' type should likely be 'id' not 'integer'. Also the requires clause will need a wider range. Changing this field to 'id' fixes the problem in my tests on Mongo, but I have no easy way to know how it would impact other databases.

I don't how or if this bug can be moved over to web2py.

[gi0baro]

I think this is web2py related and not pydal. @mdipierro ?

[ilvalle]

@stephenrauch switching the type to 'id' is not feasible for all adapters. For example on postgres the type 'id' is converted into a SERIAL PRIMARY KEY.
Switching to 'id' works with mondo because such fields are converted in object_id by the mongo adapter

[gi0baro]

@ilvalle ok, but still I think is web2py related and not pydal related.

[stephenrauch]

Being new to pydal, let me see if I understand the design:

1. "id" means primary key.
2. "reference " means foreign key to a specific table.
3. "???" is there such a thing as a foreign key to a non-specific table?

'record_id' in the above use case isn't a primary key, so that means it is a foreign key. Unless bullet 3 above is a thing, it is definitely a web2py issue and the field type should be changed to "reference

". And taking a quick look at the code it should likely be:

    Field('record_id', 'reference permission', default=0,

[gi0baro]

@stephenrauch the change to reference type is not a solution for the auth module of web2py. But I agree with your point.
@mdipierro I think this is web2py related and requires a fix into the auth module. What do you think?

[mdipierro]

We have to make a decision here:
solution 1) in web2py change Field('record_id', 'integer') into Field('record_id', 'string') for MongoDB only
solution 2) in web2py change Field('record_id', 'integer') into Field('record_id', db._adapter.gen_ref_type') where gen_ref_type (or other name) is defined in pydal by the adapter and can be string or integer.
solution 3) introduce a new type 'generic-reference' (as opposed to 'reference something') and use Field('record_id', 'generic-reference')

1 is easier but I think 3 would be best.

[gi0baro]

@mdipierro solution 3 sounds interesting but can also be quite confusing for users.

[stephenrauch]

@mdipierro

My understanding is that pydal currently has:

1. "id" means primary key.
2. "reference " means foreign key to a specific table.
3. "list:reference
" means foreign keys to a specific table.

Proposal for solution 3 is:

• "generic-reference" means foreign key to any table?

Questions:

1. Is my understanding of current behavior, and of solution 3 proposal, correct?
2. In general how does the code which is making use of the "generic-reference" know which table to look for the reference in?
3. I assume that things like ondelete="SET NULL" would not apply?