Unicode-only strings in IronPython make rpyc.utils.classic.deliver() fail if the server is Python 2

[pilcru]

When connecting an IronPython (2.7) client to a server running Python 2, rpyc.utils.classic.deliver() fails:

On the server:

&> python -V && python rpyc_classic.py
Python 2.7.13 :: Continuum Analytics, Inc.
INFO:SLAVE/18812:server started on [127.0.0.1]:18812
INFO:SLAVE/18812:accepted ('127.0.0.1', 52319) with fd 868
INFO:SLAVE/18812:welcome ('127.0.0.1', 52319)
DEBUG:SLAVE/18812:Exception caught
Traceback (most recent call last):
  File "c:\tools\Anaconda3\envs\rhinoremote\lib\site-packages\rpyc\core\protocol.py", line 347, in _dispatch_request
    res = self._HANDLERS[handler](self, *args)
  File "c:\tools\Anaconda3\envs\rhinoremote\lib\site-packages\rpyc\core\protocol.py", line 624, in _handle_call
    return self._local_objects[oid](*args, **dict(kwargs))
TypeError: loads() argument 1 must be string, not unicode

On the client:

IronPython 2.7.5 (2.7.5.0) on .NET 4.0.30319.42000 (64-bit)
Type "help", "copyright", "credits" or "license" for more information.
>>> import rpyc
>>> rpyc.__version__
(3, 4, 4)
>>> conn = rpyc.classic.connect('localhost')
>>> rpyc.utils.classic.deliver(conn, 'a')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files (x86)\IronPython 2.7\Lib\rpyc\utils\classic.py", line 289, in deliver
  File "C:\Program Files (x86)\IronPython 2.7\Lib\rpyc\core\netref.py", line 199, in __call__
  File "C:\Program Files (x86)\IronPython 2.7\Lib\rpyc\core\netref.py", line 72, in syncreq
  File "C:\Program Files (x86)\IronPython 2.7\Lib\rpyc\core\protocol.py", line 523, in sync_request
TypeError: loads() argument 1 must be string, not unicode

========= Remote Traceback (1) =========
...

I have identified the root of the problem to be in the _dump_registry in brine.py. Because IronPython does not distinguish between regular strings and unicode strings (ie. unicode is str is True), the str entry in the registry points to _dump_unicode, so thats how all strings are dumped. In turns, this makes the server decode as unicode all strings sent by value through brine, which in Python 2 cannot be passed to pickle.loads().

One way to solve this is to force plain ascii strings to be sent as regular strings by brine by changing _dump_unicode to:

    import sys
    if sys.platform == 'cli':
        @register(_dump_registry, unicode)
        def _dump_unicode(obj, stream):
            try:
                _dump_str(obj.decode('ascii'), stream)
            except UnicodeDecodeError:
                stream.append(TAG_UNICODE)
                _dump_str(obj.encode("utf8"), stream)
    else:
        @register(_dump_registry, unicode)
        def _dump_unicode(obj, stream):
            stream.append(TAG_UNICODE)
            _dump_str(obj.encode("utf8"), stream)

But this might impact performance, and it might be better to send a flag to the server for when the string should not be considered unicode. However, the question is then to define what should be unicode or not... Another way would be to force the server to convert to str before unpickling, but I haven't found a way of doing specifically that.

I know that IronPython 2 is an almost-dead project that you are not officially supporting, so I'm fine if you do not want to include this in the main branch; I just thought I'd let you know. This might have also been what was causing part of #10.

[coldfix]

Hi, thanks for reporting.

Because IronPython does not distinguish between regular strings and unicode strings (ie. unicode is str is True)

For clarification, so in Ironpython we have unicode = str as on py3, but it does still distunguish bytes from unicode so unicode = str != bytes, right?

    if sys.platform == 'cli':

IronPython shows as cli?

        def _dump_unicode(obj, stream):
            try:
                _dump_str(obj.decode('ascii'), stream)
            except UnicodeDecodeError:
                stream.append(TAG_UNICODE)
                _dump_str(obj.encode("utf8"), stream)

I think you mean obj.encode in both cases, right?

The problem with this is that it opens the door for a lot more unicode/str related problems as were common on py2 whenever mixing unicode with bytes objects (i.e. a certain variable in a certain function can be either unicode or bytes) and that manifest as exceptions only once the program is used with true unicode data but may go unnoticed a long time during testing.

To be honest, given that IronPython handles str = unicode, I don't think a reliable patch to address communication between IronPython/CPython is possible.

(Otherwise, I would be willing to merge a working solution, if it exists and is simple enough)

[pilcru]

Hi, and sorry for not replying sooner.

I agree with all your comments here, and the solution in b02e7e7. I had actually implemented something very similar before I saw it, starting from your comment that unicode != bytes on IronPython.

There is just one thing to add to solve the original problem: rpyc.utils.classic.deliver needs to cast to bytes before sending through brine (IronPython's pickle.dumps returns str, not bytes). Changing it to

deliver(conn, localobj):
    return conn.modules["rpyc.lib.compat"].pickle.loads(bytes(pickle.dumps(localobj)))

works for my case (IronPython to Python 2). It should also be fine in regular Python since pickle.dumps already returns bytes (Python 3) or str == bytes (Python 2), but I haven't tested it.

[coldfix]

No problem. I have added your suggestion. Does this solve your issue?

Best, Thomas

[pilcru]

Yes, thank you!

[pilcru]

Sorry to reopen this, it seems I forgot one of the problematic pickle.dumps. This one in protocol.py also needs to be cast to bytes before returning, or trying to load a copy of an IronPython netref into CPython will fail.

[coldfix]

Hi, thanks for the info. I hope this does what you had in mind.