IPython pretty print issue

[krey]

Hello,

First of all, thank you for the amazing library!

I'm having some trouble displaying some remote objects in IPython/Jupyter Notebooks. (Regular print works)
I've narrowed it down to this example:

import rpyc
from IPython.lib.pretty import pprint
conn = rpyc.classic.connect("hostname")
pprint(conn.modules.my_module.something)

Stack trace:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/krey/.conda/envs/py27/lib/python2.7/site-packages/IPython/lib/pretty.py", line 156, in pprint
    printer.pretty(obj)
  File "/home/krey/.conda/envs/py27/lib/python2.7/site-packages/IPython/lib/pretty.py", line 384, in pretty
    if cls in self.type_pprinters:
  File "/home/krey/.conda/envs/py27/lib/python2.7/site-packages/rpyc/core/netref.py", line 220, in method
    return syncreq(_self, consts.HANDLE_CALLATTR, name, args, kwargs)
  File "/home/krey/.conda/envs/py27/lib/python2.7/site-packages/rpyc/core/netref.py", line 75, in syncreq
    return conn.sync_request(handler, proxy, *args)
  File "/home/krey/.conda/envs/py27/lib/python2.7/site-packages/rpyc/core/protocol.py", line 471, in sync_request
    return self.async_request(handler, *args, timeout=timeout).value
  File "/home/krey/.conda/envs/py27/lib/python2.7/site-packages/rpyc/core/async_.py", line 97, in value
    raise self._obj
TypeError: descriptor '__hash__' of 'object' object needs an argument

========= Remote Traceback (1) =========
Traceback (most recent call last):
  File "c:\krey\site-packages\rpyc\core\protocol.py", line 329, in _dispatch_request
    res = self._HANDLERS[handler](self, *args)
  File "c:\krey\site-packages\rpyc\core\protocol.py", line 603, in _handle_callattr
    return self._handle_call(obj, args, kwargs)
  File "c:\krey\site-packages\rpyc\core\protocol.py", line 590, in _handle_call
    return obj(*args, **dict(kwargs))
TypeError: descriptor '__hash__' of 'object' object needs an argument

I haven't been able to find any references to this error on the internet.

Environment

Server

• rpyc version: 4.0.1
• python version: 2.7.11
• operating system: Windows 7

Client

• rpyc version 4.0.1
• python version: 2.7.15
• operating system: Linux

Minimal example

I don't have access to the server's python module unfortunately

[coldfix]

FYI, this issue was due to ipython using a mechanism similar to:

d = {...}
for cls in obj.__class__.__mro__:
  if cls in d:
    ...

This effectively requires the ability to do hash(conn.modules.numpy.array([0]).__class__) which was previously broken - partially by #268, partially because BaseNetref did not handle the python3 comparison protocol and python3 adds __hash__=None during class construction to the member namespace if an __eq__ method is present.

One point to note: I couldn't reproduce your exact error, in my case your example errored with: TypeError: 'NoneType' object is not callable. I could trigger the TypeError: descriptor '__hash__' of 'object' object needs an argument message only manually by doing hash(c.modules.numpy.array([0]).__class__.__mro__[1]). I would guess this is due to different numpy or ipython version. Both calls now work and pprint gives the expected output.

[sergio-bershadsky]

Client python 3.6, Server python is 3.7

Getting a similar issue in the IPython environment when %%pprint is ON.

import rpyc

c = rpyc.connect("sergey-nuc.local", 9090)
c.root.run(some="thing", than="returns", "dict")

Getting smth like this:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/rpyc/core/protocol.py", line 323, in _dispatch_request
    res = self._HANDLERS[handler](self, *args)
  File "/usr/local/lib/python3.7/site-packages/rpyc/core/protocol.py", line 591, in _handle_inspect
    if hasattr(self._local_objects[id_pack], '____conn__'):
  File "/usr/local/lib/python3.7/site-packages/rpyc/lib/colls.py", line 110, in __getitem__
    return self._dict[key][0]
KeyError: ('rpyc.core.netref.dict_keys', 94860646611336, 140558241863000)```

[talsaiag]

I had a similar issue (I'm writing this here, but in case this is not related, maybe should be moved to a standalone issue).

In my case, IPython checks at some point bool(type(o)) when printing object o.
This initiates rpyc's request to Connection._handle_getattr(type(o), '__len__') which returns <slot wrapper '__len__' of 'dict' objects> which cannot be called with zero parameters.

I ended up adding this workaround until a fix will be introduced:

import types
from rpyc.core.protocol import Connection
# Fix for calling bool(ClassType)
def _handle_getattr(self, obj, name):  # request handler
    out = self._access_attr(obj, name, (), "_rpyc_getattr", "allow_getattr", getattr)
    if name == '__len__' and type(obj) == type and isinstance(out, types.WrapperDescriptorType):
        return lambda: True
    return out
Connection._handle_getattr = _handle_getattr

[comrumino]

Thanks for reporting the issue and tracking the issue here is okay with me---it seems likely that it is in the same class of bugs. I'll take a look

[talsaiag]

Thanks!

I use rpyc to manipulate another application. Our client app is an IPython interpreter.
I have noticed a significant slow down (every local pretty print for a remote object takes around 10 times longer) when moving from rpyc v4.1.2 to v4.1.4 in our application.

python version: python3.7 (using docker image python:3.7.6-stretch)
operating system: linux
both rpyc server and client run inside the same container under the same version.

Was something added that was expected to cause a slowdown?

[comrumino]

Off hand, there isn't anything that I could think of that would really change the performance that much. I did take a quick glance at CHANGELOG.rst and all the changes have been related to bugs.

Could you provide empirical data or a way to reproduce the change in overhead? It would help narrow down which parts of the API are invoked more heavily for your use case. I'd be glad to look into it since improving overhead is one the upcoming mountains to move.

If I were to blindly guess what the culprit could be:

1. overhead from handling __class__
2. overhead from get_id_pack (i.e. resolving name pack such as abc.ABCMeta)
3. compatibility CodeType objects which should only affect teleport

Ironically, there were supposed to be minor performance improvements between those versions. Anyway, I'd greatly appreciate any help in measuring the overhead with timeit.

[talsaiag]

Sure, attached is a standalone reproducible code:

import threading
from IPython.lib.pretty import pprint, RepresentationPrinter

import rpyc
from rpyc.core import SlaveService
from rpyc.utils.classic import DEFAULT_SERVER_PORT
from rpyc.utils.server import ThreadedServer

server = ThreadedServer(SlaveService, port=DEFAULT_SERVER_PORT, reuse_addr=True)
server_thread = threading.Thread(target=server.start, daemon=True)
server_thread.start()

conn = rpyc.classic.connect("localhost")

print(rpyc.version.version)

class DummyFile:
    def write(self, data):
        pass

def mypprint(obj):
    # same as pprint for formatting the object but skipping printing it
    printer = RepresentationPrinter(DummyFile(), False, 79, '\n', max_seq_length=1000)
    printer.pretty(obj)
    printer.flush()

%timeit mypprint(conn.modules.__main__.server)

With: pip3 install rpyc==4.1.4:
200 ms +- 312 us per loop (mean +- std. dev. of 7 runs, 10 loops each)
With pip3 install rpyc==4.1.2:
1.05 ms +- 79.8 us per loop (mean +- std. dev. of 7 runs, 1000 loops each)

[comrumino]

Some commit times to confirm.

8fa210db9984de93b6b270287def4ef012538c9c 20.4 ms
07d70305d321cb5fc33057cc9f4e35d02fe418da 10.9 ms
020f9fd562111688959ffdb2e10013c56ded34b0 11.0 ms
1d6d51c5b281cf386d2a42bc624983cfc9841ec4 00.7 us

After pushing an improvement

pypi - (4, 1, 2)
1.136995326189208 ms ± 46.13621871819378 µs per loop (mean ± std. dev. of 40 runs, 400 loops each)
git - (4, 1, 4)
1.443553417437215 ms ± 50.15403089501748 µs per loop (mean ± std. dev. of 40 runs, 400 loops each)

I'll try to take a look at the actual pretty print issue soon.

[comrumino]

@talsaiag I failed to reproduce the issue using 4.1.4 rpyc and 7.12.0 IPython. Could you provide a minimal example that causes an issue (i.e. # Fix for calling bool(ClassType))?

I will try to reproduce on my end as well. Steps to reproduce allow me to confirm, diagnose, and fix in a more timely way.

[talsaiag]

I haven't had much time to work on this since I managed to simply work around the issue by detecting if the object to be pretty printed is from the remote endpoint, and if so I called the remote endpoint's pretty function (thus only returning a simple string object back to the calling hook)
this does not solve the issue entirely but for now it made the printing of large remote objects go form 12 seconds to 0