Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify Protobuf format #7

Open
grodtron opened this issue Jun 7, 2016 · 0 comments
Open

Simplify Protobuf format #7

grodtron opened this issue Jun 7, 2016 · 0 comments

Comments

@grodtron
Copy link
Contributor

grodtron commented Jun 7, 2016

Currently protobuf is wrapping all messages in the following global type, and holding the actual message as a serialized string, forcing a second manual serialization and deserializtion of the message itself.

message DebugMsg {

  enum Type {
    Run = 1;
    Next = 2;
    ...
  }

  required Type type = 1;
  required bytes message = 2; // protobuf serialized message
}

This (I believe) was done as a work around for the fact that the nanomsg python library attempts to decode data to be sent as ascii, and crashes on non-ascii characters (e.g. '\xff'). Some testing has shown that if we instead pass the serialized protobuf message as a list of individual characters, it will not attempt to do an ascii decode and will send the raw data correctly:

# In terminal 1
>>> import nnpy
>>> sock = nnpy.Socket(nnpy.AF_SP, nnpy.REQ)
>>> sock.connect("ipc:///tmp/nn_testing")
>>> s = '\xff\xaa\x00 This is a \xBB\xCC test'
>>> sock.send(s)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/nnpy/socket.py", line 53, in send
    data = data.encode() if isinstance(data, str) else data
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)
>>> sock.send([c for c in s])
>>> 
# In second terminal
>>> import nnpy
>>> sock = nnpy.Socket(nnpy.AF_SP, nnpy.REP)
>>> sock.bind("ipc:///tmp/nn_testing")
>>> sock.recv()
'\xff\xaa\x00 This is a \xbb\xcc test'
>>> 

This workaround I believe was also forcing other odd limitations such as using strings to represent boolean or integer fields.

A good amount of the debugger code could be simplified based on this, and would definitely be an improvement in the overall quality of the project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant