Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strictly define inter-JSON separator #1086

Closed
socketpair opened this issue Nov 19, 2019 · 2 comments
Closed

Strictly define inter-JSON separator #1086

socketpair opened this issue Nov 19, 2019 · 2 comments
Labels

Comments

@socketpair
Copy link

socketpair commented Nov 19, 2019

First, I want to use GRPC gateway to access etcd watching functionality. Namely, https://github.com/etcd-io/etcd/blob/master/Documentation/dev-guide/api_grpc_gateway.md#watch-keys .

But I don't have any specifications on how should I separate JSONs in the response stream since it is not documented anywhere.

Basically I have 3 ways:

  1. HTTP chunk = message
    This fails at least in Fedora 31 on high load because sometimes bare newline \n appears as a separate chunk. In order to fix you probably should first concatenate JSON with \n before calling to Write(). Also, intermediate proxies may re-chunk stream.

  2. Messages are separated using strictly one newline.
    It is not documented as well, and comment in newline between JSON messages #931 states that newline is added just to simplify debugging. Somewhere in the net, I found that other software uses a double newline for that purpose (allow to divide messages containing formatted JSON). Currently, I choose this way.

  3. Use JSON finite state machine to detect end of a message.
    This way will work in any case, but technically this requires additional libraries for the majority of programming languages.

So, Please strictly document exact behavior (protocol) which should be considered for implementing JSON stream parser.

See also #931

@heyitsanthony please consider that problem also.

@johanbrandhorst
Copy link
Collaborator

johanbrandhorst commented Nov 19, 2019

Should this not be raised with etcd? The gRPC-gateway has a configurable marshaler, which can implement a Delimited interface to specify what delimiter to use. See https://github.com/grpc-ecosystem/grpc-gateway/blob/master/runtime/marshaler.go#L44.

@stale
Copy link

stale bot commented Jan 18, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Jan 18, 2020
@stale stale bot closed this as completed Jan 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants