encoding/proto: simplify & optimize proto codec #3958
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dfawley
added
the
Type: Performance
|
I like this simplification but it causes error when trying to marshal |
morningsend
reviewed
Feb 5, 2021
| func (codec) Marshal(v interface{}) ([]byte, error) { | ||
| if pm, ok := v.(proto.Marshaler); ok { |
There was a problem hiding this comment.
This is breaking existing code generate with gogoproto.customtype = "MyType", where a user can write custom marshaling/unmarshaling code to have more control. IMHO should not be removed.
https://github.com/gogo/protobuf/blob/master/custom_types.md#custom-type-method-signatures
There was a problem hiding this comment.
gogoproto is not officially supported by gRPC-Go.
If you want a codec that suits your needs better, it's easy to install your own. Just copy/paste this code into another package, update the parts that are important, and import it in your main package after any grpc imports. (If you don't control main, you can give it a new name and use the Codec Call-/Dial-/Server- Options.)
There was a problem hiding this comment.
Ok thanks, I will give it a try. I have also created an issue about this. Please feel free to close it #4192 I will also comment the solution there after testing it.
morningsend
reviewed
Feb 5, 2021
| protoMsg := v.(proto.Message) | ||
| protoMsg.Reset() | ||
|
|
||
| if pu, ok := protoMsg.(proto.Unmarshaler); ok { |
There was a problem hiding this comment.
Same problem, removing this will break use case of gogoproto.customtype = "MyType"
stevendanna
added a commit
to stevendanna/cockroach
that referenced
this pull request
Feb 18, 2021
This change enables future upgrades of google.golang.org/grpc. Recent changes to google.golang.org/grpc have changed the default proto codec such that it is incompatible with gogoproto: grpc/grpc-go#3958 While `proto.Marshal` does do a check for the Marshal/Unmarshal it appears to do this through reflection methods that will panic on fields using gogoproto.nullable. To avoid this, we install a proto codec that does an interface check before dispatching to proto.Marshal/proto.Unmarshal. Note that I haven't replicated the protobuffer caching that previously existed in google.golang.org/grpc. The referenced change claims removing the caching may be a small performance win; however, they also removed the typecheck that we need to do, so it is unclear. Fixes cockroachdb#60531 Release note: None
stevendanna
added a commit
to stevendanna/cockroach
that referenced
this pull request
Mar 10, 2021
This change enables future upgrades of google.golang.org/grpc. Recent changes to google.golang.org/grpc have changed the default proto codec such that it is incompatible with gogoproto: grpc/grpc-go#3958 While `proto.Marshal` does do a check for the Marshal/Unmarshal it appears to do this through reflection methods that will panic on fields using gogoproto.nullable. To avoid this, we install a proto codec that does an interface check before dispatching to proto.Marshal/proto.Unmarshal. Note that I haven't replicated the protobuffer caching that previously existed in google.golang.org/grpc. The referenced change claims removing the caching may be a small performance win; however, they also removed the typecheck that we need to do, so it is unclear. Fixes cockroachdb#60531 Release note: None
stevendanna
added a commit
to stevendanna/cockroach
that referenced
this pull request
Mar 18, 2021
This change enables future upgrades of google.golang.org/grpc. Recent changes to google.golang.org/grpc have changed the default proto codec such that it is incompatible with gogoproto: grpc/grpc-go#3958 While `proto.Marshal` does do a check for the Marshal/Unmarshal it appears to do this through reflection methods that will panic on fields using gogoproto.nullable. To avoid this, we install a proto codec that does an interface check before dispatching to proto.Marshal/proto.Unmarshal. Note that I haven't replicated the protobuffer caching that previously existed in google.golang.org/grpc. The referenced change claims removing the caching may be a small performance win; however, they also removed the typecheck that we need to do, so it is unclear. Fixes cockroachdb#60531 Release note: None
stevendanna
added a commit
to stevendanna/cockroach
that referenced
this pull request
Mar 23, 2021
This change enables future upgrades of google.golang.org/grpc. Recent changes to google.golang.org/grpc have changed the default proto codec such that it is incompatible with gogoproto: grpc/grpc-go#3958 While `proto.Marshal` does do a check for the Marshal/Unmarshal it appears to do this through reflection methods that will panic on fields using gogoproto.nullable. To avoid this, we install a proto codec that does an interface check before dispatching to proto.Marshal/proto.Unmarshal. Note that I haven't replicated the protobuffer caching that previously existed in google.golang.org/grpc. The referenced change claims removing the caching may be a small performance win; however, they also removed the typecheck that we need to do, so it is unclear. Fixes cockroachdb#60531 Release note: None
craig bot
pushed a commit
to cockroachdb/cockroach
that referenced
this pull request
Mar 23, 2021
61333: rpc: install custom proto codec r=[erikgrinaker,pbardea] a=stevendanna This change enables future upgrades of google.golang.org/grpc. Recent changes to google.golang.org/grpc have changed the default proto codec such that it is incompatible with gogoproto: grpc/grpc-go#3958 While `proto.Marshal` does do a check for the Marshal/Unmarshal it appears to do this through reflection methods that will panic on fields using gogoproto.nullable. To avoid this, we install a proto codec that does an interface check before dispatching to proto.Marshal/proto.Unmarshal. Note that I haven't replicated the protobuffer caching that previously existed in google.golang.org/grpc. The referenced change claims removing the caching may be a small performance win; however, they also removed the typecheck that we need to do, so it is unclear. Fixes #60531 Release note: None Co-authored-by: Steven Danna <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adopt recommendation in #3400 to simplify the proto codec; it seems the buffer pool is no longer buying us any performance wins.
Performance #s:
First run:
Second run:
This appears to provide slightly better performance on average, and is considerably simpler, and would provide a lower memory footprint due to the lack of a buffer pool.
Related to #3932