jsonpb: change Marshal/Unmarshal to return error if any required field is not set #472
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -257,12 +257,17 @@ func (m *Marshaler) marshalObject(out *errWriter, v proto.Message, indent, typeU | |
| continue | ||
| } | ||
|
|
||
| prop := jsonProperties(valueField, m.OrigName) | ||
| // IsNil will panic on most value kinds. | ||
| switch value.Kind() { | ||
| case reflect.Chan, reflect.Func, reflect.Interface: | ||
| if value.IsNil() { | ||
| continue | ||
| } | ||
| case reflect.Ptr: | ||
| if prop.Required && value.IsNil() { | ||
| return fmt.Errorf("required field %q is not set", prop.Name) | ||
| } | ||
| } | ||
|
|
||
| if !m.EmitDefaults { | ||
|
|
@@ -300,8 +305,8 @@ func (m *Marshaler) marshalObject(out *errWriter, v proto.Message, indent, typeU | |
| sv := value.Elem().Elem() // interface -> *T -> T | ||
| value = sv.Field(0) | ||
| valueField = sv.Type().Field(0) | ||
| prop = jsonProperties(valueField, m.OrigName) | ||
| } | ||
| if !firstField { | ||
| m.writeSep(out) | ||
| } | ||
|
|
@@ -644,7 +649,105 @@ func (u *Unmarshaler) UnmarshalNext(dec *json.Decoder, pb proto.Message) error { | |
| // permutations of the related Marshaler. | ||
| func (u *Unmarshaler) Unmarshal(r io.Reader, pb proto.Message) error { | ||
| dec := json.NewDecoder(r) | ||
| if err := u.UnmarshalNext(dec, pb); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| return checkRequiredFields(pb) | ||
|
There was a problem hiding this comment. Would it be possible to work the code into UnmarshalNext, rather than iterate over all the fields a second time? (albeit, yes, fairly fast code.) There was a problem hiding this comment. Possibly. The current unmarshaling logic does rely on json.Unmarshal to populate fields. To embed the check for required in there means that it will still need to check for all fields that have been unmarshaled. But you are correct that doing the check w/in the unmarshaling code may be more efficient. The unmarshaling code is quite complicated though that I prefer to avoid mixing this logic in right now. A refactor/rewrite of the unmarshaling code should account for this logic. There was a problem hiding this comment. The There was a problem hiding this comment. Even though There was a problem hiding this comment. Oops, thanks for catching. Fixed. |
||
| } | ||
|
|
||
| // checkRequiredFields returns an error if any required field in the given proto message is not set. | ||
| // This function is called from Unmarshal and assumes certain actions are done ahead. | ||
| func checkRequiredFields(pb proto.Message) error { | ||
| // Most well-known type messages do not contain any required field. The "Any" type may have | ||
| // required fields, however the Unmarshaler should have invoked proto.Marshal on the value field | ||
|
There was a problem hiding this comment. Do you mean proto.Unmarshal here? There was a problem hiding this comment. Nope. During unmarshaling of an "Any" JSON to a Go proto message, after it resolves and constructs the embedded message, it will actual do a proto marshal to a serialized form and store the bytes into Any.Value field. See https://github.com/golang/protobuf/blob/dev/jsonpb/jsonpb.go#L751. I've updated the comment to explain this better. |
||
| // during unmarshaling and that would have returned an error if a required field is not set. | ||
| if _, ok := pb.(wkt); ok { | ||
| return nil | ||
| } | ||
|
|
||
| v := reflect.ValueOf(pb).Elem() | ||
|
There was a problem hiding this comment. You will need to verify that pb is a pointer to a struct. If it isn't, then return. There was a problem hiding this comment. Done. I'm guessing we'll simply ignore message if not pointer to struct. |
||
| for i := 0; i < v.NumField(); i++ { | ||
| field := v.Field(i) | ||
| sfield := v.Type().Field(i) | ||
| if strings.HasPrefix(sfield.Name, "XXX_") { | ||
| continue | ||
| } | ||
|
|
||
| // Oneof fields need special handling. | ||
| if sfield.Tag.Get("protobuf_oneof") != "" { | ||
| // field is an interface containing &T{real_value}. | ||
| v := field.Elem().Elem() // interface -> *T -> T | ||
|
There was a problem hiding this comment. Have you considered the case where a oneof contains a simple scalar (string, bytes, varint), which in proto3 is not a pointer? What will this code do in that case? There was a problem hiding this comment. Current proto compiler will always produce an interface field for a oneof field. It will also produce wrapper structs, one for each oneof field. And hence, this block will grab the oneof field inside the wrapper struct which can be any type. There was a problem hiding this comment. Need to check:
|
||
| field = v.Field(0) | ||
| sfield = v.Type().Field(0) | ||
| } | ||
|
|
||
| var prop proto.Properties | ||
| prop.Init(sfield.Type, sfield.Name, sfield.Tag.Get("protobuf"), &sfield) | ||
|
|
||
| // IsNil will panic on most value kinds. | ||
| switch field.Kind() { | ||
| case reflect.Map: | ||
|
There was a problem hiding this comment. I think we discussed this before, and you told me how a proto3 could contain a proto2 message, but maps are exclusive to proto3, so do they really need to be checked? Consider adding some code comments explaining the rational of checking maps despite their appearance being expected to be mutually exclusive with required fields. There was a problem hiding this comment. No, maps are allowed in proto2. I have a test case for this. |
||
| if field.IsNil() { | ||
| continue | ||
| } | ||
| // Check each map value. | ||
| keys := field.MapKeys() | ||
| for _, k := range keys { | ||
| v := field.MapIndex(k) | ||
| if err := checkRequiredFieldsInValue(v); err != nil { | ||
| return err | ||
| } | ||
| } | ||
| case reflect.Slice: | ||
|
There was a problem hiding this comment. The "bytes" scalar for proto will fall under this case and will need to be specially handled. There was a problem hiding this comment. Good catch, I forgot about that. Added handling of non-repeated slice. Yes, I'm going with []byte{} to mean it is set. |
||
| if field.IsNil() { | ||
| continue | ||
| } | ||
| // Check each slice item. | ||
| for i := 0; i < field.Len(); i++ { | ||
| v := field.Index(i) | ||
| if err := checkRequiredFieldsInValue(v); err != nil { | ||
| return err | ||
| } | ||
| } | ||
| case reflect.Ptr: | ||
| if field.IsNil() { | ||
| if prop.Required { | ||
| return fmt.Errorf("required field %q is not set", prop.Name) | ||
| } | ||
| continue | ||
| } | ||
| if err := checkRequiredFieldsInValue(field); err != nil { | ||
| return err | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // Handle proto2 extensions. | ||
| for _, ext := range proto.RegisteredExtensions(pb) { | ||
| if !proto.HasExtension(pb, ext) { | ||
| continue | ||
| } | ||
| ep, err := proto.GetExtension(pb, ext) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| err = checkRequiredFieldsInValue(reflect.ValueOf(ep)) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| } | ||
|
|
||
| return nil | ||
| } | ||
|
|
||
| func checkRequiredFieldsInValue(v reflect.Value) error { | ||
| if pm, ok := v.Interface().(proto.Message); ok { | ||
| if err := checkRequiredFields(pm); err != nil { | ||
|
There was a problem hiding this comment. If this isn’t in a loop, then it’s not necessary to check the error value before returning it. i.e. you can just use There was a problem hiding this comment. Good catch. I previously have this block inlined above multiple times and then pulled it out to this helper func but forgot that this added check is no longer required. Fixed. |
||
| return err | ||
| } | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| // UnmarshalNext unmarshals the next protocol buffer from a JSON object stream. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Curious, for error messages like this, should I use Name or OrigName? Or is it dependent on whether message is for marshaling or unmarshaling?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Properties.Nameis fine. It's even documented as such: "name of the field, for error messages"