Implement JSON like unmarshaling interface #115

tubocurarin · 2018-04-18T12:46:56Z

Hey folks ;)

I think it would be nice if mapstructure would support an unmarshaling interface like for the json module, so custom marshalers can be defined on a per-type basis easily:

type MyType struct {
    number int
    numberTimesTwo int
}

// For example we call the interface function for mapstructure UnmarshalMap
func (t *MyType) UnmarshalMap(value interface{}) error {
    // Although not a real-world usecase, just multiply our read number by 2.
    intValue, ok := value.(int)

    if !ok {
        return errors.New("Can't convert value to int")
    }

    t.number = value
    t.numberTimesTwo = value * 2

    return nil
}

Specifically I tried to do custom marshaling inside viper, which internally uses mapstructure to unmarshal from its internal map-representation of config-key-values to my config-struct. But there are no possibilities there to "override" marshaling behaviour for my types and I have to write quite some boilerplate to circumvent that^^

scags9876 · 2018-11-02T16:18:45Z

I would also like mapstructure to support gopkg.in/guregu/null.v3 fields, which I believe fits in with this ticket. The null fields there all implement json.Marshaller. (https://github.com/guregu/null/blob/v3.4.0/bool.go#L94)

andig · 2019-01-14T18:04:40Z

Looking at https://github.com/mitchellh/mapstructure/blob/master/mapstructure_test.go#L530 and the DecodeHook I'm wondering if this is not already implementable by supplying a hook that implements the MapUnmarshaler behaviour? ~~Have not tried this though.~~

Update

I have the feeling this cannot be done through the existing hooks:

type MapUnmarshaler interface {
	UnmarshalMap(interface{}) (interface{}, error)
}

decodeHook := func(from reflect.Type, to reflect.Type, v interface{}) (interface{}, error) {
	unmarshalerType := reflect.TypeOf((*MapUnmarshaler)(nil)).Elem()
	if to.Implements(unmarshalerType) {
                // invoke UnmarshalMap by name
		in := []reflect.Value{reflect.New(to).Elem(), reflect.ValueOf(v)}
		r := to.MethodByName("UnmarshalMap").Func.Call(in)
        
		// get first return parameter and cast reflect.Value
		v = r[0].Interface().(interface{})
	}
	return v, nil
}

This works up to the point where v has the correct type and value. However, DecodeHook is only able to perform input pre-processing.

Furthermore v cannot- even if the hook were able to do more- be returned or assigned to the result value as this would already be of the generic type (the one that that performs the actual decoding) and cannot accept specific implementations.

solarfly73 · 2020-01-15T18:33:47Z

If you are in a real pinch and are starting out with a map, you can marshal that back to JSON bytes then get it into your known structure. But if you're ingesting JSON, which has a field mapped to arbitrary JSON, you can specify that as a json.RawMessage instead of a map[string]interface{}. This allows custom UnmarshalJSON functions to be triggered.

type StringOrBool bool
func (*sb StringOrBool) UnmarshalJSON(b []byte) error {
   ...
}

var testMessage struct {
            Custom StringOrBool `json:"custom"`
}
raw := json.RawMessage(`{"custom":false}`)
err := json.Unmarshal(raw, &testMessage)
...

mitchellh · 2020-11-26T22:40:47Z

I like this suggestion, there is also #204 which overlaps with this by reusing the text unmarshaler. I see a fit for both. If a PR were to open up adding this I would work on merging it, but it hasn't yet!

Note I also merged #183 which should enable this sort of functionality more manually.

bored-engineer · 2020-11-28T20:35:32Z

For reference, here's an example of a custom unmarshaler interface via the new DecodeHookFuncValue interface. (I'm sure there is a cleaner way to do the structure creation logic than what I did):

type Unmarshaler interface {
	CustomUnmarshalMethod(interface{}) error
}

func UnmarshalerHook() mapstructure.DecodeHookFunc {
	return func(from reflect.Value, to reflect.Value) (interface{}, error) {
		// If the destination implements the unmarshaling interface
		u, ok := to.Interface().(Unmarshaler)
		if !ok {
			return from.Interface(), nil
		}
		// If it is nil and a pointer, create and assign the target value first
		if to.IsNil() && to.Type().Kind() == reflect.Ptr {
			to.Set(reflect.New(to.Type().Elem()))
			u = to.Interface().(Unmarshaler)
		}
		// Call the custom unmarshaling method
		if err := u.CustomUnmarshalMethod(from.Interface()); err != nil {
			return to.Interface(), err
		}
		return to.Interface(), nil
	}
}

ghostiam · 2021-02-08T19:15:04Z

Or use stdlib TextUnmarshaler interface

func UnmarshalerHook() mapstructure.DecodeHookFunc {
	return func(from reflect.Value, to reflect.Value) (interface{}, error) {
		if to.CanAddr() {
			to = to.Addr()
		}

		// If the destination implements the unmarshaling interface
		u, ok := to.Interface().(encoding.TextUnmarshaler)
		if !ok {
			return from.Interface(), nil
		}

		// If it is nil and a pointer, create and assign the target value first
		if to.IsNil() && to.Type().Kind() == reflect.Ptr {
			to.Set(reflect.New(to.Type().Elem()))
			u = to.Interface().(encoding.TextUnmarshaler)
		}

		var text []byte
		switch v := from.Interface().(type) {
		case string:
			text = []byte(v)
		case []byte:
			text = v
		default:
			return v, nil
		}

		if err := u.UnmarshalText(text); err != nil {
			return to.Interface(), err
		}
		return to.Interface(), nil
	}
}

I have tripped over mapstructure one too many times so this commit removes it. When/if the extra round-trip through JSON adds measurable overhead to Zed query times, we can revisit this and hopefully mapstructure will be more mature. For now, I wanted to add order.Which to proc.Sort and similar but mapstructure cannot handle custom JSON unmarshalers so this caused unpack to fail for apparently no reason. See mitchellh/mapstructure#115

@mccanne

This is an auto-generated commit with a Zed dependency update. The Zed PR brimdata/super#2703, authored by @mccanne, has been merged. remove mapstructure I have tripped over mapstructure one too many times so this commit removes it. When/if the extra round-trip through JSON adds measurable overhead to Zed query times, we can revisit this and hopefully mapstructure will be more mature. For now, I wanted to add order.Which to proc.Sort and similar but mapstructure cannot handle custom JSON unmarshalers so this caused unpack to fail for apparently no reason. See mitchellh/mapstructure#115

@mccanne

This is an auto-generated commit with a Zed dependency update. The Zed PR brimdata/super#2703, authored by @mccanne, has been merged. remove mapstructure I have tripped over mapstructure one too many times so this commit removes it. When/if the extra round-trip through JSON adds measurable overhead to Zed query times, we can revisit this and hopefully mapstructure will be more mature. For now, I wanted to add order.Which to proc.Sort and similar but mapstructure cannot handle custom JSON unmarshalers so this caused unpack to fail for apparently no reason. See mitchellh/mapstructure#115

mthjs · 2022-06-17T12:21:27Z

Is anyone still interested in this?

tubocurarin mentioned this issue Apr 18, 2018

viper.Unmarshal() ignoring UnmarshalYAML interface spf13/viper#338

Open

camdencheek mentioned this issue Mar 28, 2020

Add a value hook func #183

Merged

smyrman mentioned this issue May 15, 2020

Allow override decode semantics through decode middleware #192

Closed

mccanne mentioned this issue May 11, 2021

remove mapstructure brimdata/super#2703

Merged

mthjs mentioned this issue May 20, 2022

Allow types to determine how to decode themselves #287

Closed

mthjs linked a pull request Jun 27, 2022 that will close this issue

Allow types to determine how to decode themselves #294

Open

knadh mentioned this issue Dec 20, 2022

Unable to decode hcl file data knadh/koanf#184

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement JSON like unmarshaling interface #115

Implement JSON like unmarshaling interface #115

tubocurarin commented Apr 18, 2018

scags9876 commented Nov 2, 2018

andig commented Jan 14, 2019 •

edited

Loading

solarfly73 commented Jan 15, 2020 •

edited

Loading

mitchellh commented Nov 26, 2020

bored-engineer commented Nov 28, 2020

ghostiam commented Feb 8, 2021

mthjs commented Jun 17, 2022

Implement JSON like unmarshaling interface #115

Implement JSON like unmarshaling interface #115

Comments

tubocurarin commented Apr 18, 2018

scags9876 commented Nov 2, 2018

andig commented Jan 14, 2019 • edited Loading

solarfly73 commented Jan 15, 2020 • edited Loading

mitchellh commented Nov 26, 2020

bored-engineer commented Nov 28, 2020

ghostiam commented Feb 8, 2021

mthjs commented Jun 17, 2022

andig commented Jan 14, 2019 •

edited

Loading

solarfly73 commented Jan 15, 2020 •

edited

Loading