[C++17] Allow std::optional to convert to nlohmann::json

[Treeston]

This is what I assume to be a fairly minor change to the library's logic, but would represent a large improvement in usability.

Feature request

C++17 adds std::optional<T>. Let std::optional<T> convert to nlohmann::json.

If std::optional<T> is empty, I would expect this to result in null - if std::optional<T>, I would expect it to result in the contained value converted.

Use case example

The change would simplify (especially nested) uses of std::optional.

For example, imagine a simple key-value map, where each integer key can optionally have a localized string representation for each of 8 locales. The client dynamically selects the "best" available locale depending on user preferences.

In server code, one could express this as a std::vector<std::array<std::optional<std::string>, 8>>.
Currently, this type does not convert to nlohmann::json because std::optional<std::string> does not - resulting in needing to allocate a bunch of temporaries so the inner conversions can be done "manually".

Allowing std::optional to convert to nlohmann::json would simplify this logic greatly.

[jaredgrubb]

It's a neat idea, though this library is still intended as a C++11 library. I wonder at what point that could change to allow things like this?

[nlohmann]

The library defines a macro JSON_HAS_CPP_17 to detect C++17. With that, the feature could be realized. The required code should look similar to #760.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jnhyatt]

In addition, I believe the library would benefit from the ability to return a std::optional from a getter method -- this way, client programs could avoid having to catch exceptions to detect a missing value, etc.:

std::string name = j["name"].get_optional<std::string>().value_or("default-name");

Qt uses something similar in their config file interface. This makes it very easy to save a default json file on a load failure -- you deserialize a class from json with default values using std::optional and then serialize the resulting class back to json; if there were no changes to the class, the json should be identical, otherwise missing data will be added to the output json.

[nlohmann]

Wouldn't this be similar to j.value("name", "default-name")?

[jnhyatt]

Yes, it would be very similar in most cases. I admit I hadn't searched the library exhaustively for such a feature before having made this suggestion.

[djcb]

It's often useful to differentiate about not getting the value, and getting some default value. Also, perhaps the item is there but it's not of the expected type.

I'm dealing with such cases all the time, and using something like:

template<typename T>
std::optional<T> get_at_optional(const nlohmann::json& obj, const std::string& key) try {
        return obj.at(key).get<T>();
} catch (...) {
        return std::nullopt;
}

Without it, I need to add type-checks and/or try/catch blocks everywhere, which isn't very nice.

[nlohmann]

I started working on this in a feature branch https://github.com/nlohmann/json/tree/feature/optional. Unfortunately, I currently cannot make the code compile with MSVC: https://ci.appveyor.com/project/nlohmann/json/builds/29069989

Any ideas?

[remedi]

I looked at AppVeyor and noticed that the builds fail only when the latest language standard is being used in Visual Studio. Essentially the builds fail when std::optional should be available.

I checked out your branch, tested on local Visual Studio 2017 and found out the issue. The preprocessor definition JSON_HAS_CPP_17 is being used too early before the language standard detection is done. The "error C2039: 'optional': is not a member of 'std'" is caused by the later code in compilation using std::optional while the #include statement is skipped because JSON_HAS_CPP_17 has not been defined yet.

However, now there is another error in unit-conversions.cpp lines 1624 and 1633 where there is no suitable conversion from nlohmann::json to std::vector containing std::optional (1624) or std::map containing std::optional (1633). I am not familiar with the code base so I am unsure where exactly the conversion should happen.

[jmoore91]

Looking through this I do not think it is always good to treat an optional as null, there may be times when you just wish to exclude it all together from the json. Is it possible with the current to_json/from_json methods to allow the behaviour to be configurable as to how we wish to treat optional values?

[nlohmann]

No, it is not possible to pass additional parameters.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[nlohmann]

@jmoore91 It is possible to override the default behavior by defining a adl_serializer in your code. So we can add a default conversion from/to std::optional while still allowing different conversions.

[nlohmann]

Unfortunately, I cannot make #2117 get to work with some MSVC configuration, and I have no clue why...

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[YarikTH]

I want to use optional values using std::optional. Desired behavior is to have std::nullopt if the corresponding field is absent and vice versa don't emit field at all if the value is std::nullopt.

To support optional values I wrote some ugly code:

https://godbolt.org/z/zxj97Y

template<class J, class T>
void optional_to_json(J& j, const char* name, const std::optional<T>& value)
{
    if (value)
    {
        j[name] = *value;
    }
}

template<class J, class T>
void optional_from_json(const J& j, const char* name, std::optional<T>& value)
{
    const auto it = j.find(name);
    if (it != j.end())
    {
        value = it->template get<T>();
    }
    else
    {
        value = std::nullopt;
    }
}

struct MyStruct
{
    std::string name;
    std::optional<bool> optBool;
};

template<class T>
inline void to_json(T& j, const MyStruct& config)
{
    j = T
    {
        {"name", config.name},
        //{"optBool", config.optBool},
    };
    //j["optBool"] = config.optBool;
    optional_to_json(j, "optBool", config.optBool);
}

template<class T>
inline void from_json(const T& j, MyStruct& config)
{
    j.at("name").get_to(config.name);
    //j.at("optBool").get_to(config.optBool);
    optional_from_json(j, "optBool", config.optBool);
}

Is there a better variant that allows me to work with optional values exactly the same way as with other values?

[ignus2]

@YarikTH This is what I use (put it right after including nlohmann json):

namespace nlohmann {

	template <class T>
	void to_json(nlohmann::json& j, const std::optional<T>& v)
	{
		if (v.has_value())
			j = *v;
		else
			j = nullptr;
	}

	template <class T>
	void from_json(const nlohmann::json& j, std::optional<T>& v)
	{
		if (j.is_null())
			v = std::nullopt;
		else
			v = j.get<T>();
	}
	
} // namespace nlohmann

Not "nice", but works just fine. You probably have to disable implicit json -> value conversions as well (which shouldn't be used anyway). Use "#define JSON_USE_IMPLICIT_CONVERSIONS 0" or "-DJSON_ImplicitConversions=OFF" (CMake).

[narnaud]

My take on this: https://www.kdab.com/jsonify-with-nlohmann-json/

Like previously, an optional_ from_json/optional_to_json like explain here: #1749 (comment)

Now, with a bit of glue on top of that:

template <typename>
constexpr bool is_optional = false;
template <typename T>
constexpr bool is_optional<std::optional<T>> = true;
  
template <typename T>
void extended_to_json(const char *key, nlohmann::json &j, const T &value) {
    if constexpr (is_optional<T>)
        optional_to_json(j, key, value);
    else
        j[key] = value;
}
template <typename T>
void extended_from_json(const char *key, const nlohmann::json &j, T &value) {
    if constexpr (is_optional<T>)
        optional_from_json(j, key, value);
    else
        j.at(key).get_to(value);
}

#define EXTEND_JSON_TO(v1) extended_to_json(#v1, nlohmann_json_j, nlohmann_json_t.v1);
#define EXTEND_JSON_FROM(v1) extended_from_json(#v1, nlohmann_json_j, nlohmann_json_t.v1);
  
#define NLOHMANN_JSONIFY_ALL_THINGS(Type, ...)                                          \
  inline void to_json(nlohmann::json &nlohmann_json_j, const Type &nlohmann_json_t) {   \ 
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_TO, __VA_ARGS__))            \
  }                                                                                     \
  inline void from_json(const nlohmann::json &nlohmann_json_j, Type &nlohmann_json_t) { \
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_FROM, __VA_ARGS__))          \
  }

I can then write:

struct Foo {
    std::optional<bool> data;
};
NLOHMANN_JSONIFY_ALL_THINGS(Foo, data)

Please note that the data is not set in the json for the optional to be null.

[Ericson2314]

It would be good to reopen this.

[Ericson2314]

I am confused why the stalled PR is so complex, isn't

template<typename T>
struct adl_serializer<std::optional<T>> {
    static std::optional<T> from_json(const json & json) {
        return json.is_null()
            ? std::nullopt
            : std::optional { adl_serializer<T>::from_json(json) };
    }
    static void to_json(json & json, std::optional<T> t) {
        if (t)
            adl_serializer<T>::to_json(json, *t);
        else
            json = nullptr;
    }
};

all that is needed, if we are ignoring all the "missing fields OK" funny business, and just making it be value or null?

Yes, this is incorrect if T itself can be null, but I didn't see that comes up yet, and we could at just a little marker thing to indicate a claim of non-nullness + a static_assert

[creatio-ua]

I am confused why the stalled PR is so complex, isn't

Agree.

P.S. For some reason, your solution didn't compile for me (apple-clang v.13), so I ended up with this:

template<typename T>
struct adl_serializer<std::optional<T>> {
    static void from_json(const json & j, std::optional<T>& opt) {
        if(j.is_null()) {
            opt = std::nullopt;
        } else {
            opt = j.get<T>();
        }
    }
    static void to_json(json & json, std::optional<T> t) {
        if (t) {
            json = *t;
        } else {
            json = nullptr;
        }
    }
};

[patrikhuber]

The above code works, but it would indeed be nice if it also worked if the element was missing from the json file, and not present but null.

[tlaemmlein]

@patrikhuber
I found a solution.

This code works if the json has no entry:

// clang-format off
#define NLOHMANN_JSON_FROM_AND_CONTAINS(v1) \
  if (nlohmann_json_j.contains(#v1)) \
     nlohmann_json_j.at(#v1).get_to(nlohmann_json_t.v1);


#define NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE_AND_CONTAINS(Type, ...)  \
    inline void to_json(nlohmann::json& nlohmann_json_j, const Type& nlohmann_json_t) { NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(NLOHMANN_JSON_TO, __VA_ARGS__)) } \
    inline void from_json(const nlohmann::json& nlohmann_json_j, Type& nlohmann_json_t) { NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(NLOHMANN_JSON_FROM_AND_CONTAINS, __VA_ARGS__)) }
// clang-format on

and to dump empty json you can use:
#995 (comment)
I added a little helper function

// Dump without null values
std::string Dump(nlohmann::json& jsonObject)
{
  RecurseAndFilterNullValues(jsonObject);
  return jsonObject.dump();
}

[patrikhuber]

@tlaemmlein Thank you for posting that!

[dav1d-wright]

@tlaemmlein thank you indeed! I modified this slightly in our code base, such that it differentiates between optional and non-optional members. For non-optional members the contains check is not performed, as we expect those to be present:

template <typename T>
struct is_std_optional : std::false_type
{
};

template <typename T>
struct is_std_optional<std::optional<T>> : std::true_type
{
};

template <typename T>
constexpr bool is_std_optional_v = is_std_optional<T>::value;

// clang-format off

// Helper macro to support reading from json objects where some members are optional
#define NLOHMANN_JSON_FROM_WITH_CONTAINS(v1) \
    if constexpr (is_std_optional_v<decltype(nlohmann_json_t.v1)>) { \
        if (nlohmann_json_j.contains(#v1)) \
            nlohmann_json_j.at(#v1).get_to(nlohmann_json_t.v1); \
    } else { \
        nlohmann_json_j.at(#v1).get_to(nlohmann_json_t.v1); \
    }

// clang-format on

[vid512]

Thank you! Will this change be incorporated into nlohmann-json? As a new user, this is the behavior I expected from NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE (after defining to/from_json for optional, as described in the readme).

[vid512]

I also would change to_json, to not write optional values if they are nullopt.

template<typename T>
void nlohmann_dump_val(nlohmann::json& j, std::string_view name, const std::optional<T>& val) {
    if (val.has_value()) {
        j[name] = val.value();
    }
}

template<typename T>
void nlohmann_dump_val(nlohmann::json& j, std::string_view name, T& val) {
    j[name] = val;
}

#define NLOHMANN_JSON_TO_WITH_CONTAINS(v1) \
    nlohmann_dump_val(nlohmann_json_j, #v1, nlohmann_json_t.v1);

(My template-fu is very weak, so I had to use overloading to implement the different logic for std::optionals. Not sure if this is always correct. Worked for me so far.)

Example of concrete use case where this behavior is useful: Many optional values in https://microsoft.github.io/debug-adapter-protocol//specification.html

[Xtreme-G]

template<typename T>
void nlohmann_dump_val(nlohmann::json& j, std::string_view name, const std::optional<T>& val) {
    if (val.has_value()) {
        j[name] = val.value();
    }
}

template<typename T>
void nlohmann_dump_val(nlohmann::json& j, std::string_view name, T& val) {
    j[name] = val;
}

#define NLOHMANN_JSON_TO_WITH_CONTAINS(v1) \
    nlohmann_dump_val(nlohmann_json_j, #v1, nlohmann_json_t.v1);

I would like to enable this behavior as well, could you please clarify how and where you added this?

[Ericson2314]

In https://github.com/NixOS/nix/blob/3180c09723c3b61d682732c308c74e2087ac628f/src/libutil/json-utils.hh#L43-L124 I made a little trait class to avoid issues where null being used in the underlying serialization would break round-trip parsing. Do we have anything like that in the merged PR? Should we have anything like that?

[nlohmann]

In NixOS/nix@3180c09/src/libutil/json-utils.hh#L43-L124 I made a little trait class to avoid issues where null being used in the underlying serialization would break round-trip parsing. Do we have anything like that in the merged PR? Should we have anything like that?

I think so - especially before the current develop version is actually released and a later change would be a breaking change.