Custom builtin functions

[aplsek]

Is there a way how I can call golang functions from jsonnet?

Now that there is a go port of jsonnet and for example ksonnet is adding custom native functions (see here ksonnet/ksonnet#521) are there any plans on how to support this?

I have many packages written in golang (with unit-testing, etc) and now it seems like I will need to rewrite some of them into jsonnet.

[sbarzowski]

It is possible and quite easy. These are called "native functions" in jsonnet.

Take a look at VM in vm.go:
nativeFuncs map[string]*NativeFunction

The following code explains what a native function is:

// NativeFunction represents a function implemented in Go.
type NativeFunction struct {
	Func   func([]interface{}) (interface{}, error)
	Params ast.Identifiers
	Name   string
}

The interface{} in this case is the canonical Go representation of json as described here.

So for example (code taken from our tests):

var jsonToString = &NativeFunction{
	Name:   "jsonToString",
	Params: ast.Identifiers{"x"},
	Func: func(x []interface{}) (interface{}, error) {
		bytes, err := json.Marshal(x[0])
		if err != nil {
			return nil, err
		}
		return string(bytes), nil
	},
}
vm.NativeFunction(jsonToString)

Then the native function can be accessed using std.native. It's a bit awkward, so it's usually wrapped in "real" jsonnet function. Example from DeepMind's Kapitan: https://github.com/deepmind/kapitan/blob/master/kapitan/lib/kapitan.libjsonnet

There are some gotchas:

• You need to make sure that your native functions are pure and hermetic or bad things will happen. They should be functions-as-in-math basically, no side effects and the result should only depend on the arguments (like a jsonnet function).
• Native functions cannot use laziness or call any jsonnet functions - they operate on plain JSON and return plain JSON
• The API should not be considered stable. We want to clean it up a bit.
• My personal opinion is that non-standard extensions should be minimized and it's preferable to use jsonnet as a command and not as a library. But of course we support both ways.

I hope it helps. And btw if you think any of your functions would be useful to a wider crowd, please let us know, we are interested in extending the stdlib.

[aplsek]

Nice, thank you for the explanation. I noticed this in the ksonnet PR.

But to make this work you have to effectively recompile jsonnet or wrap jsonnet VM in your own code?
There is no e.g. "drop-in" lib directory where I could put my native function implementation and the standard jsonnet would pick it up (with some manual jsonnet "reinstall/recompile" native functions command).

[sbarzowski]

No, it is not pluggable in this way. And yes, to add a native function, you need to use jsonnet as a library and call vm.NativeFunction (importing shouldn't cause any problems).

Btw could you describe what kind of functionality you want to use as native functions? What are these Go functions for?

[sparkprime]

Maybe one day we'll have importnative "foo/bar.so"

[aplsek]

@sbarzowski Just to answer your question about the desired functionality of native functions. Its really nothing complicated that could not be done in jsonnet but I already have these libraries implemented in Go with all unit testing etc.

One of the uses cases is that I use ksonnet to parametrize my K8s manifests, there I need for example to merge start arguments for my docker binary with the default arguments, I have a package that does that and checks that there are no duplicates, required flags are there, etc. So really its only string manipulation for most part but I dont want to duplicate that in jsonnet again.

[sourcec0de]

@sparkprime @sbarzowski I just opened #243 for an idea to extend go-jsonnet with a native exec function which would parse JSON output from a sub command.

• allows jsonnet to exec an arbitrary command
• if that command returns valid json it can be used
• if it fails to return valid json the exec command throws a descriptive error

local terraform_output = std.native('exec')('terraform', ['output', '-json'])