How to programmatically validate data from a module?

[jbcpollak]

I'd like to write a tool that effectively wraps and executes cue programmatically, say from a server. Ideally there'd be a function that takes the unique name of a module and a byte-stream of json, and returns cue's output from running vet.

For example on the command line I can do this an capture the output:

cd my-module/
cue vet . ../path/to/my/config.json

How can I do this? I was able to hard code the cue and json data to a string and use Context.CompileString(), but now I want to make the cue data be loaded from a module and the json data be from a separate source. It feels like I should be able to do this with BuildInstance(), but I'd have to assemble the entire instance myself?

And a few related follow ups:

• On the command line, is there a way to vet using a module without being in the module? When I try and run cue vet my-module/ path/to/my/config.json I get cannot find package "my-module"?
• Is there a way to run cue against a module you haven't gotten locally yet? Something like cue vet example.com/my-cue-module my-data.json?

Thanks.

[myitcv]

Hi @jbcpollak - welcome to the CUE project!

Splitting your question into two parts: the loading of the CUE schema, and the loading of the JSON.

Loading of the CUE schema

Exactly as you observed, you ultimately have a choice: "hardcode" the CUE in the server binary, or load at runtime. There are actually options within those two options, and a nuance based on the current state of package management.

Hardcoding could be literally hardcoding as a string, or it could be via something like go:embed.

Loading a package path at runtime will generally require the server binary to be running in the context of a CUE module. CUE modules are used to resolve CUE package paths, and this all happens via cue/load.Instances. "In the context of" simply means that the working directory of the server binary must be a directory contained by a CUE module. The same is true for cmd/cue, otherwise CUE has no means of resolving a package/module path.

Note also that for now (until #851 or a variation of it lands) package/module, module resolution is limited to those modules that are either the main module itself, or modules vendored within the cue.mod/pkg/... directory hierarchy. Put another way, CUE does not yet resolve remote module/package paths, but this is very much the plan (like Go).

Loading the JSON

This is much simpler story, not least because it sounds like you will be receiving a []byte stream via a connection/similar: you can use encoding/json.Extract (or the decoder equivalent).

Combining the two

You can then use the cue.Context methods to get cue.Value's from your respective sources. Then use cue.Value.Unify to combine the results. Unification is exactly what happens when you vet something. The value that results from unification will have a non-nil cue.Value.Err() in case validation failed, at which point you have your validation result.

In the context of your server program: you can keep the cue.Value that results from the loading of the schema hanging around, and reuse that agains the cue.Value you build from each request.

Your bullet points

• On the command line, is there a way to vet using a module without being in the module? When I try and run cue vet my-module/ path/to/my/config.json I get cannot find package "my-module"?

As of today, no. Per my comments above, the working directory for cue needs to be contained within a CUE module.

• Is there a way to run cue against a module you haven't gotten locally yet? Something like cue vet example.com/my-cue-module my-data.json?

Please correct me if I'm wrong, but it sounds like you want to resolve example.com/my-cue-module outside of a CUE module. This is analogous to doing go install $pkg@$version in Go, where you don't need a Go module in order to resolve $pkg, it happens "on the fly". We could very well support something like this for the various CUE commands in the future, likely using a similar $pkg@$version syntax, but for now all resolution needs to happen via an actual CUE module, and for now remote package resolution is not yet implemented.

[jbcpollak]

Hi, thanks for the detailed response! Hi have a few follow up questions, just to check that I understand:

For loading the module, it looks like cue/load.Instances returns []build.Instance. I'll then need to convert them all to []Value via Context.BuildInstances(). I'd then combine them all into a single Value but iterating over the list with Value.Unify() correct?

For validating the JSON, at that point instead of using another call to Value.Unify(), could I use encoding/json.Validate() (probably after running encoding.json/Valid(). Is there an advantage to one or the other?

Per my comments above, the working directory for cue needs to be contained within a CUE module.

is os.Chdir() sufficient?

Please correct me if I'm wrong, but it sounds like you want to resolve example.com/my-cue-module outside of a CUE module. This is analogous to doing go install $pkg@$version in Go, where you don't need a Go module in order to resolve $pkg, it happens "on the fly". We could very well support something like this for the various CUE commands in the future, likely using a similar $pkg@$version syntax, but for now all resolution needs to happen via an actual CUE module, and for now remote package resolution is not yet implemented.

Yes, exactly - I'd like to (server side) be able to (programmatically) effectively run go install $pkg@$version (or a cue equivalent, then validate json against that installed module. So the server wouldn't need to have a given package already installed locally before validating. In the Javascript world this would be equivalent to npx, if you are familiar with that. (npx <tool-name> <params> would execute , fetching the package from npmjs.com if it wasn't already locally cached.)

[myitcv]

For loading the module, it looks like cue/load.Instances returns []build.Instance. I'll then need to convert them all to []Value via Context.BuildInstances(). I'd then combine them all into a single Value but iterating over the list with Value.Unify() correct?

If that's what you need in your situation, absolutely. Not everyone needs/wants to Unify the result, but that's certainly a legitimate thing to need/want to do.

For validating the JSON, at that point instead of using another call to Value.Unify(), could I use encoding/json.Validate() (probably after running encoding.json/Valid(). Is there an advantage to one or the other?

Yes, you absolutely could. The reason for suggesting the more "verbose" method was to highlight its existence more than anything, because sometimes people need to actually do something with the JSON prior to validating.

Yes, exactly - I'd like to (server side) be able to (programmatically) effectively run go install $pkg@$version (or a cue equivalent, then validate json against that installed module. So the server wouldn't need to have a given package already installed locally before validating. In the Javascript world this would be equivalent to npx, if you are familiar with that. (npx <tool-name> <params> would execute , fetching the package from npmjs.com if it wasn't already locally cached.)

Understood, thanks. I've added a comment to #851.

[myitcv]

Per my comments above, the working directory for cue needs to be contained within a CUE module.

is os.Chdir() sufficient?

Sorry, I missed this question.

If you are actually running cmd/cue then set os/exec.Cmd.Dir, otherwise if you are using the cue/load package then set cue/load.Config.Dir.