cmd/cue: path resolution issue for get go generated files

[cueckoo]

Originally opened by @errordeveloper in cuelang/cue#452

I have a Go+CUE project, the Go import root is github.com/isovalent/foo/operator (i.e. repo is github.com/isovalent/foo, and code is in operator subdirectory, I shall refer to any files using paths relative to the operator subdirectory).

I have created subdirectory cue.mod and cue.mod/module.cue contains the following line:

module: "github.com/isovalent/foo/operator"

I have created Go types defined in subdirectory api/v1alpha1, and I generated CUE schema for these types using the following command:

cue get go github.com/isovalent/foo/operator/api/v1alpha1

I created a CUE file pkg/template/testassets/template.cue, which contains:

package template

import "github.com/isovalent/foo/operator/api/v1alpha1"

And this is what I am seeing:

$ cue eval ./pkg/template/testassets/template.cue
import failed: cannot find package "github.com/isovalent/foo/operator/api/v1alpha1":
    ./pkg/template/testassets/template.cue:6:8
$

I was not able to tell what is really going on, according to the docs it should find the following directory:

cue.mod/gen/github.com/isovalent/gke-test-cluster-management/operator/api/v1alpha1

Having made a few guesses, I realised that it's looking in the same directory where Go types are, and there no CUE files in that directory.

I was able to confirm this by placing a dummy file api/v1alpha1/touch.cue:

package v1alpha1

foo: {}

And running:

$ cue eval github.com/isovalent/gke-test-cluster-management/operator/api/v1alpha1
foo: {}
$

Next, I tried to modify cue.mod/module.cue and append a string to the path:

module: "github.com/isovalent/foo/operator/cue"

No I got it working as expected, but it's still quite odd and very confusing. If Go source directories are used during import resolution, why is go.mod needed? I suppose this is just a bug.

[cueckoo]

Original reply by @mpvl in cuelang/cue#452 (comment)

One thing to test is whether it is the name pkg. This used to have special meaning when we didn't have the cue.mod system yet. And it still does, but should be ignored when it finds a cue.mod directory.

It is a bit hard to piece out what your setup from the text though. Easier would be to have a txtar reproducer (can use txtar-c to create it) and/or have a tree output of your directory structure.

[cueckoo]

Original reply by @seh in cuelang/cue#452 (comment)

This pkg directory came up in this Slack discussion in the "language" channel. Is this special treatment of the top-level pkg directory deprecated?

Here's an example that only works when the import target's files are situated beneath a top-level pkg directory:

-- outside/outside.cue --
package outside

import "example.com/component"

top: component.name
-- pkg/example.com/component/component.cue --
package component

name: "component"

I couldn't find any documentation that mentions this treatment, nor could I find where in the CUE source code it handles this case.

[grantzvolsky]

I resolved the same "import failed: cannot find package" error by making sure that the module root defined in cue.mod/module.cue is not a prefix of the generated package.

txtar to reproduce the error:

-- a/b/greeting.go --
package b

type Greeting struct {
  Who string
  Whom string
  Message string
}
-- cue/g1.cue --
import "example.com/a/b"

g: b.#Greeting
-- cue.mod/gen/example.com/a/b/greeting_go_gen.cue --
// Code generated by cue get go. DO NOT EDIT.

//cue:generate cue get go example.com/a/b

package b

#Greeting: {
        Who:     string
        Whom:    string
        Message: string
}
-- cue.mod/module.cue --
module: "example.com/a"
-- go.mod --
module example.com

go 1.16

The command cue eval cue/g1.cue yields

import failed: cannot find package "example.com/a/b":
    ./cue/g1.cue:1:8

Subsequent sed -i 's:com\/a:com\/cue:g' cue.mod/module.cue; cue eval cue/g1.cue yields

g: {
    Who:     string
    Whom:    string
    Message: string
}

[myitcv]

@errordeveloper - apologies, I missed this issue when you first created it. @grantzvolsky - thanks for the follow up which caused me to see it.

Incidentally, @mpvl's reference to a txtar reproducer is now captured in the following wiki entry: https://github.com/cue-lang/cue/wiki/Creating-test-or-performance-reproducers

Assuming I have accurately reproduced the scenario described by @errordeveloper, then I don't think this is anything to do with the pkg directory.

Firstly, to reproduce the error you are seeing:

exec cue get go github.com/isovalent/foo/operator/api/v1alpha1
exec cue eval ./pkg/template/testassets:template

-- cue.mod/module.cue --
module: "github.com/isovalent/foo"
-- go.mod --
module github.com/isovalent/foo

go 1.17
-- operator/api/v1alpha1/v1alpha1.go --
package v1alpha1

type T1 string
type T2 int
-- pkg/template/testassets/template.cue --
package template

import "github.com/isovalent/foo/operator/api/v1alpha1"

x: v1alpha1.#T1

gives:

> exec cue get go github.com/isovalent/foo/operator/api/v1alpha1
> exec cue eval ./pkg/template/testassets:template
[stderr]
import failed: no CUE files in github.com/isovalent/foo/operator/api/v1alpha1:
    ./pkg/template/testassets/template.cue:3:8
[exit status 1]
FAIL: /tmp/testscript212280920/repro.txt/script.txt:2: unexpected command failure

The missing piece in the puzzle here is the --local argument to cue get go. The reason that is required here is that cue get go is run in the context of a CUE module of the same path. So when the import comes to be resolved, the cue.mod/{pkg,gen,usr} commands are not consulted, because cmd/cue knows the package is part of the main module.

Trying with the --local flag:

exec cue get go --local github.com/isovalent/foo/operator/api/v1alpha1
exec cue eval ./pkg/template/testassets:template

-- cue.mod/module.cue --
module: "github.com/isovalent/foo"
-- go.mod --
module github.com/isovalent/foo

go 1.17
-- operator/api/v1alpha1/v1alpha1.go --
package v1alpha1

type T1 string
type T2 int
-- pkg/template/testassets/template.cue --
package template

import "github.com/isovalent/foo/operator/api/v1alpha1"

x: v1alpha1.#T1

gives:

> exec cue get go --local github.com/isovalent/foo/operator/api/v1alpha1
> exec cue eval ./pkg/template/testassets:template
[stdout]
x: string

I think the right answer here is to drop the --local flag altogether. Because the need for it can always be inferred in a modules world. This was already captured in #646. I will retitle this issue accordingly, and leave it open, ready to be fixed by #646.

[errordeveloper]

@myitcv thanks a lot, I was looking back at this as I'm picking up a new project where I'm going to be using CUE, and just confirmed that cue get go --local indeed solves the problem. I guess it's true that in most cases it's got to be possible to infer --local, however it's wont be before one created cue.mod/module.cue and set the import path there, right? Which is actually likely to be the case for most folks who are just getting started. Perhaps cue get will need to take go.mod into acccount, iunt which case one might question weather cue.mod/module.cue is reather optional in Go+CUE projects (personally, I must say, I would agree with that idea).

[myitcv]

however it's wont be before one created cue.mod/module.cue and set the import path there, right?

Correct. I'm not sure cue get go makes sense in a context without a cue.mod, because otherwise it's impossible to know for sure whether the thing being cue get-ed is the main module or not.

That said, I think you're coming at this from the perspective of a Go+CUE project, so onto your next point.

question weather cue.mod/module.cue is reather optional in Go+CUE projects

This is an interesting point. I suspect we can't make it optional, but I'm copying @rogpeppe who is looking at modules.