custom built 'k3s kubectl' and 'k3s crictl' do not work #231

ggardet · 2019-03-14T12:50:04Z

Describe the bug
k3s kubectl --help returns:

Error: unknown command "kubectl" for "kubectl"
Run 'kubectl --help' for usage.
unknown command "kubectl" for "kubectl"

k3s crictl--help returns:

No help topic for 'crictl'

To Reproduce
Steps to reproduce the behavior:
Build k3s, symlink kubectl and crictl to k3s and run: k3s kubectl --help or k3s crictl --help

Expected behavior
k3s kubectl should handle kubectl requests. Same for crictl.

Screenshots
None

Additional context
It is built from sources, v0.2.0. Maybe some flags/options are missing to enable kubectl and crictl support?

The text was updated successfully, but these errors were encountered:

erikwilson · 2019-03-14T21:10:19Z

The recommended way to build k3s is using ./scripts/build && ./scripts/package, so everything is bundled together and it knows the correct path to find the related executables when they are unpacked. Is your path setup to use the directory where the files are symlinked?

erikwilson · 2019-03-25T21:02:32Z

Is there any more info needed here @ggardet? Maybe better documentation on building?

ggardet · 2019-03-28T14:54:45Z

I was following https://github.com/rancher/k3s#building-from-source to build

and if I try to use the build script I get:

Building server
+ CGO_ENABLED=1
+ go build -tags 'ctrd apparmor seccomp no_btrfs netgo osusergo' -ldflags '-X github.com/rancher/k3s/pkg/version.Version=v0.3.0-rc3 -X github.com/rancher/k3s/pkg/version.GitCommit=93cc6462 -w -s ' -o bin/containerd ./cmd/server/main.go
cmd/server/main.go:7:2: cannot find package "github.com/docker/docker/pkg/reexec" in any of:
        /usr/lib64/go/1.11/src/github.com/docker/docker/pkg/reexec (from $GOROOT)
        /home/guillaume/go/src/github.com/docker/docker/pkg/reexec (from $GOPATH)
cmd/server/main.go:8:2: cannot find package "github.com/kubernetes-sigs/cri-tools/cmd/crictl" in any of:
        /usr/lib64/go/1.11/src/github.com/kubernetes-sigs/cri-tools/cmd/crictl (from $GOROOT)
        /home/guillaume/go/src/github.com/kubernetes-sigs/cri-tools/cmd/crictl (from $GOPATH)
cmd/server/main.go:9:2: cannot find package "github.com/rancher/k3s/pkg/cli/agent" in any of:
        /usr/lib64/go/1.11/src/github.com/rancher/k3s/pkg/cli/agent (from $GOROOT)
        /home/guillaume/go/src/github.com/rancher/k3s/pkg/cli/agent (from $GOPATH)
cmd/server/main.go:10:2: cannot find package "github.com/rancher/k3s/pkg/cli/cmds" in any of:
        /usr/lib64/go/1.11/src/github.com/rancher/k3s/pkg/cli/cmds (from $GOROOT)
        /home/guillaume/go/src/github.com/rancher/k3s/pkg/cli/cmds (from $GOPATH)
cmd/server/main.go:11:2: cannot find package "github.com/rancher/k3s/pkg/cli/crictl" in any of:
        /usr/lib64/go/1.11/src/github.com/rancher/k3s/pkg/cli/crictl (from $GOROOT)
        /home/guillaume/go/src/github.com/rancher/k3s/pkg/cli/crictl (from $GOPATH)
cmd/server/main.go:12:2: cannot find package "github.com/rancher/k3s/pkg/cli/kubectl" in any of:
        /usr/lib64/go/1.11/src/github.com/rancher/k3s/pkg/cli/kubectl (from $GOROOT)
        /home/guillaume/go/src/github.com/rancher/k3s/pkg/cli/kubectl (from $GOPATH)
cmd/server/main.go:13:2: cannot find package "github.com/rancher/k3s/pkg/cli/server" in any of:
        /usr/lib64/go/1.11/src/github.com/rancher/k3s/pkg/cli/server (from $GOROOT)
        /home/guillaume/go/src/github.com/rancher/k3s/pkg/cli/server (from $GOPATH)
cmd/server/main.go:14:2: cannot find package "github.com/rancher/k3s/pkg/containerd" in any of:
        /usr/lib64/go/1.11/src/github.com/rancher/k3s/pkg/containerd (from $GOROOT)
        /home/guillaume/go/src/github.com/rancher/k3s/pkg/containerd (from $GOPATH)
cmd/server/main.go:15:2: cannot find package "github.com/rancher/k3s/pkg/kubectl" in any of:
        /usr/lib64/go/1.11/src/github.com/rancher/k3s/pkg/kubectl (from $GOROOT)
        /home/guillaume/go/src/github.com/rancher/k3s/pkg/kubectl (from $GOPATH)
cmd/server/main.go:16:2: cannot find package "github.com/sirupsen/logrus" in any of:
        /usr/lib64/go/1.11/src/github.com/sirupsen/logrus (from $GOROOT)
        /home/guillaume/go/src/github.com/sirupsen/logrus (from $GOPATH)
cmd/server/main.go:17:2: cannot find package "github.com/urfave/cli" in any of:
        /usr/lib64/go/1.11/src/github.com/urfave/cli (from $GOROOT)
        /home/guillaume/go/src/github.com/urfave/cli (from $GOPATH)

rhrn · 2019-05-14T21:37:40Z

@ggardet try this

go get -v github.com/rancher/k3s
go get -v github.com/sirupsen/logrus
go get -v github.com/urfave/cli

warmchang · 2019-07-18T12:07:26Z

I guess it caused by GOPATH.

brandond · 2020-12-05T00:31:06Z

Closing due to age.

hramrach · 2021-03-26T17:50:31Z

still broken in 1.20.4+k3s1

brandond · 2021-03-26T23:07:17Z

@hramrach would you like to expand on that comment?

hramrach · 2021-03-27T11:11:43Z

strace -f k3s kubectl get nodes |& grep ENOENT
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
[pid 2483] <... openat resumed>) = -1 ENOENT (No such file or directory)
[pid 2483] <... openat resumed>) = -1 ENOENT (No such file or directory)
[pid 2483] newfstatat(AT_FDCWD, "k3s", 0xc0007aa108, 0) = -1 ENOENT (No such file or directory)
[pid 2483] <... openat resumed>) = -1 ENOENT (No such file or directory)
[pid 2483] openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 2483] openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 2483] openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 2483] openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 2483] newfstatat(AT_FDCWD, "/sbin/kubectl-kubectl-get-nodes", 0xc0012ec1d8, 0) = -1 ENOENT (No such file or directory)
[pid 2483] newfstatat(AT_FDCWD, "/usr/sbin/kubectl-kubectl-get-nodes", 0xc0012ec2a8, 0) = -1 ENOENT (No such file or directory)
[pid 2483] newfstatat(AT_FDCWD, "/usr/local/sbin/kubectl-kubectl-get-nodes", 0xc0012ec378, 0) = -1 ENOENT (No such file or directory)
[pid 2483] newfstatat(AT_FDCWD, "/root/bin/kubectl-kubectl-get-nodes", 0xc0012ec448, 0) = -1 ENOENT (No such file or directory)
[pid 2483] newfstatat(AT_FDCWD, "/usr/local/bin/kubectl-kubectl-get-nodes", 0xc0012ec518, 0) = -1 ENOENT (No such file or directory)
[pid 2483] newfstatat(AT_FDCWD, "/usr/bin/kubectl-kubectl-get-nodes", 0xc0012ec5e8, 0) = -1 ENOENT (No such file or directory)
[pid 2483] newfstatat(AT_FDCWD, "/bin/kubectl-kubectl-get-nodes", 0xc0012ec6b8, 0) = -1 ENOENT (No such file or directory)
[pid 2483] newfstatat(AT_FDCWD, "/sbin/kubectl-kubectl-get", 0xc0012ec788, 0) = -1 ENOENT (No such file or directory)
[pid 2483] newfstatat(AT_FDCWD, "/usr/sbin/kubectl-kubectl-get", 0xc0012ec858, 0) = -1 ENOENT (No such file or directory)
[pid 2483] newfstatat(AT_FDCWD, "/usr/local/sbin/kubectl-kubectl-get", 0xc0012ec928, 0) = -1 ENOENT (No such file or directory)
[pid 2483] <... newfstatat resumed>0xc0012ec9f8, 0) = -1 ENOENT (No such file or directory)
[pid 2483] <... newfstatat resumed>0xc0012ecac8, 0) = -1 ENOENT (No such file or directory)
[pid 2483] <... newfstatat resumed>0xc0012ecb98, 0) = -1 ENOENT (No such file or directory)
[pid 2483] <... newfstatat resumed>0xc0012ecc68, 0) = -1 ENOENT (No such file or directory)
[pid 2483] <... newfstatat resumed>0xc0012ecd38, 0) = -1 ENOENT (No such file or directory)
[pid 2483] <... newfstatat resumed>0xc0012ece08, 0) = -1 ENOENT (No such file or directory)
[pid 2483] <... newfstatat resumed>0xc0012eced8, 0) = -1 ENOENT (No such file or directory)
[pid 2483] <... newfstatat resumed>0xc0012ecfa8, 0) = -1 ENOENT (No such file or directory)
[pid 2483] <... newfstatat resumed>0xc0012ed078, 0) = -1 ENOENT (No such file or directory)
[pid 2483] newfstatat(AT_FDCWD, "/usr/bin/kubectl-kubectl", 0xc0012ed148, 0) = -1 ENOENT (No such file or directory)
[pid 2483] newfstatat(AT_FDCWD, "/bin/kubectl-kubectl", 0xc0012ed218, 0) = -1 ENOENT (No such file or directory)

This is totally bogus.

it searches for external command implementing a functionality that's supposed to be included to start with
it searches for kubectl-kubectl - WTF?

hramrach · 2021-03-27T11:13:32Z

Note that the message

Error: unknown command "kubectl" for "kubectl"

already hints at this problem.

hramrach · 2021-03-27T11:17:05Z

strace -f k3s crictl --help |& grep ENOENT
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
[pid 14211] <... openat resumed>) = -1 ENOENT (No such file or directory)
[pid 14211] openat(AT_FDCWD, "/etc/apache/mime.types", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 14211] newfstatat(AT_FDCWD, "k3s", 0xc0014685e8, 0) = -1 ENOENT (No such file or directory)
[pid 14211] openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 14211] openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 14211] openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 14211] openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 14211] openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 14211] newfstatat(AT_FDCWD, "k3s", 0xc001468858, 0) = -1 ENOENT (No such file or directory)
[pid 14211] openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 14211] openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 14211] openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 14211] openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 14211] openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 14211] openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 14211] openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 14211] newfstatat(AT_FDCWD, "/etc/crictl.yaml", 0xc0014689f8, 0) = -1 ENOENT (No such file or directory)
[pid 14211] newfstatat(AT_FDCWD, "/usr/bin/crictl.yaml", 0xc001468ac8, 0) = -1 ENOENT (No such file or directory)

brandond · 2021-03-27T17:19:44Z

Are you seeing this when building from source, or on a released version? Note that k3s is a self-extracting multicall binary, that's how we package containerd, kubectl, etc. If you are building from source but not running the correct binary you will see this behavior.

hramrach · 2021-03-28T19:15:25Z

Build from source, that's what this bug is about. What it the correct binary?
When executing the generic go packaging macro apparently the wrong binary gets installed.

brandond · 2021-03-28T22:30:40Z

Are you following the instructions in BUILDING.md, or just running go build?

hramrach · 2021-03-29T14:02:16Z

I am running

%{goprep} github.com/rancher/k3s
%{gobuild}

The BUILDING.md adivises

mkdir -p build/data && ./scripts/download && go generate
make

There is problem with both commands.
./scripts/download requires network connection which is not available in the build environment
make does not apply system-specific go flags

brandond · 2021-03-29T16:53:58Z

K3s is not just K3s - it also embeds kubernetes, containerd, flannel, etc. For that reason it cannot be built as simply as a more traditional standalone go binary. You need to build it the way we build it. If necessary, you may need to adapt your build pipeline to pull down the Traefik artifacts and place them in the appropriate location as if you were running the download script. If you need to customize the go flags, you can tweak the makefile as necessary for your platform.

hramrach · 2021-04-17T09:06:27Z

scripts/download downloads binaries during build.

Is there no way to build k3s from source?

brandond · 2021-04-18T06:45:05Z

K3s is built from many pieces. It includes a full set of userspace binaries, and manifests for all the packaged components. Some of these are pulled from other repos in this project, while others come from different projects. Which binary are you having a problem with?

SchoolGuy · 2021-11-02T18:24:00Z

@brandond I have to agree with @hramrach here. We need to be able to build this from source in an offline environment like the openSUSE Build Service. For example I ran into #3704 because the version you ship in the OBS is outdated. I know Golang packaging is a pain in the OBS but compliance is a thing these days. At least I don't want to download random stuff from the internet, the past should show that this is not a clever thing to do.

P.S.: You do find me in Slack, so if you need help with the packaging we can go through the pain together.

Oats87 · 2021-11-02T18:25:46Z

@brandond I have to agree with @hramrach here. We need to be able to build this from source in an offline environment like the openSUSE Build Service. For example I ran into #3704 because the version you ship in the OBS is outdated. I know Golang packaging is a pain in the OBS but compliance is a thing these days. At least I don't want to download random stuff from the internet, the past should show that this is not a clever thing to do.

P.S.: You do find me in Slack, so if you need help with the packaging we can go through the pain together.

I don't believe K3s maintainers are utilizing OBS to ship anything at this point. The K3s package that exists for OpenSUSE is not an official package.

SchoolGuy · 2021-11-02T18:28:49Z

@Oats87 I know this is politically incorrect but Rancher has joined SUSE and thus they live in our Ecosystem. Customers do want RPMs from the IBS. Since at SUSE we have the Factory first policy, I personally (can't speak for the company though) would believe they have inherited the maintenance if they want or not. I mean we do offer MicroOS Flavors with preinstalled k3s.

Oats87 · 2021-11-02T18:33:38Z

@SchoolGuy I'm not so sure I agree with that. I'm sure folks want official RPMs for K3s but at this point we (the K3s maintainers) don't even have that on our roadmap.

To my understanding, the folks that initially created the K3s package for OpenSUSE are part of the openSUSE Kubic project. OpenSUSE is exactly what it is stated as -- open, and thus maintenance of the packages for for OpenSUSE, while they may be influenced by SUSE, are still technically community driven.

brandond · 2021-11-02T18:49:43Z

I'll repost what I said on slack, which is a paraphrased version of what I posted above.

The k3s binary itself IS a self-extracting package. It includes containerd, crictl, kubernetes, a full static userspace from buildroot (by way of k3s-root), etc. Having all those things together in a specific combination of versions is what makes a K3s release, and is a big part of its portability value proposition. If you start pulling those things out - using an external containerd, dropping the userspace, etc - you don’t really have K3s, but something else that’s downstream from K3s.... which is what I’d kinda argue the current opensuse packaging is.

Also keep in mind that K3s is a CNCF project, and does not need to (nor should it be asked to) suse-ify itself. We have other downstream projects like RKE2 that are more tightly tied to business deliverables.

Also, related: #4347

hramrach · 2021-11-02T20:46:15Z

OK, I think we should remove K3s from openSUSE then. It is not meant to be buildable from source so we cannot distribute it in openSUSE.

brandond · 2021-11-02T20:59:12Z

It is not meant to be buildable from source

That is an incorrect statement. If it's not built from source, then what is it built from?

Like many Go projects, it is not buildable solely from the source code in this repository. It is easiest to build it online so that the build scripts can pull down various external components. It can be built in an isolated environment with some additional preparation.

hramrach · 2021-11-02T21:22:33Z

No, it cannot be built from source. It uses other repositories but does not build the software from those repositories from source, it downloads prebuilt binaries. It does not support using binaries preinstalled on the system for its dependencies either, it must be a binary package downloaded form the Internet (either during build or in advance). Some of these projects it relies on require binaries of themselves to build, there is no provision for building from source whatsoever.

SchoolGuy · 2021-11-02T21:30:51Z

@hramrach Then you can't sell any of our products except SLE. Vendoring is part of the OBS currently. And because this is Golang it can be built offline. So no we don't have to remove it from the OBS.

hramrach · 2021-11-03T11:08:27Z

@hramrach Then you can't sell any of our products except SLE. Vendoring is part of the OBS currently. And because this is Golang it can be built offline. So no we don't have to remove it from the OBS.

Vendoring sources or vendoring binaries?

k3s is vendoring binaries while openSUSE only allows vendoring sources AFAIK.

SchoolGuy · 2021-11-03T11:58:36Z

@hramrach We do both afaik. At least if you consider shipping minified JS as a binary.

hramrach · 2021-11-03T12:04:38Z

@hramrach We do both afaik. At least if you consider shipping minified JS as a binary.

That probably escaped review because it is not obvious that the file is a binary.

SchoolGuy · 2021-11-03T12:22:54Z

@hramrach Let's take this discussion elsewhere to not spam this issue.

erikwilson added the kind/question No code change, just asking/answering a question label Mar 25, 2019

erikwilson changed the title ~~'k3s kubectl' and 'k3s crictl' do not work~~ custom built 'k3s kubectl' and 'k3s crictl' do not work Mar 25, 2019

deniseschannon assigned ibuildthecloud Apr 3, 2019

deniseschannon added kind/documentation Improvements or additions to documentation and removed kind/question No code change, just asking/answering a question labels May 7, 2019

deniseschannon unassigned ibuildthecloud May 7, 2019

davidnuzik added this to the Backlog milestone Jan 16, 2020

davidnuzik added [zube]: To Triage and removed [zube]: To Triage labels Jan 16, 2020

cjellick added [zube]: To Triage and removed [zube]: Backlog labels Apr 28, 2020

davidnuzik added the [zube]: Unscheduled label May 1, 2020

zube bot removed the [zube]: To Triage label May 1, 2020

brandond closed this as completed Dec 5, 2020

davidnuzik added [zube]: Done and removed [zube]: Unscheduled labels Dec 5, 2020

eapolinario mentioned this issue May 12, 2023

[BUG] error: unknown command "kubectl" for "kubectl" flyteorg/flyte#3653

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

custom built 'k3s kubectl' and 'k3s crictl' do not work #231

custom built 'k3s kubectl' and 'k3s crictl' do not work #231

ggardet commented Mar 14, 2019 •

edited

Loading

erikwilson commented Mar 14, 2019

erikwilson commented Mar 25, 2019

ggardet commented Mar 28, 2019 •

edited

Loading

rhrn commented May 14, 2019

warmchang commented Jul 18, 2019

brandond commented Dec 5, 2020

hramrach commented Mar 26, 2021

brandond commented Mar 26, 2021

hramrach commented Mar 27, 2021

hramrach commented Mar 27, 2021

hramrach commented Mar 27, 2021

brandond commented Mar 27, 2021

hramrach commented Mar 28, 2021

brandond commented Mar 28, 2021

hramrach commented Mar 29, 2021

brandond commented Mar 29, 2021 •

edited

Loading

hramrach commented Apr 17, 2021

brandond commented Apr 18, 2021

SchoolGuy commented Nov 2, 2021

Oats87 commented Nov 2, 2021

SchoolGuy commented Nov 2, 2021

Oats87 commented Nov 2, 2021 •

edited

Loading

brandond commented Nov 2, 2021 •

edited

Loading

hramrach commented Nov 2, 2021

brandond commented Nov 2, 2021

hramrach commented Nov 2, 2021

SchoolGuy commented Nov 2, 2021

hramrach commented Nov 3, 2021

SchoolGuy commented Nov 3, 2021

hramrach commented Nov 3, 2021

SchoolGuy commented Nov 3, 2021

custom built 'k3s kubectl' and 'k3s crictl' do not work #231

custom built 'k3s kubectl' and 'k3s crictl' do not work #231

Comments

ggardet commented Mar 14, 2019 • edited Loading

erikwilson commented Mar 14, 2019

erikwilson commented Mar 25, 2019

ggardet commented Mar 28, 2019 • edited Loading

rhrn commented May 14, 2019

warmchang commented Jul 18, 2019

brandond commented Dec 5, 2020

hramrach commented Mar 26, 2021

brandond commented Mar 26, 2021

hramrach commented Mar 27, 2021

hramrach commented Mar 27, 2021

hramrach commented Mar 27, 2021

brandond commented Mar 27, 2021

hramrach commented Mar 28, 2021

brandond commented Mar 28, 2021

hramrach commented Mar 29, 2021

brandond commented Mar 29, 2021 • edited Loading

hramrach commented Apr 17, 2021

brandond commented Apr 18, 2021

SchoolGuy commented Nov 2, 2021

Oats87 commented Nov 2, 2021

SchoolGuy commented Nov 2, 2021

Oats87 commented Nov 2, 2021 • edited Loading

brandond commented Nov 2, 2021 • edited Loading

hramrach commented Nov 2, 2021

brandond commented Nov 2, 2021

hramrach commented Nov 2, 2021

SchoolGuy commented Nov 2, 2021

hramrach commented Nov 3, 2021

SchoolGuy commented Nov 3, 2021

hramrach commented Nov 3, 2021

SchoolGuy commented Nov 3, 2021

ggardet commented Mar 14, 2019 •

edited

Loading

ggardet commented Mar 28, 2019 •

edited

Loading

brandond commented Mar 29, 2021 •

edited

Loading

Oats87 commented Nov 2, 2021 •

edited

Loading

brandond commented Nov 2, 2021 •

edited

Loading