[MicroK8s] mirrord_agent: main -> mirrord-agent start exiting with error

[DazWilkin]

Bug Description

Email thread w/ [email protected] and posting here at Aviram's request.

Writing a Kubernetes Operator

mirrord works w/ GKE, LKE, and VKE!

mirrord doesn't work with MicroK8s.

Perhaps the fact that MicroK8s is running on the same host as mirrord!?

However, mirrord would still be useful in this scenario to save the build-deploy cycle.

Steps to Reproduce

1. Deploy the operator in namespace Foo
2. Able to query the Kubernetes API and kubectl get {resource}
3. Unable to:

mirrord exec \
--target=deployment/${NAME} \
--target-namespace=${NAME} \
--steal \
--agent-log-level=trace \
./target/debug/{bin}

Result:

x mirrord cli starting
  x preparing to launch process
  ✓ layer extracted
  ✓ agent pod created
  x waiting for pod to be ready...
Error: 
  × Agent wasn't ready in time

The agent (job|pod) returns the logs below.

Backtrace

No response

Relevant Logs

2023-03-13T22:26:38.341052Z DEBUG ThreadId(01) mirrord_agent: main -> Initializing mirrord-agent.
2023-03-13T22:26:38.341099Z DEBUG ThreadId(01) mirrord_agent: start_iptable_guard -> Initializing iptable-guard.
2023-03-13T22:26:38.341439Z ERROR ThreadId(01) mirrord_agent: main -> mirrord-agent `start` exiting with error Transport(
    tonic::transport::Error(
        Transport,
        hyper::Error(
            Connect,
            Os {
                code: 2,
                kind: NotFound,
                message: "No such file or directory",
            },
        ),
    ),
)

### Your operating system and version

Linux hades-canyon 5.15.0-67-generic #74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

### Local process

mirrord exec --target deployment/${NAME} --target-namespace=${NAMESPACE} --steal --agent-log-level=trace --help ./target/debug/{bin}

### Local process version

_No response_

### Additional Info

MicroK8s:L v1.26.1

```bash
kubectl version --output=yaml
clientVersion:
  buildDate: "2022-12-08T19:58:30Z"
  compiler: gc
  gitCommit: b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d
  gitTreeState: clean
  gitVersion: v1.26.0
  goVersion: go1.19.4
  major: "1"
  minor: "26"
  platform: linux/amd64
kustomizeVersion: v4.5.7
serverVersion:
  buildDate: "2023-01-19T13:44:28Z"
  compiler: gc
  gitCommit: 8f94681cd294aa8cfd3407b8191f6c70214973a4
  gitTreeState: clean
  gitVersion: v1.26.1
  goVersion: go1.19.5
  major: "1"
  minor: "26"
  platform: linux/amd64

And (upgraded mirrord today without difference):

mirrord --version
mirrord 3.32.1

[aviramha]

Thank you for the report!

The issue is probably with the location MicroK8S puts the containerd socket. I think a fairly simple solution is to add another fallback (as we have already few fallbacks there).

[DazWilkin]

This works, thanks!

I was unable to curl ... version 3.32.2 but I cloned the repo and it now works correctly.

[aviramha]

Thanks for the update!