"unexpected response - expected env vars response None" #670
Comments
|
Hey @JasonkayZK thanks for opening a bug report. Could you provide a bit more info about the issue:
|
|
While, the main program is simple: package main
import (
"fmt"
"net/http"
)
func HelloHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello World")
}
func main () {
http.HandleFunc("/", HelloHandler)
fmt.Println("Start!")
http.ListenAndServe(":8000", nil)
}That, i wanted to start a server in the pod. So, it's simply a go binary executable file, that created by The $ file ./main
./main: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), not strippedActually, the |
|
Thanks! |
|
The go version is I tried the command, and it did print some logs: $ k logs mirrord-agent-paqtt5rtlx-ps9sb
2022-11-01T07:28:53.571551Z INFO ThreadId(01) mirrord_agent: agent ready
2022-11-01T07:28:54.691187Z ERROR ThreadId(03) mirrord_agent: ClientConnectionHandler::start -> Client 0 disconnected with error: SnifferCommand sender failed with `channel closed`
2022-11-01T07:29:24.692296Z ERROR ThreadId(01) mirrord_agent: start_agent -> sniffer task failed with error: Agent failed with `Protocol not available (os error 92)`
2022-11-01T07:29:24.692349Z INFO ThreadId(01) mirrord_agent: main -> mirrord-agent `start` exiting successfully.Maybe some network problem? But i can't figure it out... |
|
Sorry about the troubles. I think we're close to figuring it out. |
|
Sure, i re-run the command, and here is the log: $ k logs mirrord-agent-wps50o6hqg-7gcws
2022-11-01T08:52:51.311452Z DEBUG ThreadId(01) mirrord_agent: main -> Initializing mirrord-agent.
2022-11-01T08:52:51.311547Z DEBUG ThreadId(01) mirrord_agent: starting with args Args { container_id: Some("5334850498a712d9a912da38b35496c916e220646b4ecad2eb45ad1a7589a201"), container_runtime: Some("containerd"), communicate_port: 47181, communication_timeout: 30, interface: "eth0", ephemeral_container: false }
2022-11-01T08:52:51.312405Z INFO ThreadId(01) mirrord_agent: agent ready
2022-11-01T08:52:51.313503Z DEBUG ThreadId(18) mirrord_agent::runtime: set_namespace -> fd 18
2022-11-01T08:52:51.314229Z DEBUG ThreadId(19) mirrord_agent::runtime: set_namespace -> fd 19
2022-11-01T08:52:51.314244Z DEBUG ThreadId(19) mirrord_agent::sniffer: preparing sniffer
2022-11-01T08:52:51.314247Z DEBUG ThreadId(19) mirrord_agent::sniffer: prepare_sniffer -> Preparing interface.
2022-11-01T08:52:51.791816Z DEBUG ThreadId(01) mirrord_agent: start -> Connection accepted from 127.0.0.1:43026
2022-11-01T08:52:51.791897Z TRACE ThreadId(03) new: mirrord_agent::file: enter pid=Some(270887)
2022-11-01T08:52:51.791912Z DEBUG ThreadId(03) new: mirrord_agent::file: Agent root path >> "/proc/270887/root" pid=Some(270887)
2022-11-01T08:52:51.791924Z TRACE ThreadId(03) new: mirrord_agent::file: exit pid=Some(270887)
2022-11-01T08:52:51.791953Z ERROR ThreadId(03) mirrord_agent: ClientConnectionHandler::start -> Client 0 disconnected with error: SnifferCommand sender failed with `channel closed`
2022-11-01T08:53:21.792566Z DEBUG ThreadId(01) mirrord_agent: start_agent -> main thread timeout, no clients connected
2022-11-01T08:53:21.792587Z DEBUG ThreadId(01) mirrord_agent: start_agent -> shutting down start
2022-11-01T08:53:21.792593Z ERROR ThreadId(01) mirrord_agent: start_agent -> sniffer task failed with error: Agent failed with `Protocol not available (os error 92)`
2022-11-01T08:53:21.792609Z DEBUG ThreadId(01) mirrord_agent: shutdown done
2022-11-01T08:53:21.792637Z INFO ThreadId(01) mirrord_agent: main -> mirrord-agent `start` exiting successfully.I run the k8s(single node) as Kind which is provided by: And here are these tools version: k version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-03-06T21:32:53Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.21) and server (1.23) exceeds the supported minor version skew of +/-1And by the way i firstly tried run in my k3s cluster, but it did has some network problems. |
|
I have a good guess on what's the issue - can you check if you have |
|
P.S How did you install Centos? It seems that the default libc isn't compatible with mirrord? |
|
No, there does not has the |
No it's our fault. We default to use eth0 since it's the case in the vast majority. We can add interface detection in a simple manner. We will work on adding it soon and let you know once it's ready. |
|
Well, that's great! Thx! 👍 |
|
hey @JasonkayZK, |
|
I tried with the newest version $ MIRRORD_AGENT_RUST_LOG="mirrord=trace" MIRRORD_AGENT_TTL=60 mirrord exec --target pod/busybox-58d584f97b-pbtv8 -- ./main
✓ layer initialized
✓ agent running
✓ agent pod created
✓ pod is ready
mirrord-layer received an unexpected response from the agent pod!
- Suggestions:
>> When trying to run a program with arguments in the form of `app -arg value`, run it as
`app -- -arg value` instead.
- If you're still stuck and everything looks fine:
>> Please open a new bug report at https://github.com/metalbear-co/mirrord/issues/new/choose
>> Or join our discord https://discord.com/invite/J5YSrStDKD and request help in #mirrord-help.
Expected env vars response, but got None
TerminatedHere are the logs: |
|
Hi @JasonkayZK, apologies for the delayed response. Our newest version (3.9.0) supports a configuration value for specifying which network interface to use. Could you try running it with |
|
I tried with: $ MIRRORD_AGENT_NETWORK_INTERFACE=eth1 MIRRORD_AGENT_RUST_LOG="mirrord=trace" MIRRORD_AGENT_TTL=60 mirrord exec --target pod/busybox-58d584f97b-pbtv8 -- ./mainStill got: Here's the logs: |
|
@JasonkayZK sorry I don't have ideas atm, do you want to schedule a debug session using my calendly? |
|
Yep, next tuesday works fine with me. And here is the deployment config file, maybe you can test it? busybox-deploy.yaml apiVersion: apps/v1
kind: Deployment
metadata:
name: busybox
# namespace: test
spec:
replicas: 1
selector:
matchLabels:
app: busybox
template:
metadata:
labels:
app: busybox
spec:
containers:
- name: busybox
image: busybox
args:
- /bin/sh
- -c
- sleep 10; touch /tmp/healthy; sleep 3600
readinessProbe:
exec:
command:
- cat
- /tmp/healthy
initialDelaySeconds: 10
periodSeconds: 5
ports:
- containerPort: 8000 |
|
I figured it out! It just my develop machine's network problem. $ MIRRORD_AGENT_RUST_LOG="mirrord=trace" MIRRORD_AGENT_TTL=60 mirrord exec --target pod/busybox-58d584f97b-dggqz -- ./main
✓ layer initialized
✓ agent running
✓ agent pod created
✓ pod is ready
Start!Thanks for you guys help! ❤️ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug Description
I tried to use a binary executable file for testing.
And the pod crashed:
Steps to Reproduce
mirrord exec --target pod/busybox-58d584f97b-pbtv8 -- ./main;Backtrace
No response
Relevant Logs
No response
Your operating system and version
Centos 7.9
Local process
None
Local process version
No response
Additional Info
No response
The text was updated successfully, but these errors were encountered: