[extension/jaegerremotesampling] Flaky test - TestStartAndShutdownRemote

[jpkrohling]

Logs:

2022-04-06T19:52:18.7017926Z make -C ./extension/jaegerremotesampling test
2022-04-06T19:52:18.7035598Z make[2]: Entering directory '/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/extension/jaegerremotesampling'
2022-04-06T19:52:18.7089756Z go test -race -v -timeout 300s --tags="" ./...
2022-04-06T19:52:19.8199960Z === RUN   TestLoadConfig
2022-04-06T19:52:19.8201021Z --- PASS: TestLoadConfig (0.01s)
2022-04-06T19:52:19.8201622Z === RUN   TestValidate
2022-04-06T19:52:19.8202491Z === RUN   TestValidate/no_receiving_protocols
2022-04-06T19:52:19.8202884Z === RUN   TestValidate/no_sources
2022-04-06T19:52:19.8203215Z === RUN   TestValidate/too_many_sources
2022-04-06T19:52:19.8203777Z --- PASS: TestValidate (0.00s)
2022-04-06T19:52:19.8204379Z     --- PASS: TestValidate/no_receiving_protocols (0.00s)
2022-04-06T19:52:19.8204827Z     --- PASS: TestValidate/no_sources (0.00s)
2022-04-06T19:52:19.8205274Z     --- PASS: TestValidate/too_many_sources (0.00s)
2022-04-06T19:52:19.8205573Z === RUN   TestNewExtension
2022-04-06T19:52:19.8205908Z --- PASS: TestNewExtension (0.00s)
2022-04-06T19:52:19.8206458Z === RUN   TestStartAndShutdownLocalFile
2022-04-06T19:52:19.8206880Z --- PASS: TestStartAndShutdownLocalFile (0.00s)
2022-04-06T19:52:19.8207233Z === RUN   TestStartAndShutdownRemote
2022-04-06T19:52:19.8207674Z 2022/04/06 19:52:19 FATAL: [core] grpc: Server.RegisterService after Server.Serve for "jaeger.api_v2.SamplingManager"
2022-04-06T19:52:19.8208354Z FAIL	github.com/open-telemetry/opentelemetry-collector-contrib/extension/jaegerremotesampling	0.035s
2022-04-06T19:52:19.8208813Z === RUN   TestMissingClientConfigManager
2022-04-06T19:52:19.8209241Z --- PASS: TestMissingClientConfigManager (0.00s)
2022-04-06T19:52:19.8209560Z === RUN   TestStartAndStop
2022-04-06T19:52:19.8209898Z --- PASS: TestStartAndStop (0.00s)
2022-04-06T19:52:19.8210192Z === RUN   TestEndpointsAreWired
2022-04-06T19:52:19.8210490Z === RUN   TestEndpointsAreWired/legacy
2022-04-06T19:52:19.8210798Z === RUN   TestEndpointsAreWired/new
2022-04-06T19:52:19.8211167Z --- PASS: TestEndpointsAreWired (0.00s)
2022-04-06T19:52:19.8211626Z     --- PASS: TestEndpointsAreWired/legacy (0.00s)
2022-04-06T19:52:19.8212094Z     --- PASS: TestEndpointsAreWired/new (0.00s)
2022-04-06T19:52:19.8212416Z === RUN   TestServiceNameIsRequired
2022-04-06T19:52:19.8212804Z --- PASS: TestServiceNameIsRequired (0.00s)
2022-04-06T19:52:19.8213149Z === RUN   TestErrorFromClientConfigManager
2022-04-06T19:52:19.8213583Z --- PASS: TestErrorFromClientConfigManager (0.00s)
2022-04-06T19:52:19.8214155Z PASS
2022-04-06T19:52:19.8214712Z ok  	github.com/open-telemetry/opentelemetry-collector-contrib/extension/jaegerremotesampling/internal	(cached)
2022-04-06T19:52:19.8215086Z FAIL
2022-04-06T19:52:19.8328315Z make[2]: *** [../../Makefile.Common:46: test] Error 1
2022-04-06T19:52:19.8332324Z make[1]: *** [Makefile:162: for-all-target-./extension/jaegerremotesampling] Error 2
2022-04-06T19:52:19.8333144Z make[2]: Leaving directory '/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/extension/jaegerremotesampling'
2022-04-06T19:52:19.8336794Z make[1]: Leaving directory '/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib'
2022-04-06T19:52:19.8337572Z make: *** [Makefile:72: gotest] Error 2
2022-04-06T19:52:19.8359336Z ##[error]Process completed with exit code 2.

[jpkrohling]

And here: https://github.com/open-telemetry/opentelemetry-collector-contrib/runs/5887191395?check_suite_focus=true

[pmm-sumo]

Here as well: https://github.com/open-telemetry/opentelemetry-collector-contrib/runs/5972401892?check_suite_focus=true

[djaglowski]

This failed again here: https://github.com/open-telemetry/opentelemetry-collector-contrib/runs/7957402089?check_suite_focus=true#step:8:940

=== RUN   TestStartAndShutdownRemote
    extension_test.go:83: 
        	Error Trace:	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/extension/jaegerremotesampling/extension_test.go:83
        	Error:      	Received unexpected error:
        	            	error while starting the gRPC server: failed to listen on gRPC port: listen tcp :14250: bind: address already in use
        	Test:       	TestStartAndShutdownRemote

[jpkrohling]

cc @frzifus, I think I pinged you on the wrong issue. I wanted to ping you on this one here. Sorry about the noise.

[frzifus]

Sure, i can take over :)

[frzifus]

i wonder that it fails in line 83.

opentelemetry-collector-contrib/extension/jaegerremotesampling/extension_test.go

Line 83 in ea97fda

assert.NoError(t, e.Start(context.Background(), componenttest.NewNopHost()))

Since the port is already allocated in line 58.

opentelemetry-collector-contrib/extension/jaegerremotesampling/extension_test.go

Lines 57 to 58 in ea97fda

	lis, err := net.Listen("tcp", "localhost:0")
	require.NoError(t, err)

I changed localhost:0 to localhost:8080 while blocking that port with nc -l -p 8080.
Then i see:

=== RUN   TestStartAndShutdownRemote
    extension_test.go:58: 
        	Error Trace:	/home/frzifus/git/go/observability/opentelemetry-collector-contrib/extension/jaegerremotesampling/extension_test.go:58
        	Error:      	Received unexpected error:
        	            	listen tcp 127.0.0.1:8080: bind: address already in use
        	Test:       	TestStartAndShutdownRemote
--- FAIL: TestStartAndShutdownRemote (0.00s)
FAIL
exit status 1

If you take a closer look at the extention start method, only a connection to the server is established but no port is allocated.

What do you think about adding a retry mechanism to the test evaluation?

[djaglowski]

Again here: https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/3197849469/jobs/5221638347#step:8:1027

[github-actions]

Pinging code owners: @jpkrohling. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jpkrohling]

What do you think about adding a retry mechanism to the test evaluation?

That might be worth trying, but this failure pattern happens elsewhere. I think there's an underlying problem we need to solve, I just don't know yet which problem this is. Without digging too much into it, I would guess this is related to how we run the tests (with -race and some parallel test executions).

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• extension/jaegerremotesampling: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atoulme]

We just hit this again:
https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/3851828150/jobs/6563425908

=== RUN   TestStartAndShutdownRemote
    extension_test.go:83: 
        	Error Trace:	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/extension/jaegerremotesampling/extension_test.go:83
        	Error:      	Received unexpected error:
        	            	error while starting the gRPC server: failed to listen on gRPC port: listen tcp 127.0.0.1:14250: bind: address already in use
        	Test:       	TestStartAndShutdownRemote
--- FAIL: TestStartAndShutdownRemote (0.00s)

[atoulme]

Just hit this again:

=== RUN   TestStartAndShutdownRemote
    extension_test.go:83: 
        	Error Trace:	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/extension/jaegerremotesampling/extension_test.go:83
        	Error:      	Received unexpected error:
        	            	error while starting the gRPC server: failed to listen on gRPC port: listen tcp 127.0.0.1:14250: bind: address already in use
        	Test:       	TestStartAndShutdownRemote

https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/4301204205/jobs/7498184609

[atoulme]

another occurrence: https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/4378772863/jobs/7663958484

[lewis262626]

I have experienced the same in https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/4387063396/jobs/7685532139#step:9:520