Flaky tests are not ignored on CI #625
Assignees
Labels
Comments
hawkw
added a commit
that referenced
this issue
Mar 27, 2018
Flaky proxy tests were not actually being ignored properly. This is due to our use of a Cargo workspace; as it turns out that Cargo doesn't propagate feature flags from the workspace to the crates in the workspace (see rust-lang/cargo#4753). If I run `cargo test --no-default-features` in the root directory, the `flaky_tests` feature is still passed, and the flaky tests still run: ``` ➜ cargo test --no-default-features Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs Running target/debug/deps/conduit_proxy-0e0ab2829c6b743f running 13 tests test fully_qualified_authority::tests::test_normalized_authority ... ok test ctx::transport::tests::same_addr_ip6_compat_ipv4 ... ok test ctx::transport::tests::same_addr_ipv4 ... ok test ctx::transport::tests::same_addr_ip6_mapped_ipv4 ... ok test ctx::transport::tests::same_addr_ipv6 ... ok test telemetry::tap::match_::tests::http_from_proto ... ok test inbound::tests::recognize_default_no_ctx ... ok test telemetry::tap::match_::tests::tcp_from_proto ... ok test telemetry::tap::match_::tests::tcp_matches ... ok test inbound::tests::recognize_default_no_loop ... ok test transparency::tcp::tests::duplex_doesnt_hang_when_one_half_finishes ... ok test inbound::tests::recognize_default_no_orig_dst ... ok test inbound::tests::recognize_orig_dst ... ok test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/debug/deps/conduit_proxy-74584a35ef749a60 running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/debug/deps/discovery-73cd0b65bd7a45ae running 16 tests test http1::absolute_uris::outbound_reconnects_if_controller_stream_ends ... ok test http1::outbound_reconnects_if_controller_stream_ends ... ok test http1::absolute_uris::outbound_uses_orig_dst_if_not_local_svc ... ok test http1::outbound_asks_controller_without_orig_dst ... ok test http1::absolute_uris::outbound_asks_controller_api ... ok test http1::outbound_asks_controller_api ... ok test http1::absolute_uris::outbound_asks_controller_without_orig_dst ... ok test http2::outbound_reconnects_if_controller_stream_ends ... ok test http2::outbound_asks_controller_api ... ok test http2::outbound_asks_controller_without_orig_dst ... ok test http1::outbound_uses_orig_dst_if_not_local_svc ... ok server h1 error: invalid HTTP version specified test http2::outbound_uses_orig_dst_if_not_local_svc ... ok ERROR 2018-03-26T20:54:09Z: conduit_proxy: turning Error caused by underlying HTTP/2 error: protocol error: frame with invalid size into 500 test outbound_updates_newer_services ... ok ERROR 2018-03-26T20:54:09Z: conduit_proxy: turning operation timed out after Duration { secs: 0, nanos: 100000000 } into 500 test http1::absolute_uris::outbound_times_out ... ok ERROR 2018-03-26T20:54:09Z: conduit_proxy: turning operation timed out after Duration { secs: 0, nanos: 100000000 } into 500 test http2::outbound_times_out ... ok ERROR 2018-03-26T20:54:09Z: conduit_proxy: turning operation timed out after Duration { secs: 0, nanos: 100000000 } into 500 test http1::outbound_times_out ... ok test result: ok. 16 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/debug/deps/telemetry-cb5bee2d2b94332c running 12 tests test metrics_endpoint_inbound_request_count ... ok test metrics_endpoint_inbound_request_duration ... ok test metrics_endpoint_outbound_request_count ... ok test records_latency_statistics ... ignored test telemetry_report_errors_are_ignored ... ok test metrics_endpoint_outbound_request_duration ... ok test metrics_have_no_double_commas ... ok test http1_inbound_sends_telemetry ... ok test inbound_sends_telemetry ... ok test inbound_aggregates_telemetry_over_several_requests ... ok test metrics_endpoint_inbound_response_latency ... ok test metrics_endpoint_outbound_response_latency ... ok test result: ok. 11 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out Running target/debug/deps/transparency-9d14bf92d8ba3700 running 19 tests ERROR 2018-03-26T20:54:10Z: conduit_proxy: turning Error caused by underlying HTTP/2 error: protocol error: unexpected internal error encountered into 500 test http11_upgrade_not_supported ... ok test http11_absolute_uri_differs_from_host ... ok test http10_without_host ... ok test http1_head_responses ... ok test http10_with_host ... ok test http1_connect_not_supported ... ok test http1_bodyless_responses ... ok test http1_content_length_zero_is_preserved ... ok test http1_removes_connection_headers ... ok test http1_one_connection_per_host ... ok test inbound_http1 ... ok test inbound_tcp ... ok test http1_requests_without_body_doesnt_add_transfer_encoding ... ok test http1_response_end_of_file ... ok test http1_requests_without_host_have_unique_connections ... ok test outbound_tcp ... ok test tcp_with_no_orig_dst ... ok test tcp_connections_close_if_client_closes ... ok test outbound_http1 ... ok test result: ok. 19 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/debug/deps/conduit_proxy_controller_grpc-7fdac3528475b1dc running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/debug/deps/conduit_proxy_router-024926cac5d328ee running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/debug/deps/convert-ae9bd3b8fee21c85 running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/debug/deps/futures_mpsc_lossy-4afd31454ff77b40 running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Doc-tests conduit-proxy running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Doc-tests conduit-proxy-controller-grpc running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Doc-tests convert running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Doc-tests conduit-proxy-router running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Doc-tests futures-mpsc-lossy running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out ``` This also happens if the `-p` flag is used to run tests only in the `conduit-proxy` crate: ``` ➜ cargo test -p conduit-proxy --no-default-features Compiling conduit-proxy v0.3.0 (file:///Users/eliza/Code/go/src/github.com/runconduit/conduit/proxy) Finished dev [unoptimized + debuginfo] target(s) in 17.27 secs Running target/debug/deps/conduit_proxy-0e0ab2829c6b743f running 13 tests test fully_qualified_authority::tests::test_normalized_authority ... ok test ctx::transport::tests::same_addr_ip6_mapped_ipv4 ... ok test ctx::transport::tests::same_addr_ipv6 ... ok test ctx::transport::tests::same_addr_ipv4 ... ok test ctx::transport::tests::same_addr_ip6_compat_ipv4 ... ok test inbound::tests::recognize_default_no_loop ... ok test telemetry::tap::match_::tests::http_from_proto ... ok test inbound::tests::recognize_default_no_orig_dst ... ok test inbound::tests::recognize_default_no_ctx ... ok test transparency::tcp::tests::duplex_doesnt_hang_when_one_half_finishes ... ok test telemetry::tap::match_::tests::tcp_from_proto ... ok test inbound::tests::recognize_orig_dst ... ok test telemetry::tap::match_::tests::tcp_matches ... ok test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/debug/deps/conduit_proxy-74584a35ef749a60 running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/debug/deps/discovery-73cd0b65bd7a45ae running 16 tests test http1::absolute_uris::outbound_reconnects_if_controller_stream_ends ... ok test http1::outbound_reconnects_if_controller_stream_ends ... ok test http1::absolute_uris::outbound_asks_controller_without_orig_dst ... ok test http1::absolute_uris::outbound_uses_orig_dst_if_not_local_svc ... ok test http1::outbound_asks_controller_without_orig_dst ... ok test http1::absolute_uris::outbound_asks_controller_api ... ok test http1::outbound_asks_controller_api ... ok test http1::outbound_uses_orig_dst_if_not_local_svc ... ok test http2::outbound_reconnects_if_controller_stream_ends ... ok test http2::outbound_asks_controller_without_orig_dst ... ok test http2::outbound_asks_controller_api ... ok test http2::outbound_uses_orig_dst_if_not_local_svc ... ok server h1 error: invalid HTTP version specified ERROR 2018-03-26T20:56:50Z: conduit_proxy: turning Error caused by underlying HTTP/2 error: protocol error: frame with invalid size into 500 test outbound_updates_newer_services ... ok ERROR 2018-03-26T20:56:50Z: conduit_proxy: turning operation timed out after Duration { secs: 0, nanos: 100000000 } into 500 test http1::absolute_uris::outbound_times_out ... ok ERROR 2018-03-26T20:56:50Z: conduit_proxy: turning operation timed out after Duration { secs: 0, nanos: 100000000 } into 500 test http1::outbound_times_out ... ok ERROR 2018-03-26T20:56:50Z: conduit_proxy: turning operation timed out after Duration { secs: 0, nanos: 100000000 } into 500 test http2::outbound_times_out ... ok test result: ok. 16 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/debug/deps/telemetry-cb5bee2d2b94332c running 12 tests test metrics_endpoint_inbound_request_duration ... ok test metrics_endpoint_inbound_request_count ... ok test metrics_endpoint_outbound_request_count ... ok test metrics_endpoint_outbound_request_duration ... ok test telemetry_report_errors_are_ignored ... ok test metrics_have_no_double_commas ... ok test inbound_sends_telemetry ... ok test http1_inbound_sends_telemetry ... ok test inbound_aggregates_telemetry_over_several_requests ... ok test metrics_endpoint_inbound_response_latency ... ok test metrics_endpoint_outbound_response_latency ... ok test records_latency_statistics ... ok test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/debug/deps/transparency-9d14bf92d8ba3700 running 19 tests ERROR 2018-03-26T20:56:55Z: conduit_proxy: turning Error caused by underlying HTTP/2 error: protocol error: unexpected internal error encountered into 500 test http1_connect_not_supported ... ok test http11_upgrade_not_supported ... ok test http10_without_host ... ok test http11_absolute_uri_differs_from_host ... ok test http1_head_responses ... ok test http10_with_host ... ok test http1_bodyless_responses ... ok test http1_content_length_zero_is_preserved ... ok test http1_removes_connection_headers ... ok test http1_one_connection_per_host ... ok test http1_response_end_of_file ... ok test http1_requests_without_host_have_unique_connections ... ok test inbound_http1 ... ok test inbound_tcp ... ok test http1_requests_without_body_doesnt_add_transfer_encoding ... ok test outbound_tcp ... ok test tcp_with_no_orig_dst ... ok test tcp_connections_close_if_client_closes ... ok test outbound_http1 ... ok test result: ok. 19 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Doc-tests conduit-proxy running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out ``` However, if I `cd` into the `proxy` directory (so that Cargo treats the `conduit-proxy` crate as the root project, rather than the workspace) and pass the `--no-default-features` flag, the flaky tests are skipped as expected: ``` ➜ (cd proxy && exec cargo test --no-default-features) Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs Running /Users/eliza/Code/go/src/github.com/runconduit/conduit/target/debug/deps/conduit_proxy-ac198a96228a056e running 13 tests test fully_qualified_authority::tests::test_normalized_authority ... ok test ctx::transport::tests::same_addr_ipv4 ... ok test ctx::transport::tests::same_addr_ip6_compat_ipv4 ... ok test ctx::transport::tests::same_addr_ipv6 ... ok test ctx::transport::tests::same_addr_ip6_mapped_ipv4 ... ok test telemetry::tap::match_::tests::tcp_from_proto ... ok test telemetry::tap::match_::tests::http_from_proto ... ok test transparency::tcp::tests::duplex_doesnt_hang_when_one_half_finishes ... ok test telemetry::tap::match_::tests::tcp_matches ... ok test inbound::tests::recognize_default_no_ctx ... ok test inbound::tests::recognize_default_no_loop ... ok test inbound::tests::recognize_default_no_orig_dst ... ok test inbound::tests::recognize_orig_dst ... ok test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running /Users/eliza/Code/go/src/github.com/runconduit/conduit/target/debug/deps/conduit_proxy-41e0f900f97e194b running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running /Users/eliza/Code/go/src/github.com/runconduit/conduit/target/debug/deps/discovery-7ba7fe16345a347a running 16 tests test http1::absolute_uris::outbound_times_out ... ignored test http1::outbound_times_out ... ignored test http1::absolute_uris::outbound_reconnects_if_controller_stream_ends ... ok test http1::outbound_reconnects_if_controller_stream_ends ... ok test http1::absolute_uris::outbound_uses_orig_dst_if_not_local_svc ... ok test http1::outbound_uses_orig_dst_if_not_local_svc ... ok test http1::absolute_uris::outbound_asks_controller_without_orig_dst ... ok test http1::outbound_asks_controller_without_orig_dst ... ok test http1::outbound_asks_controller_api ... ok test http1::absolute_uris::outbound_asks_controller_api ... ok test http2::outbound_times_out ... ignored server h1 error: invalid HTTP version specified ERROR 2018-03-26T21:48:32Z: conduit_proxy: turning Error caused by underlying HTTP/2 error: protocol error: frame with invalid size into 500 test http2::outbound_reconnects_if_controller_stream_ends ... ok test http2::outbound_uses_orig_dst_if_not_local_svc ... ok test http2::outbound_asks_controller_api ... ok test http2::outbound_asks_controller_without_orig_dst ... ok test outbound_updates_newer_services ... ok test result: ok. 13 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out Running /Users/eliza/Code/go/src/github.com/runconduit/conduit/target/debug/deps/telemetry-b0763b64edd8fc68 running 12 tests test metrics_endpoint_inbound_request_count ... ignored test metrics_endpoint_inbound_request_duration ... ignored test metrics_endpoint_inbound_response_latency ... ignored test metrics_endpoint_outbound_request_count ... ignored test metrics_endpoint_outbound_request_duration ... ignored test metrics_endpoint_outbound_response_latency ... ignored test records_latency_statistics ... ignored test telemetry_report_errors_are_ignored ... ok test metrics_have_no_double_commas ... ok test http1_inbound_sends_telemetry ... ok test inbound_sends_telemetry ... ok test inbound_aggregates_telemetry_over_several_requests ... ok test result: ok. 5 passed; 0 failed; 7 ignored; 0 measured; 0 filtered out Running /Users/eliza/Code/go/src/github.com/runconduit/conduit/target/debug/deps/transparency-300fd801daa85ccf running 19 tests ERROR 2018-03-26T21:48:32Z: conduit_proxy: turning Error caused by underlying HTTP/2 error: protocol error: unexpected internal error encountered into 500 test http1_connect_not_supported ... ok test http11_upgrade_not_supported ... ok test http10_without_host ... ok test http10_with_host ... ok test http11_absolute_uri_differs_from_host ... ok test http1_head_responses ... ok test http1_bodyless_responses ... ok test http1_removes_connection_headers ... ok test http1_content_length_zero_is_preserved ... ok test http1_one_connection_per_host ... ok test http1_response_end_of_file ... ok test http1_requests_without_body_doesnt_add_transfer_encoding ... ok test inbound_tcp ... ok test inbound_http1 ... ok test http1_requests_without_host_have_unique_connections ... ok test outbound_tcp ... ok test tcp_connections_close_if_client_closes ... ok test tcp_with_no_orig_dst ... ok test outbound_http1 ... ok test result: ok. 19 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Doc-tests conduit-proxy running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out ``` I'm wrapping the `cd` and `cargo test` command in a subshell so that the CWD on Travis is still in the repo root when the command exits, but the return value from `cargo test` is propagated. Closes #625
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
See #614 (comment)
It looks like this is because we're running
cargo testin the root of the repo (i.e. against the cargo workspace) and not against theconduit-proxycrate, which exposes the feature. Cargo doesn't propagate feature flags from the workspace to the crates in the workspace (see rust-lang/cargo#4753).The text was updated successfully, but these errors were encountered: