upstream: fix subset_lb crash when host configured with no metadata #15171

yanjunxiang-google · 2021-02-24T15:06:35Z

Problem Description:
The problem is that when executing a fuzz test case with empty hostmetadata, envoy crashed at subset_lb.cc:rebuildSingle(): line 131.

Root cause:
The root cause is that current envoy code is assuming host->medata is valid, and directly using it without NULL check. In the case if metadata is actually NULL, which is a valid configuration, it crashes.

Fix:
1)The fix is to add a if (metadat != nullptr) in subset_lb.cc:rebuildSingle() before accessing metadata.
2)GTEST code is added to reproduce the issue, also verifies the fix.
3)The oss-fuzz test file which exposed the issue is pushed with the change.

Testing:
1)The fix is tested by running the oss-fuzz test with the special test file which has host configured without metadata.
2)The fix is also tested by the newly added GTEST code.

Release Notes:
N/A

Issues:

Fix #30705 : https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=30705

Signed-off-by: Yanjun Xiang [email protected]

For an explanation of how to fill out the fields, please see the relevant section
in PULL_REQUESTS.md

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Deprecated:]
[Optional API Considerations:]

https://oss-fuzz.com/testcase-detail/5135200453525504 The problem is that when executing a fuzz test case with empty host metadata, envoy crashed at subset_lb.cc:rebuildSingle(): line 131. The root cause is that current envoy code is assuming host->medata is valid, and directly using it without NULL check. In the case if metadata is actually NULL, which is a valid configuration, it crashes. The fix is to add a if (metadat != nullptr) before accessing metadata. GTEST code is added to reproduce the issue, and verified the fix. The fix is also verified by running the oss-fuzz test with that special testcase input file. Signed-off-by: Yanjun Xiang <[email protected]>

yanjunxiang-google · 2021-02-24T15:28:28Z

/assign @asraa @adisuissa

asraa

Thanks @yanjunxiang-google!
Could you please make the issue title something like upstream: fix subset_lb crash when host configured with no metadata?

Also, could you please change the bug link to the monorail one? https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=30705. The detailed report eventually expires after some time. You can also commit the regression corpus entry as an additional regression test.

test/integration/http_subset_lb_integration_test.cc

adisuissa

Thanks for working on this!
I've added a couple of comments.
Please update the PR first comment fields, such as risk level, etc.
@asraa: should the fuzz test-case be pushed as well?

test/integration/http_subset_lb_integration_test.cc

Signed-off-by: Yanjun Xiang <[email protected]>

yanjunxiang-google · 2021-02-25T21:19:08Z

Thanks @yanjunxiang-google!
Could you please make the issue title something like upstream: fix subset_lb crash when host configured with no metadata?

Also, could you please change the bug link to the monorail one? https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=30705. The detailed report eventually expires after some time. You can also commit the regression corpus entry as an additional regression test.

Done!

yanjunxiang-google · 2021-02-25T21:22:36Z

/retest

repokitteh-read-only · 2021-02-25T21:22:39Z

Retrying Azure Pipelines:
Check envoy-presubmit didn't fail.

🐱

Caused by: a #15171 (comment) was created by @yanjunxiang-google.

see: more, trace.

yanjunxiang-google · 2021-02-25T21:31:20Z

Reopen the PR.

yanjunxiang-google · 2021-02-25T21:33:33Z

/retest

repokitteh-read-only · 2021-02-25T21:33:36Z

Retrying Azure Pipelines:
Check envoy-presubmit isn't fully completed, but will still attempt retrying.
Check envoy-presubmit didn't fail.
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #15171 (comment) was created by @yanjunxiang-google.

see: more, trace.

adisuissa

Thanks for fixing this, LGTM!

asraa · 2021-02-26T14:13:02Z

test/integration/http_subset_lb_integration_test.cc

+                                     {":authority", "host"}});
+  response->waitForEndStream();
+  EXPECT_TRUE(response->complete());
+  EXPECT_THAT(response->headers(), Http::HttpStatusIs("404"));


Do you know why this is giving a 404 instead of 503?
It might be configurable, but the default is 503

envoy/api/envoy/config/route/v3/route_components.proto

Line 564 in 7adc039

enum ClusterNotFoundResponseCode {

The configuration for this test is below: cluster and endpoints are configured. My understand is that with this configuration service is available, but page is not found, hence 404 is returned.

static_resources:
clusters:
- name: cluster_0
connect_timeout: 5s
lb_subset_config:
subset_selectors:
- keys:
- type
single_host_per_subset: true
load_assignment:
cluster_name: cluster_0
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 46423
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 40245
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 39563
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 37041

Actually now that I think about it I think we're probably hitting a NR here and not actually hitting the bad code. I think the test file sets up a requirement for the header x-type to be set to either a or b in order to actually route to the upstream. Can you take a look and make sure we're actually attempting to route via the subset lb?

Thanks! Let me take a look.

The bad code in subset_lb.cc:SubsetLoadBalancer::rebuildSingle() is hit by configuration update, not by traffic. Which means in the newly added test code:
TEST_P(HttpSubsetLbIntegrationTest, SubsetLoadBalancerSingleHostPerSubsetNoMetadata)

initialize(), which eventually call SubsetLoadBalancer::rebuildSingle() can trigger crash. We reproduced with this, also verified the fix with it.

The original test function: TEST_P(HttpSubsetLbIntegrationTest, SubsetLoadBalancer) setup the config with metadata as
...
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 34033
metadata:
filter_metadata:
envoy.lb:
type: a
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 35437
metadata:
filter_metadata:
envoy.lb:
type: b

...
also call runTest() with type_a and type_b as in TEST_P(..., SubsetLoadBalancer), in this test, Envoy is expected to route via subset lb with this test.

However, the newly added test function doesn't have type a/b metadata config, neither call runTest with the two different request types, it probably won't route via the subset lb.

In the newly added test function, the code after initialize(), i.e, creating HTTP connection and sending a HTTP request and verify the response is not specially testing the code change in SubsetLoadBalancer::rebuildSingle(), but a general purpose test, i.e, just to send some traffic, and verify response as expected.

Please let me know whether this addressed your question.

Thanks!

snowp

Thanks for the fix!

snowp · 2021-02-26T16:31:36Z

source/common/upstream/subset_lb.cc

-            // Two hosts with the same metadata value were found. Ignore all but one of them, and
-            // set a metric for how many times this happened.
-            collision_count++;
+      if (metadata != nullptr) {


Maybe do

if (metadata == nullptr) { continue; }

to reduce the nesting?

sure, will do.

snowp · 2021-02-26T16:33:14Z

test/integration/http_subset_lb_integration_test.cc

+    // Set single_host_per_subset to be true
+    auto* subset_selector = cluster->mutable_lb_subset_config()->mutable_subset_selectors(0);
+    subset_selector->set_single_host_per_subset(true);


Seems like this crash is limited to when this option is enabled, could we be more explicit about this in the comments or test name? Maybe even add a test case where this is disabled for completeness sake.

Yes, this crash only happens when single_host_per_subset is set to be true. Without it subset_lb.cc:rebuildSingle() will bailout early hence no crash. I will add some comments here to cover this.

yanjunxiang-google · 2021-02-26T19:44:26Z

Logged in with another Github account. Just deleted that comments, and recommented from the google account.

…

On Fri, Feb 26, 2021 at 2:25 PM KevinMass2018 ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In test/integration/http_subset_lb_integration_test.cc <#15171 (comment)>: > + for (uint32_t i = 0; i < num_hosts_; i++) { + auto* lb_endpoint = endpoints->mutable_lb_endpoints(i); + lb_endpoint->clear_metadata(); + } + }); + + initialize(); + codec_client_ = makeHttpConnection(lookupPort("http")); + auto response = codec_client_->makeHeaderOnlyRequest( + Http::TestRequestHeaderMapImpl{{":method", "GET"}, + {":path", "/test/long/url"}, + {":scheme", "http"}, + {":authority", "host"}}); + response->waitForEndStream(); + EXPECT_TRUE(response->complete()); + EXPECT_THAT(response->headers(), Http::HttpStatusIs("404")); The configuration used to test the issue is below: cluster and host are configured. It looks like in this case, 404 is returned. static_resources: clusters: - name: cluster_0 connect_timeout: 5s lb_subset_config: subset_selectors: - keys: - type single_host_per_subset: true load_assignment: cluster_name: cluster_0 endpoints: - lb_endpoints: - endpoint: address: socket_address: address: 127.0.0.1 port_value: 46423 - endpoint: address: socket_address: address: 127.0.0.1 port_value: 40245 - endpoint: address: socket_address: address: 127.0.0.1 port_value: 39563 - endpoint: address: socket_address: address: 127.0.0.1 port_value: 37041 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15171 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASZIHLHC7HAEEEMY3BO3RBTTA7YRXANCNFSM4YEVVESQ> .

Signed-off-by: Yanjun Xiang <[email protected]>

yanjunxiang-google · 2021-02-27T16:08:14Z

/retest

repokitteh-read-only · 2021-02-27T16:08:17Z

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #15171 (comment) was created by @yanjunxiang-google.

see: more, trace.

asraa · 2021-03-01T14:41:57Z

I think you will need to merge main when this is merged #15238

yanjunxiang-google · 2021-03-01T14:54:22Z

I think you will need to merge main when this is merged #15238
Thanks for the info!

yanjunxiang-google · 2021-03-02T14:54:25Z

/retest

repokitteh-read-only · 2021-03-02T14:54:29Z

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #15171 (comment) was created by @yanjunxiang-google.

see: more, trace.

snowp

Thanks!

snowp · 2021-03-02T15:50:40Z

test/integration/http_subset_lb_integration_test.cc

+                                     {":authority", "host"}});
+  response->waitForEndStream();
+  EXPECT_TRUE(response->complete());
+  EXPECT_THAT(response->headers(), Http::HttpStatusIs("404"));


Actually now that I think about it I think we're probably hitting a NR here and not actually hitting the bad code. I think the test file sets up a requirement for the header x-type to be set to either a or b in order to actually route to the upstream. Can you take a look and make sure we're actually attempting to route via the subset lb?

yanjunxiang-google · 2021-03-04T15:22:42Z

@snowp

I checked the code, and also the test logs. Here is some comments:

The bad code in subset_lb.cc:SubsetLoadBalancer::rebuildSingle() is hit by configuration update, not by traffic. Which means in the newly added test code:
TEST_P(HttpSubsetLbIntegrationTest, SubsetLoadBalancerSingleHostPerSubsetNoMetadata)
initialize(), which eventually call SubsetLoadBalancer::rebuildSingle() can trigger crash. We reproduced with this, also verified the fix with it.

The original test function: TEST_P(HttpSubsetLbIntegrationTest, SubsetLoadBalancer) setup the config with metadata as
...

lb_endpoints:
endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 34033
metadata:
filter_metadata:
envoy.lb:
type: a
endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 35437
metadata:
filter_metadata:
envoy.lb:
type: b
...
also call runTest() with type_a_request and type_b_request as in TEST_P(..., SubsetLoadBalancer), in this test, Envoy is routing via subset lb, and verified.

However, the newly added test function doesn't have type a/b metadata config, neither call runTest with the two different request types, it probably won't route via the subset lb.

In the newly added test function, the code after initialize(), i.e, creating HTTP connection and sending a HTTP request and verify the response is not specially testing the code change in SubsetLoadBalancer::rebuildSingle(), but a general purpose test, i.e, just to send some traffic, and verify response as expected.

Please let me know whether this addressed your question.

Thanks!

snowp · 2021-03-04T15:36:46Z

Even if we're not having to validate the behavior via the traffic it seems odd to have a request that doesn't hit the subset LB at all? Would it be hard to make it route via the subset LB? I think we'd just need to set a single header

yanjunxiang-google · 2021-03-04T16:53:28Z

Even if we're not having to validate the behavior via the traffic it seems odd to have a request that doesn't hit the subset LB at all? Would it be hard to make it route via the subset LB? I think we'd just need to set a single header

Sure, let me add this.

yanjunxiang-google · 2021-03-09T14:52:44Z

working on this today

Signed-off-by: Yanjun Xiang <[email protected]>

yanjunxiang-google · 2021-03-09T19:01:54Z

Even if we're not having to validate the behavior via the traffic it seems odd to have a request that doesn't hit the subset LB at all? Would it be hard to make it route via the subset LB? I think we'd just need to set a single header

Sure, let me add this.

With below RequestHeader setup,
Http::TestRequestHeaderMapImpl{{":method", "GET"},
{":path", "/test"},
{":scheme", "http"},
{":authority", "host"},
{"x-type", "a"},
{"x-hash", "hash-a"}});

It's observed Envoy tried to route the request, but failed due to "no healthy host for HTTP connection pool". I think this is expected due to no metadata present in host configuration. Below is the complete error logs. Please check the latest code change.

2021-03-09 16:05:42.229][130][trace][connection] [source/common/network/connection_impl.cc:349] [C47] readDisable: disable=true disable_count=0 state=0 buffer_length=80
[2021-03-09 16:05:42.229][130][debug][http] [source/common/http/conn_manager_impl.cc:883] [C47][S7508058493963056235] request headers complete (end_stream=true):
':authority', 'host'
':path', '/test'
':method', 'GET'
'x-type', 'a'
'x-hash', 'hash-a'
'content-length', '0'

[2021-03-09 16:05:42.229][130][debug][http] [source/common/http/filter_manager.cc:774] [C47][S7508058493963056235] request end stream
[2021-03-09 16:05:42.229][130][debug][router] [source/common/router/router.cc:428] [C47][S7508058493963056235] cluster 'cluster_0' match for URL '/test'
[2021-03-09 16:05:42.229][130][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1367] no healthy host for HTTP connection pool
[2021-03-09 16:05:42.230][130][debug][http] [source/common/http/filter_manager.cc:858] [C47][S7508058493963056235] Sending local reply with details no_healthy_upstream
[2021-03-09 16:05:42.230][130][trace][misc] [source/common/event/scaled_range_timer_manager_impl.cc:60] enableTimer called on 0x6543f616080 for 300000ms, min is 300000ms
[2021-03-09 16:05:42.230][130][debug][http] [source/common/http/conn_manager_impl.cc:1454] [C47][S7508058493963056235] encoding headers via codec (end_stream=false):
':status', '503'
'content-length', '19'
'content-type', 'text/plain'
'date', 'Tue, 09 Mar 2021 16:05:42 GMT'
'server', 'envoy'

Thanks!

yanjunxiang-google · 2021-03-09T19:29:45Z

/retest

repokitteh-read-only · 2021-03-09T19:29:49Z

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #15171 (comment) was created by @yanjunxiang-google.

see: more, trace.

Signed-off-by: Yanjun Xiang <[email protected]>

snowp · 2021-03-09T21:10:29Z

Yeah that looks right to me, thanks for updating the test!

…nvoyproxy#15171) This commit is to fix oss-fuzz test issue: https://oss-fuzz.com/testcase-detail/5135200453525504 The problem is that when executing a fuzz test case with empty host metadata, envoy crashed at subset_lb.cc:rebuildSingle(): line 131. The root cause is that current envoy code is assuming host->medata is valid, and directly using it without NULL check. In the case if metadata is actually NULL, which is a valid configuration, it crashes. The fix is to add a if (metadat != nullptr) before accessing metadata. GTEST code is added to reproduce the issue, and verified the fix. The fix is also verified by running the oss-fuzz test with that special testcase input file. Signed-off-by: Yanjun Xiang <[email protected]> Signed-off-by: Auni Ahsan <[email protected]>

repokitteh-read-only bot assigned adisuissa and asraa Feb 24, 2021

asraa reviewed Feb 24, 2021

View reviewed changes

test/integration/http_subset_lb_integration_test.cc Outdated Show resolved Hide resolved

adisuissa reviewed Feb 24, 2021

View reviewed changes

test/integration/http_subset_lb_integration_test.cc Outdated Show resolved Hide resolved

test/integration/http_subset_lb_integration_test.cc Show resolved Hide resolved

upstream: fix subset_lb crash when host configured with no metadata

c5401fc

Signed-off-by: Yanjun Xiang <[email protected]>

yanjunxiang-google changed the title ~~This commit is to fix oss-fuzz test issue:~~ upstream: fix subset_lb crash when host configured with no metadata Feb 25, 2021

yanjunxiang-google closed this Feb 25, 2021

yanjunxiang-google reopened this Feb 25, 2021

adisuissa previously approved these changes Feb 26, 2021

View reviewed changes

asraa reviewed Feb 26, 2021

View reviewed changes

asraa assigned snowp Feb 26, 2021

snowp suggested changes Feb 26, 2021

View reviewed changes

upstream: fix subset_lb crash when host configured with no metadata

b46d0da

Signed-off-by: Yanjun Xiang <[email protected]>

yanjunxiang-google dismissed adisuissa’s stale review via b46d0da February 26, 2021 22:26

snowp approved these changes Mar 2, 2021

View reviewed changes

snowp suggested changes Mar 2, 2021

View reviewed changes

snowp added the waiting label Mar 3, 2021

upstream: fix subset_lb crash when host configured with no metadata

7d4f55e

Signed-off-by: Yanjun Xiang <[email protected]>

repokitteh-read-only bot removed the waiting label Mar 9, 2021

upstream: fix subset_lb crash when host configured with no metadata

01c5a75

Signed-off-by: Yanjun Xiang <[email protected]>

snowp approved these changes Mar 9, 2021

View reviewed changes

snowp merged commit e8cd93d into envoyproxy:main Mar 10, 2021

yanjunxiang-google deleted the yanjun_oss_fuzz_crash branch July 19, 2021 21:15

upstream: fix subset_lb crash when host configured with no metadata #15171

upstream: fix subset_lb crash when host configured with no metadata #15171

Conversation

yanjunxiang-google commented Feb 24, 2021 • edited Loading

yanjunxiang-google commented Feb 24, 2021

asraa left a comment • edited Loading

Choose a reason for hiding this comment

adisuissa left a comment

Choose a reason for hiding this comment

yanjunxiang-google commented Feb 25, 2021

yanjunxiang-google commented Feb 25, 2021

repokitteh-read-only bot commented Feb 25, 2021

yanjunxiang-google commented Feb 25, 2021

yanjunxiang-google commented Feb 25, 2021

repokitteh-read-only bot commented Feb 25, 2021

adisuissa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yanjunxiang-google Feb 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

snowp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yanjunxiang-google commented Feb 26, 2021 via email

yanjunxiang-google commented Feb 27, 2021

repokitteh-read-only bot commented Feb 27, 2021

asraa commented Mar 1, 2021

yanjunxiang-google commented Mar 1, 2021

yanjunxiang-google commented Mar 2, 2021

repokitteh-read-only bot commented Mar 2, 2021

snowp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yanjunxiang-google commented Mar 4, 2021

snowp commented Mar 4, 2021

yanjunxiang-google commented Mar 4, 2021

yanjunxiang-google commented Mar 9, 2021

yanjunxiang-google commented Mar 9, 2021

yanjunxiang-google commented Mar 9, 2021

repokitteh-read-only bot commented Mar 9, 2021

snowp commented Mar 9, 2021

yanjunxiang-google commented Feb 24, 2021 •

edited

Loading

asraa left a comment •

edited

Loading

yanjunxiang-google Feb 26, 2021 •

edited

Loading