Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Heartbeat] HTTP request should include the beat name and the version as part of the user-agent. #10170

Closed
ph opened this issue Jan 18, 2019 · 11 comments · Fixed by #14297
Closed

[Heartbeat] HTTP request should include the beat name and the version as part of the user-agent. #10170

ph opened this issue Jan 18, 2019 · 11 comments · Fixed by #14297
Assignees
@andrewvc
Labels
enhancement Heartbeat libbeat Team:obs-ds-hosted-services Label for the Observability Hosted Services team

Comments

@ph
Copy link
Contributor

ph commented Jan 18, 2019

Currently, the HTTP request from beats doesn't contain the agent name of the version number, it would be useful to add the information to help to debug or generating proxy rules.
@andrewkroh
Copy link
Member

+1. Not sure what scope you had in mind, but I'd like both Heartbeat (http monitor) and Metricbeat (all the http based pollers) to do this.

@ph
Copy link
Contributor Author

ph commented Jan 21, 2019

@andrewvc I am a strong believer that all projects (not just beat) should correctly report themselves so I would include Heartbeat and Metricbeat. So probably the best way to do it would be that libbeat provides a thin wrapper over the official http package.

Note: Also I think it would be sufficient to just scope the user agent to a beat (and not the feature), so in the case of Metricbeat requests to external service and central management requests to Kibana would have use the same user agent.

@andrewvc
Copy link
Contributor

I'm +1 on this for heartbeat. It's a little wonky in that we'd need to let users override it. We do let users set headers, and my feeling is they could disable the UA by setting it to "".

@andrewvc
Copy link
Contributor

WRT the wrapper, one thing about HB is that we override a lot of stuff already. I'll have to look at the code once your wrapper is in, but I'm a little worried we may need to do our own impl.

@ph
Copy link
Contributor Author

ph commented Jan 23, 2019

@andrewvc Oh, I wonder if we could do the following:

  1. If the user-agent is not set in the headers we add it, done by the wrapper.
  2. If users explicitly override it, we log it when we do the config unpack?

Concerning 2, I am not sure how Golang behave if we explicitly set the user-agent to an empty string.

@ghost
Copy link

ghost commented Mar 13, 2019

I'd love to see this. In operations, the growing number of things that all claim to be Go-http-client/1.1 makes things tricky.

Imagine if all the major browsers said "Oh hai, I'm C++".

@andrewvc andrewvc changed the title Beats HTTP request should include the beat name and the version as part of the user-agent. [Heartbeat] HTTP request should include the beat name and the version as part of the user-agent. Oct 29, 2019
@andrewvc
Copy link
Contributor

Reproducing the text of #14291 here since it's a dupe:

Heartbeat currently uses the default golang user-agent Go-http-client/1.1. This appears to be blacklisted by some sites, including our own, when checking some URLs, like https://www.elastic.co/products/beats/heartbeat . You can test the blocked nature of the golang UA by running curl -A "Go-http-client/1.1" -I https://www.elastic.co/products/beats/heartbeat.

We should override the default golang UA and use our own, perhaps elastic-heartbeat/VERSION where VERSION is the heartbeat version number. That should be compliant with the relevant RFC https://tools.ietf.org/html/rfc7231#section-5.5.3 .

@andrewvc andrewvc mentioned this issue Oct 29, 2019
Closed
@andrewvc andrewvc mentioned this issue Oct 29, 2019
Merged
andrewvc added a commit to andrewvc/beats that referenced this issue Oct 29, 2019
@andrewvc
[Heartbeat] Set HTTP user-agent to 'elastic_heartbeat'
6caa091 
By default heartbeat uses the golang user agent, which is blacklisted
by a number of services, including elastic.co. URLs like
https://www.elastic.co/products/beats/heartbeat do not work with the
default go user agent due to such blacklists.

This changes the default user agent to be 'elastic_heartbeat'.

While the HTTP spec allows UAs to add version numbers to the end this PR
does not add the version number for two reasons:

1. Adding the version wouldn't practically be of use to anyone as one UA
version numbers for an uptime check aren't practically useful.
2. It would needlessly add complexity to this commit.

Resolves elastic#10170 (comment)
@andrewvc andrewvc added the Team:obs-ds-hosted-services Label for the Observability Hosted Services team label Oct 29, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/uptime (:uptime)

@andrewvc andrewvc self-assigned this Oct 29, 2019
andrewvc added a commit that referenced this issue Oct 31, 2019
@andrewvc
Set HTTP user-agent in Heartbeat +Others (#14297)
956f87c 
By default heartbeat uses the golang user agent, which is blacklisted by a number of services, including elastic.co. URLs like https://www.elastic.co/products/beats/heartbeat do not work with the
default go user agent due to such blacklists.

This changes the default user agent to be 'Elastic Heartbeat/VERSION (PLATFORM INFO)'

Resolves #10170 (comment)
andrewvc added a commit to andrewvc/beats that referenced this issue Oct 31, 2019
@andrewvc
Set HTTP user-agent in Heartbeat +Others (elastic#14297)
85a4d60 
By default heartbeat uses the golang user agent, which is blacklisted by a number of services, including elastic.co. URLs like https://www.elastic.co/products/beats/heartbeat do not work with the
default go user agent due to such blacklists.

This changes the default user agent to be 'Elastic Heartbeat/VERSION (PLATFORM INFO)'

Resolves elastic#10170 (comment)

(cherry picked from commit 956f87c)
@andrewvc andrewvc mentioned this issue Oct 31, 2019
Closed
@archon810
Copy link

archon810 commented Apr 24, 2020 •
edited
Loading

Any reason this wasn't done to Metricbeat as well, for the same reason? In the meantime, I was able to specify the http header via the config file, which overrode the user agent successfully.

@andrewvc
Copy link
Contributor

@archon810 can you open a separate issue for that? This issue is closed so it won't get much attention.

@archon810 archon810 mentioned this issue May 3, 2020
Closed
@archon810
Copy link

Done.

jorgemarey pushed a commit to jorgemarey/beats that referenced this issue Jun 8, 2020
@andrewvc @jorgemarey
Set HTTP user-agent in Heartbeat +Others (elastic#14297)
73febf6 
By default heartbeat uses the golang user agent, which is blacklisted by a number of services, including elastic.co. URLs like https://www.elastic.co/products/beats/heartbeat do not work with the
default go user agent due to such blacklists.

This changes the default user agent to be 'Elastic Heartbeat/VERSION (PLATFORM INFO)'

Resolves elastic#10170 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andrewvc andrewvc

Labels
enhancement Heartbeat libbeat Team:obs-ds-hosted-services Label for the Observability Hosted Services team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Heartbeat] Set HTTP user-agent in Heartbeat andrewvc/beats
5 participants
@ph @andrewvc @archon810 @andrewkroh @elasticmachine