I'm happy to announce that Httpbeat will become a core feature of the Beats framework 6.0 with the introduction of the Http module of Metricbeat.
I would like to give special thanks to Nicolas Ruflin for his support during the migration of the Httpbeat into the new Http module.
I will continue the support for Httpbeat for Beats 5.x, but all users of Beats 6.0 and newer versions are encouraged to migrate to the Http module of Metricbeat.
Welcome to Httpbeat.
Httpbeat is a Beat used to call HTTP endpoints. Multiple endpoints can be configured which are polled in a regular interval and the result is shipped to the configured output channel.
Httpbeat is inspired by the Logstash http_poller input filter but doesn't require that the endpoint is reachable by Logstash as Httpbeat pushes the data to Logstash or Elasticsearch. This is often necessary in security restricted network setups, where Logstash is not able to reach all servers. Instead the server to be monitored itself has Httpbeat installed and can send the data or a collector server has Httpbeat installed which is deployed in the secured network environment and can reach all servers to be monitored.
Example use cases are:
- Monitor Apache Stats
- Monitor Java application with Jolokia
- Monitor Spring Boot Actuators
- Monitor Docker Remote API
Ensure that this folder is at the following location:
${GOPATH}/src/github.com/christiangalsterer
Pre-compiled binaries for different operating systems are available for download.
Install the package for your operation system by running the respective package manager or unzipping the package.
Adjust the httpbeat.yml configuration file to your needs. You may take httpbeat.full.yml as an example containing all possible configuration values.
In order to start Httpbeat please use the respective startup script, e.g. /usr/bin/httpbeat.sh.
Where supported Httpbeat can be started also using the respetive service scripts, e.g. etc/init.d/httpbeat.
To build the binary for httpbeat run the command below. This will generate a binary in the same directory with the name httpbeat.
make clean && make
To run httpbeat with debugging output enabled, run:
./httpbeat -c httpbeat.yml -e -d "*"
To run httpbeat in normal mode, run:
./httpbeat -c httpbeat.yml
To test httpbeat, run the following command:
make testsuite
alternatively:
make unit-tests
make system-tests
make integration-tests
make coverage-report
The test coverage is reported in the folder ./build/coverage/
Each beat has a template for the mapping in elasticsearch and a documentation for the fields
which is automatically generated based on etc/fields.yml.
To generate etc/httpbeat.template.json and etc/httpbeat.asciidoc
make update
To clean httpbeat source code, run the following commands:
make check
make simplify
To clean up the build directory and generated artifacts, run:
make clean
To clone httpbeat from the git repository, run the following commands:
mkdir -p ${GOPATH}/github.com/christiangalsterer
cd ${GOPATH}/github.com/christiangalsterer
git clone https://github.com/christiangalsterer/httpbeat
For further development, check out the beat developer guide.
The beat frameworks provides tools to crosscompile and package your beat for different platforms. This requires docker and vendoring as described above. To build packages of your beat, run the following command:
make package
This will fetch and create all images required for the build process. The hole process to finish can take several minutes.
4.1.0 (2017-11-01) Download
Feature and Bugfix release containing the following changes:
- Update to beats v5.6.3
- Support returning HTTP status phrase
- Fix: Set correct version in package names and package metadata
- Fix: HTTP Status code in event doesn't match documentation
- Fix: Making the template work for kibana - Use
objectinstead ofnested
4.0.0 (2017-02-19) Download
Feature and bugfix release containing the following breaking changes:
- Renamed configuration parameter
urlstohosts. Please update your configuration accordingly. - Renamed configuration parameter
crontoschedule. Please update your configuration accordingly. - Update to beats v5.2.1
- Fix: Default schedule not working
3.3.0 (2017-02-04) Download
Feature release containing the following changes:
- Update to beats v5.2.0
- Allow to specify the replacement character for
json_dot_mode : replace, default replacement character is_
3.2.0 (2017-01-14) Download
Feature and bugfix release containing the following changes:
- Various documentation improvements
- Update to beats v5.1.2
- Update to Go 1.7.4
- Fix: Panic when using TLS configuration (Go 1.7.4)
- Fix: Proper support for ssl.verification_mode
3.1.1 (2017-01-07) Download
Starting with this release pre-compiled binaries for different operating systems are available under the respective tag in the github project.
Bugfix release containing the following changes:
- Move files into correct place to allow correct bulding with
make package - Move files into correct place to allow correct bulding with
make update - Cleanup of documentation
Feature release containing the following changes:
- Update to beats v5.1.1
Feature release containing the following changes:
- Output format of response body is now defined via output_format parameter. Default is 'string'
Bugfix release containing the following changes:
Feature release containing the following changes:
- Update to beats v5.0.1
Please note that this release contains the following breaking changes introduced by beats 5.0.X, see also Beats Changelog
- SSL Configuration
- rename tls configurations section to ssl
- rename certificate_key configuration to key.
- replace tls.insecure with ssl.verification_mode setting.
- replace tls.min/max_version with ssl.supported_protocols setting requiring full protocol name
Feature release containing the following changes:
- Update to Go 1.6
- Update to libbeat 1.2.3
- Use Glide for dependency management
Feature release containing the following changes:
Bugfix release containing the following changes:
- Initial release
See here for more information.
There is exactly one document type exported:
type: httpbeathttp request and response information
See here for a detailed description of all exported fields.
{
"_index": "httpbeat-2015.12.05",
"_type": "httpbeat",
"_source": {
"@timestamp": "2015-12-05T11:16:13.070Z",
"beat": {
"hostname": "mbp.box",
"name": "mbp.box"
},
"count": 1,
"fields": {
"host": "test"
},
"request": {
"url": "http://httpbin.org/headers",
"method": "get",
"headers": {
"Accept": "application/json",
"Foo": "bar"
}
},
"response": {
"statusCode": 200,
"headers": {
"Access-Control-Allow-Credentials": "true",
"Access-Control-Allow-Origin": "*",
"Connection": "keep-alive",
"Content-Length": "220",
"Content-Type": "application/json",
"Date": "Sat, 05 Dec 2015 11:16:13 GMT",
"Server": "nginx"
},
"body": "{\n \"headers\": {\n \"Accept\": \"application/json\", \n \"Accept-Encoding\": \"gzip\", \n \"Authorization\": \"Basic Zm9vOmJhcg==\", \n \"Foo\": \"bar\", \n \"Host\": \"httpbin.org\", \n \"User-Agent\": \"Go-http-client/1.1\"\n }\n}\n"
},
"type": "httpbeat"
},
"fields": {
"timestamp": [
1449314173
]
},
"sort": [
1449314173
]
}
To apply the Httpbeat template:
curl -XPUT 'http://localhost:9200/_template/httpbeat' [email protected]
All sorts of contributions are welcome. Please create a pull request and/or issue.