Etcdbeat is an Elastic Beat that reads stats from the Etcd v2 API and indexes them into Elasticsearch or Logstash.
Etcd is an open-source distributed key value store that provides a reliable way to store data across a cluster of machines.
An etcd cluster keeps track of a number of statistics including latency, bandwidth and uptime. Etcdbeat collects this stats with etcd API.
Check out this blog post on how it works.
There are three types of documents exported:
type: leaderContains etcd leader statstype: selfContains etcd self statstype: storeContains etcd store stats
Leader Stats
The leader has a view of the entire cluster and keeps track of two interesting statistics: latency to each peer in the cluster, and the number of failed and successful Raft RPC requests.
Self Stats
Each node keeps a number of internal statistics:
id: the unique identifier for the memberleaderInfo.leader: id of the current leader memberleaderInfo.uptime: amount of time the leader has been leadername: this member's namerecvAppendRequestCnt: number of append requests this node has processedrecvBandwidthRate: number of bytes per second this node is receiving (follower only)recvPkgRate: number of requests per second this node is receiving (follower only)sendAppendRequestCnt: number of requests that this node has sentsendBandwidthRate: number of bytes per second this node is sending (leader only). This value is undefined on single member clusters.sendPkgRate: number of requests per second this node is sending (leader only). This value is undefined on single member clusters.state: either leader or followerstartTime: the time when this node was started
Document example:
{
"self": {
"id": "8e9e05c52164694d",
"leaderInfo": {
"leader": "8e9e05c52164694d",
"startTime": "2017-08-07T12:24:14.306354646+03:00",
"uptime": "8m52.578056951s"
},
"name": "node2",
"recvAppendRequestCnt": 0,
"recvBandwidthRate": 6345,
"recvPkgRate": 824.1758351191694,
"sendAppendRequestCnt": 11.111234716807138,
"startTime": "2017-08-07T12:24:13.805809934+03:00",
"state": "StateLeader"
}
}
Store Stats
The store statistics include information about the operations that this node has handled.Operations that modify the store's state like create, delete, set and update are seen by the entire cluster and the number will increase on all nodes.
Document example:
{
"store": {
"compareAndDeleteFail": 0,
"compareAndDeleteSuccess": 0,
"compareAndSwapFail": 0,
"compareAndSwapSuccess": 0,
"createFail": 0,
"createSuccess": 2,
"deleteFail": 0,
"deleteSuccess": 0,
"expireCount": 0,
"getsFail": 4,
"getsSuccess": 42,
"setsFail": 3,
"setsSuccess": 6,
"updateFail": 0,
"updateSuccess": 0,
"watchers": 0
}
}
Check out this blog post for more details.
Adjust the etcdbeat.yml configuration file to your needs.
Defines how often to read statistics. Default to 30 s.
Defines the etcd port serviced. Default to 2379
Host name of ElasticSearch. Default to localhost
You can decide which statistics to collect.
statistics:
leader: false
self: true
store: true
Authentication to be used for API connection. Default to enable: false
authentication:
enable: false
username: test
password: test1234
The default template is provided, if you add any queries you should update the template accordingly.
To apply default Etcdbeat template run:
curl -XPUT 'http://<host>:9200/_template/etcdbeat' [email protected]
Ensure that this folder is at the following location:
${GOPATH}/github.com/gamegos
- Golang 1.7
To get running with Etcdbeat and also install the dependencies, run the following command:
make setup
It will create a clean git history for each major step. Note that you can always rewrite the history if you wish before pushing your changes.
To push Etcdbeat in the git repository, run the following commands:
git remote set-url origin https://github.com/gamegos/etcdbeat
git push origin master
For further development, check out the beat developer guide.
To build the binary for Etcdbeat run the command below. This will generate a binary in the same directory with the name etcdbeat.
make
To run Etcdbeat with debugging output enabled, run:
./etcdbeat -c etcdbeat.yml -e -d "*"
To test Etcdbeat, run the following command:
make testsuite
alternatively:
make unit-tests
make system-tests
make integration-tests
make coverage-report
The test coverage is reported in the folder ./build/coverage/
Each beat has a template for the mapping in elasticsearch and a documentation for the fields
which is automatically generated based on etc/fields.yml.
To generate etc/etcdbeat.template.json and etc/etcdbeat.asciidoc
make update
To clean Etcdbeat source code, run the following commands:
make fmt
make simplify
To clean up the build directory and generated artifacts, run:
make clean
To clone Etcdbeat from the git repository, run the following commands:
mkdir -p ${GOPATH}/github.com/gamegos
cd ${GOPATH}/github.com/gamegos
git clone https://github.com/gamegos/etcdbeat
For further development, check out the beat developer guide.
The beat frameworks provides tools to crosscompile and package your beat for different platforms. This requires docker and vendoring as described above. To build packages of your beat, run the following command:
make package
This will fetch and create all images required for the build process. The hole process to finish can take several minutes.
Berfin Sarı <berfinsari21 'at' gmail.com>
Covered under the Apache License, Version 2.0 Copyright (c) 2017 Berfin Sarı