Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Focus README on docker-compose; README fixups; reduce docker-compose dep #7

Merged
merged 2 commits into from
Oct 26, 2020
Merged

Focus README on docker-compose; README fixups; reduce docker-compose dep #7

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2946aaa
Focus README on docker-compose; README fixups; reduce docker-compose dep
rshaull Oct 22, 2020
63a77b3
Fix typo
rshaull Oct 26, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
190 changes: 129 additions & 61 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<img src="images/nuodb.svg" width="200" height="200" />
<img src="images/nuodb.svg" width="200" height="200" />

# NuoDB Collector

Expand Down Expand Up @@ -28,87 +28,75 @@ Built into this container are 4 input plugins to collect metrics from the NuoDB
Collects host machine resource consumption statistics on a regular 10s interval.

# NuoDB Collector Page Outline
[Setup on Bare Metal](#Setup-on-bare-metal)
[Quick Start with Docker Compose](#Quick-start-with-docker-compose)

[Setup in Docker](#Setup-in-docker)
[Setup Manually in Docker](#Setup-in-docker)

[Setup in Kubernetes](#Setup-in-Kubernetes)

[Setup on Bare Metal](#Setup-on-bare-metal)

# Setup on bare metal
[Check Collection Status](#Check-collection-status)

## Installation
# Quick Start with Docker Compose

### 1) Install dependencies
```
pidof - installed - sysvinit-tools package
python2.7
pip
```
For a complete example on how to set up the NuoDB domain with NuoDB collector, you can use `docker compose`.
This repository contains a Docker Compose file (`docker-compose.yml`) which will start:

### 2) Download and install `telegraf`
These steps are for RedHat or CentOS. For other platforms, see [Telegraf Documentation](https://portal.influxdata.com/downloads/).
- 2 Admin Processes
- 1 Storage Manager
- 2 Transaction Engines
- 2 NuoDB Collector containers (1 for SM, 1 for TE)
- InfluxDB database

```
wget https://dl.influxdata.com/telegraf/releases/telegraf-1.15.2-1.x86_64.rpm
sudo yum localinstall telegraf-1.15.2-1.x86_64.rpm
```
Clone the NuoDB Collector repository and `cd` into it:

### 3) Download and install NuoDB Collector
```
git clone [email protected]:nuodb/nuodb-collector.git
cd nuodb-collector
pip install -r requirements.txt
sudo cp -r nuocd /opt/
```

## Configuration
The `conf/nuodb.conf` file in this repository configures all 4 input plugins for NuoDB running on localhost as described in the section above.
The `conf/outputs.conf` file configures an output plugin to a InfluxDB instance defined by the `$INFLUXURL` environmental variable.
Replace the `<hostinflux>` placeholder with the URL of a running InfluxDB instance.
```
sudo cp conf/nuodb.conf /etc/telegraf/telegraf.d
sudo cp conf/outputs.conf /etc/telegraf/telegraf.d
sudo edit /etc/telegraf/telegraf.conf
sudo chown -R telegraf.telegraf /etc/telegraf
sudo cat >> /etc/default/telegraf <<EOF
INFLUXURL=http://<hostinflux>:8086
PYTHONPATH=/opt/nuocd/pylib
EOF
```
Then run `docker-compose up` to start the processes specified in the Docker Compose file:

### Start NuoDB Collector
```
sudo systemctl daemon-reload
sudo systemctl restart telegraf
DOCKER_IMAGE=nuodb/nuodb-ce:latest docker-compose up -d
```

**NOTE:** If not starting telegraf via `systemd` then the variables set in `/etc/default/telegraf` are not picked up automatically.
Instead you can start telegraf with the following command:
Stop processes started with `docker-compose up` by running the following command:

```
sh -c "$(cat /etc/default/telegraf | tr '\n' ' ') telegraf --config /etc/telegraf/telegraf.conf --config-directory /etc/telegraf/telegraf.d"
DOCKER_IMAGE=nuodb/nuodb-ce:latest docker-compose down
```

## Setup in Docker
# Setup Manually in Docker

## Download the NuoDB Collector Docker Image

### Download the NuoDB Collector Docker Image
```
docker pull docker.pkg.github.com/nuodb/nuodb-collector/nuocd:latest
```

### Building docker image from source
## Building docker image from source (Optional)

Instead of pulling a pre-built Docker image, you can build it yourself from source.

```
git clone [email protected]:nuodb/nuodb-collector.git
cd nuodb-collector
docker build .
docker build .
docker tag <SHA> <TAG>
```

### Running Prerequisites
## Running Prerequisites

### NuoDB Domain

As a prerequisite you must have a running NuoDB domain.
To start NuoDB in Docker, follow the [NuoDB Docker Blog Part I](https://nuodb.com/blog/deploy-nuodb-database-docker-containers-part-i).
Following this tutorial will also create the Docker network `nuodb-net`.

After following the [NuoDB Docker Blog Part I](https://nuodb.com/blog/deploy-nuodb-database-docker-containers-part-i) tutorial, verify your domain by running `nuocmd`:

To verify your domain, run `nuocmd`:
```
$ docker exec -it nuoadmin1 nuocmd show domain
server version: 4.1.1-3-2203dab8dd, server license: Community
Expand All @@ -123,8 +111,11 @@ Databases:
[TE] test-te-1/172.20.0.6:48006 [start_id = 1] [server_id = nuoadmin1] [pid = 39] [node_id = 2] [last_ack = 0.14] MONITORED:RUNNING
```

This collector also assumes that an InfluxDB instance is already running.
If you do not already have InfluxDB running, you can start a simple install like so.
### InfluxDB

The collector assumes that an InfluxDB instance is running.
Start InfluxDB with Docker:

```
docker run -d --name influxdb \
--network nuodb-net \
Expand All @@ -133,15 +124,16 @@ docker run -d --name influxdb \
influxdb:latest
```

### Running NuoDB Collector
## Running NuoDB Collector

Each NuoDB Collector runs colocated with a NuoDB engine in the same process namespace.
As such, you must start a NuoDB collector docker container for every running NuoDB engine you want to monitor.

The following value replacement must be done to start a NuoDB Collector container:
- Replace the `<hostname>` with the hostname of the monitored engine container. The hostnames must match. In our example it will be `test-sm-1`
- Replace the `<hostinflux>` placeholder with the URL of a running InfluxDB container. In our example, it will be `influxdb`.
- Replace the `<nuoadmin>` placeholder with the URL of a running NuoDB admin container. In our example, it will be `nuoadmin1`.
- Replace the `<enginecontainer>` placeholder with the URL of a running NuoDB Engine container. In our example, it will be `test-sm-1`.
- Replace the `<hostname>` with the hostname of the monitored engine container. The hostnames must match. In our example it will be `test-sm-1`

```
docker run -d --name nuocd-sm \
Expand All @@ -155,19 +147,95 @@ docker run -d --name nuocd-sm \

Repeat the steps above for all running NuoDB engine containers you want to monitor.

### Docker Example
For a complete example on how to set up the NuoDB domain with NuoDB collector, you can use `docker compose`.
The following command will start:
- 2 Admin Processes
- 1 Storage Manager
- 2 Transaction Engines
- 2 NuoDB Collector containers (1 for SM, 1 for TE)
- InfluxDB time-series database
# Setup in Kubernetes

NuoDB Helm Chart support for the NuoDB Collector coming soon.

# Setup on Bare Metal

## Installation

### 1) Install dependencies

```
DOCKER_IMAGE=nuodb/nuodb-ce:latest docker-compose up -d
pidof
sysvinit-tools
python2.7
pip
```

### 2) Download and install `telegraf`

These steps are for RedHat or CentOS. For other platforms, see [Telegraf Documentation](https://portal.influxdata.com/downloads/).

```
wget https://dl.influxdata.com/telegraf/releases/telegraf-1.15.2-1.x86_64.rpm
sudo yum localinstall telegraf-1.15.2-1.x86_64.rpm
```

### 3) Download and install NuoDB Collector

```
git clone [email protected]:nuodb/nuodb-collector.git
cd nuodb-collector
pip install -r requirements.txt
sudo cp -r nuocd /opt/
```

## Configuration

The `conf/nuodb.conf` file in this repository configures all 4 input plugins for NuoDB running on localhost as described in the section above.
The `conf/outputs.conf` file configures an output plugin to a InfluxDB instance defined by the `$INFLUXURL` environment variable.
Replace the `<hostinflux>` placeholder with the hostname for a running InfluxDB instance.
```
sudo cp conf/nuodb.conf /etc/telegraf/telegraf.d
sudo cp conf/outputs.conf /etc/telegraf/telegraf.d
sudo edit /etc/telegraf/telegraf.conf
sudo chown -R telegraf.telegraf /etc/telegraf
cat <<EOF | sudo tee -a /etc/default/telegraf >/dev/null
INFLUXURL=http://<hostinflux>:8086
PYTHONPATH=/opt/nuocd/pylib
EOF
INFLUXURL=http://<hostinflux>:8086
PYTHONPATH=/opt/nuocd/pylib
EOF
Comment on lines +196 to +198
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these three lines are now duplicated. Is that on purpose?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, good catch. Must have made an error when cleaning the changes up. I'll push a fix now.

```

## Setup in Kubernetes
## Start NuoDB Collector

Follow the documentation in the [NuoDB Helm Charts](https://github.com/nuodb/nuodb-helm-charts) repository.
```
sudo systemctl daemon-reload
sudo systemctl restart telegraf
```

**NOTE:** If not starting telegraf via `systemd` then the variables set in `/etc/default/telegraf` are not picked up automatically.
Instead you can start telegraf with the following command:
```
sh -c "$(cat /etc/default/telegraf | tr '\n' ' ') telegraf --config /etc/telegraf/telegraf.conf --config-directory /etc/telegraf/telegraf.d"
```


# Check Collection Status

After starting collection, your InfluxDB instance should contain the `nuodb_internal` and `nuodb` databases.
You can check if `nuodb_internal` and `nuodb` exist by launching the InfluxDB CLI:

```
influx
```

Then, from the InfluxDB CLI, run the `SHOW DATABASES` command and check that `nuodb_internal` exists (it may take a minute for the database to be created):

```
$ influx
Connected to http://localhost:8086 version 1.8.3
InfluxDB shell version: 1.8.3
> show databases
name: databases
name
----
_internal
telegraf
nuodb_internal
nuodb
```
8 changes: 4 additions & 4 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
version: "3.7"
version: "3.3"
services:
nuoadmin1:
image: $DOCKER_IMAGE
Expand All @@ -11,15 +11,15 @@ services:
hostname: sm
environment:
NUOCMD_API_SERVER: https://nuoadmin1:8888
depends_on:
depends_on:
- nuoadmin1
command: ["nuodocker", "start", "sm", "--db-name", "hockey", "--server-id", "nuoadmin1", "--dba-user", "dba", "--dba-password", "goalie"]
te:
image: $DOCKER_IMAGE
hostname: te
environment:
NUOCMD_API_SERVER: https://nuoadmin1:8888
depends_on:
depends_on:
- nuoadmin1
- sm
command: ["nuodocker", "start", "te", "--db-name", "hockey", "--server-id", "nuoadmin1"]
Expand Down Expand Up @@ -50,4 +50,4 @@ services:
INFLUXURL: http://influxdb:8086
NUOCMD_API_SERVER: https://nuoadmin1:8888
NUOCD_HOSTNAME: te
pid: 'service:te'
pid: 'service:te'