Skip to content

Commit

Permalink
reorg docs (KxSystems#98)
Browse files Browse the repository at this point in the history 
* Reformat, remove dups

* reorder sub reference
  • Loading branch information
@sshanks-kx
sshanks-kx authored Mar 12, 2023
1 parent 68f0239 commit 3da6115
Show file tree
Hide file tree
Showing 4 changed files with 676 additions and 696 deletions.
142 changes: 10 additions & 132 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,146 +16,24 @@ This interface is supported for the following platforms

Kdb+ is the world’s fastest time-series database, optimized for ingesting, analyzing and storing massive amounts of structured data. To get started with kdb+, see https://code.kx.com/q for downloads and developer information. For general information, visit https://kx.com/

## Documentation
## API Documentation

- [User guide](docs)
- [Reference](docs/reference.md)
:point_right: [`API reference`](docs/reference.md)

## Installation Documentation

## Running
:point_right: [`Install guide`](docs/install.md)

### Linux and macOS
## Example Setup

#### Step 1
:point_right: [`Example setup guide`](docs/example.md)

Build or install the latest version of `librdkafka`. The minimum required version is v0.11.0.
## Performance and Tuning

##### Install
:point_right:
[edenhill/librdkafka/wiki/How-to-decrease-message-latency](https://github.com/edenhill/librdkafka/wiki/How-to-decrease-message-latency)

_macOS_

```bash
brew install librdkafka
```

_Ubuntu/Debian (unstable)_

```bash
sudo apt-get install librdkafka-dev
```

_RHEL/CentOS_

```bash
sudo yum install librdkafka-devel
```

##### Build from source

Follow [requirements for `librdkafka` compilation](https://github.com/edenhill/librdkafka#requirements).

To build 32-bit versions on 64-bit OS you need 32-bit versions of libraries and a toolchain.

```bash
#CentOS/RHEL
sudo yum install glibc-devel.i686 libgcc.i686 libstdc++.i686 zlib-devel.i686
# Ubuntu
sudo apt-get install gcc-multilib
```

```bash
git clone https://github.com/edenhill/librdkafka.git
cd librdkafka
make clean # to make sure nothing left from previous build or if upgrading/rebuilding
# If using OpenSSL, remove --disable-ssl from configure command below
# On macOS with OpenSSL you might need to set `export OPENSSL_ROOT_DIR=/usr/local/Cellar/openssl/1.0.2k` before proceeding


# 32 bit
./configure --prefix=$HOME --disable-sasl --disable-lz4 --disable-ssl --mbits=32
# 64 bits
./configure --prefix=$HOME --disable-sasl --disable-lz4 --disable-ssl --mbits=64

make
make install
```


#### Step 2

Compile, install and move shared object file to appropriate location.

1. Make sure you have `QHOME` set as an environment variable.

2. If your librdkafka install is not located in `$HOME/include` as defaulted by the above instructions set `KAFKA_ROOT` environment variable to the appropriate include location.

3. Run the following set of commands to set up the system as outlined

```bash
// in kfk source folder
make
// move installed `.so` to `$QHOME/<arch>`
make install
// remove `.so` from kfk source folder
make clean
```

Note: If compiling dynamically linked `libkfk.so` make sure you have `librdkafka.so.1` in your `LD_LIBRARY_PATH`.

```bash
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/lib
```

### Windows

At present the Windows build of the `.dll` for the kafka interface has been tested on Visual Studio 2017 with `librdkafka.redist.1.0.0`. The following are the steps completed to achieve this

1. Install nuget for Windows: <https:/nuget.com/downloads>

2. Install the redistributed version of librdkafka which is suitable for use with windows, documentation can be found [here](https://www.nuget.org/packages/librdkafka.redist/1.0.0).

```bash
nuget install librdkafka.redist
```

3. Ensure you have Windows Visual Studio 2017 installed.

4. Place the installed librdkafka.redist into an appropriate location (suggestions `%HOME%` or `%QHOME%`).
For the remaining instructions, the path to the install is `C:/Users/jdoe/librdkafka.redist.1.0.0`

5. Update `LIB` user environment variable to include the following path to `rdkafka.h` for appropriate Windows architecture in this case

```bash
LIB = C:/Users/jdoe/librdkafka.redist.1.0.0/build/native/lib/win/x64/win-x64-Release/v120
```

6. Update `PATH` system environment variable to include path to native folder for the appropriate computer architecture

```bash
PATH = C:/Users/jdoe/librdkafka.redist.1.0.0/runtimes/win-x64/native
```

7. Create a user environment variable `KAFKA_NATIVE` in line with the following example path

```bash
KAFKA_NATIVE = C:/Users/jdoe/librdkafka.redist.1.0.0/build/native/include/
```

8. Clone the Kafka interface from the KxSystems GitHub repo

```bash
git clone https://github.com/kxsystems/kafka
```

9. Move to the `kafka/build/` folder within the GitHub clone and run the following;

```bash
call "build.bat"
```

10. If prompted for input, follow instructions accordingly.

11. Move the created `.dll` from the build folder to `%QHOME%/<arch>`
There are numerous configuration options and it is best to find settings that suit your needs and setup.

## Status

Expand Down
36 changes: 5 additions & 31 deletions docs/README.md → docs/example.md
View file
Original file line number Diff line number Diff line change
@@ -1,16 +1,10 @@
# ![Apache Kafka](kafka.png) Using Kafka with kdb+
# kdb+ Kafka Example

Follow the [installation instructions](install.md) for set-up.

To run examples on this page you will need a Kafka broker available.

`kfk` is a thin wrapper for kdb+ around the
`librdkafka` C API (available on [MacOS/Linux](https://github.com/edenhill/librdkafka) or [Windows](https://www.nuget.org/packages/librdkafka.redist/1.0.0)) for [Apache Kafka](https://kafka.apache.org/).

Follow the [installation instructions](../README.md#building-and-installation) for set-up.

To run examples on this page you will need a Kafka broker available. It is easy to [set up a local instance for testing](../README.md#setting-up-test-kafka-instance).


## API
## API Introduction

The library follows the `librdkafka` API closely where possible.
As per its [introduction](https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md):
Expand All @@ -21,7 +15,6 @@ As per its [introduction](https://github.com/edenhill/librdkafka/blob/master/INT
Both clients and topics accept an optional configuration dictionary.
`.kfk.Client` and `.kfk.Topic` return an int which acts as a Client or Topic ID (index into an internal array). Client IDs are used to create topics and Topic IDs are used to publish or subscribe to data on that topic. They can also be used to query metadata – state of subscription, pending queues, etc.


### Minimal producer example

```q
Expand All @@ -40,7 +33,6 @@ show "Published 1 message";
:point_right:
[KxSystems/kafka/examples/test_producer.q](https://github.com/KxSystems/kafka/blob/master/examples/test_producer.q)


### Minimal consumer example

```q
Expand All @@ -60,20 +52,10 @@ data:();
:point_right:
[KxSystems/kafka/examples/test_consumer.q](https://github.com/KxSystems/kafka/blob/master/examples/test_consumer.q) for a slightly more elaborate version


## Configuration

The library supports and uses all configuration options exposed by `librdkafka`, except callback functions, which are identical to Kafka options by design of `librdkafka`.

:point_right:
[edenhill/librdkafka/CONFIGURATION.md](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) for a list of options


## Testing
## Test Environment

One can use either existing Kafka broker or start a test Kafka broker as described below.


### Setting up a test Kafka instance

To start a Kafka instance for testing follow the instructions outlined in the link below
Expand All @@ -95,7 +77,6 @@ Start Kafka broker.
bin/kafka-server-start.sh config/server.properties
```


### Running examples

Start producer.
Expand Down Expand Up @@ -129,10 +110,3 @@ mtype topic client partition offset msgtime data ..

The messages will now flow from producer to consumer, the publishing rate can be adjusted via `\t x` in the producer process.


## Performance and tuning

:point_right:
[edenhill/librdkafka/wiki/How-to-decrease-message-latency](https://github.com/edenhill/librdkafka/wiki/How-to-decrease-message-latency)

There are numerous configuration options and it is best to find settings that suit your needs and setup. See [Configuration](#configuration) above.
137 changes: 137 additions & 0 deletions docs/install.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
# Kafka kdb+ library installation

## Build/Install

### Linux and macOS

#### Step 1

Build or install the latest version of `librdkafka`. The minimum required version is v0.11.0.

##### Install

_macOS_

```bash
brew install librdkafka
```

_Ubuntu/Debian (unstable)_

```bash
sudo apt-get install librdkafka-dev
```

_RHEL/CentOS_

```bash
sudo yum install librdkafka-devel
```

##### Build from source

Follow [requirements for `librdkafka` compilation](https://github.com/edenhill/librdkafka#requirements).

To build 32-bit versions on 64-bit OS you need 32-bit versions of libraries and a toolchain.

```bash
#CentOS/RHEL
sudo yum install glibc-devel.i686 libgcc.i686 libstdc++.i686 zlib-devel.i686
# Ubuntu
sudo apt-get install gcc-multilib
```

```bash
git clone https://github.com/edenhill/librdkafka.git
cd librdkafka
make clean # to make sure nothing left from previous build or if upgrading/rebuilding
# If using OpenSSL, remove --disable-ssl from configure command below
# On macOS with OpenSSL you might need to set `export OPENSSL_ROOT_DIR=/usr/local/Cellar/openssl/1.0.2k` before proceeding


# 32 bit
./configure --prefix=$HOME --disable-sasl --disable-lz4 --disable-ssl --mbits=32
# 64 bits
./configure --prefix=$HOME --disable-sasl --disable-lz4 --disable-ssl --mbits=64

make
make install
```


#### Step 2

Compile, install and move shared object file to appropriate location.

1. Make sure you have `QHOME` set as an environment variable.

2. If your librdkafka install is not located in `$HOME/include` as defaulted by the above instructions set `KAFKA_ROOT` environment variable to the appropriate include location.

3. Run the following set of commands to set up the system as outlined

```bash
// in kfk source folder
make
// move installed `.so` to `$QHOME/<arch>`
make install
// remove `.so` from kfk source folder
make clean
```

Note: If compiling dynamically linked `libkfk.so` make sure you have `librdkafka.so.1` in your `LD_LIBRARY_PATH`.

```bash
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/lib
```

### Windows

At present the Windows build of the `.dll` for the kafka interface has been tested on Visual Studio 2017 with `librdkafka.redist.1.0.0`. The following are the steps completed to achieve this

1. Install nuget for Windows: <https:/nuget.com/downloads>

2. Install the redistributed version of librdkafka which is suitable for use with windows, documentation can be found [here](https://www.nuget.org/packages/librdkafka.redist/1.0.0).

```bash
nuget install librdkafka.redist
```

3. Ensure you have Windows Visual Studio 2017 installed.

4. Place the installed librdkafka.redist into an appropriate location (suggestions `%HOME%` or `%QHOME%`).
For the remaining instructions, the path to the install is `C:/Users/jdoe/librdkafka.redist.1.0.0`

5. Update `LIB` user environment variable to include the following path to `rdkafka.h` for appropriate Windows architecture in this case

```bash
LIB = C:/Users/jdoe/librdkafka.redist.1.0.0/build/native/lib/win/x64/win-x64-Release/v120
```

6. Update `PATH` system environment variable to include path to native folder for the appropriate computer architecture

```bash
PATH = C:/Users/jdoe/librdkafka.redist.1.0.0/runtimes/win-x64/native
```

7. Create a user environment variable `KAFKA_NATIVE` in line with the following example path

```bash
KAFKA_NATIVE = C:/Users/jdoe/librdkafka.redist.1.0.0/build/native/include/
```

8. Clone the Kafka interface from the KxSystems GitHub repo

```bash
git clone https://github.com/kxsystems/kafka
```

9. Move to the `kafka/build/` folder within the GitHub clone and run the following;

```bash
call "build.bat"
```

10. If prompted for input, follow instructions accordingly.

11. Move the created `.dll` from the build folder to `%QHOME%/<arch>`

Loading

0 comments on commit 3da6115

Please sign in to comment.