Skip to content
This repository has been archived by the owner on Feb 17, 2018. It is now read-only.

Latest commit

 

History

History
50 lines (32 loc) · 1.63 KB

README.md

File metadata and controls

50 lines (32 loc) · 1.63 KB
Raw

Kafka File Tailer

This application tails a file into a Kafka Log.

I leveraged the following technologies for this:

  1. DropWizard - This is a great piece of software by Coda Hale that allows you to create self contained java applications. Your DevOps team will love you.
  2. Kafka - Developed by LinkedIn - this is a great disk persistent message bus
  3. Apache Commons IO - If you haven't worked with the Tailer component, this project will offer a simple example.
  4. MessagePack - Fast and effecient protocol buffer for objects.

Setting up the application

Since this project is built using Maven, run the following commands to build the jar:

mvn build

You will of course need to download the latest release of Apache Kafka to make this work. I prefer to use the binaries for ease of implementation.

When you download the binaries, you need to kick off Kafka and Zookeeper. For the sake of this application, use the default configurations in the config/ directory.

Start Zookeeper:

$ bin/zookeeper-server-start.sh config/zookeeper.properties

Start Kafka:

$ bin/kafka-server-start.sh config/server.properties

To Load files, run the load-data command:

$ java ... com.twodotsolutions.kafka.KafkaApplication load-data -l sample.log conf/config.yaml

To process files, run the read-data command:

$ java ... com.twodotsolutions.kafka.KafkaApplication read-data conf/config.yaml