Webtrends to Kafka pipeline

This tutorial shows how to set up a very basic pipeline which simply reads content from a source and copies it into a destination (here: from webtrends socket to kafka topic).

To learn more about the configuration format, please read the associated documentation on that topic.

We assume that you have a standalone processing node up and running. If not, see the build and deployment instruction for guidance.

General settings

The pipeline set up in this use case is named webtrends-to-kafka.

Queues

As this pipeline serves as bridge between a source and a sink it has only two components configured which need one queue to exchange data. The queue used in this case is named webtrends-content according to its contents.

Components

The pipeline configures two components: a source and an emitter. The source component establishes a connection with the Webtrends streaming api using the spqr-webtrends source. All data is handed over to the Kafka emitter which exports the contents using the spqr-kafka emitter.

Deployment

To put the pipeline to life, we assume that you have a Kafka instance available somewhere and that a processing node is running in standalone mode (to keep things simple).

As the processing node comes with a set of pre-installed components, all you have to do, is POST the configuration shown below to processing node running at port 7070 (localhost): (configuration is assumed to live inside file webtrends-to-kafka.json)

curl -H "Content-Type: application/json" -X POST -d @webtrends-to-kafka.json http://localhost:7070/pipelines/

The response should look like the following and the Kafka topic should receive incoming data:

{
  state: "OK"
  msg: ""
  pid: "webtrends-to-kafka"
}

Configuration

{
  "id" : "webtrends-to-kafka",
  "queues" : [ {
    "id" : "webtrends-content",
    "queueSettings" : null
  } ],
  "components" : [ {
    "id" : "webtrends-stream-reader",
    "type" : "SOURCE",
    "name" : "webtrendsSource",
    "version" : "0.0.1",
    "settings" : {
      "webtrends.stream.version" : "2.1",
      "webtrends.auth.audience" : "auth.webtrends.com",
      "webtrends.stream.type" : "return_all",
      "webtrends.schema.version" : "2.1",
      "webtrends.auth.scope" : "sapi.webtrends.com",
      "webtrends.stream.url" : "ws://sapi.webtrends.com/streaming",
      "webtrends.auth.url" : "https://sauth.webtrends.com/v1/token",
      "webtrends.client.id" : "<your_client_id>",
      "webtrends.client.secret" : "<your_client_secret>",
      "webtrends.stream.query" : "select *"
    },
    "fromQueue" : "",
    "toQueue" : "webtrends-content"
  }, {
    "id" : "kafka-topic-emitter",
    "type" : "EMITTER",
    "name" : "kafkaEmitter",
    "version" : "0.0.1",
    "settings" : {
      "clientId" : "webtrendsToKafka",
      "topic" : "webtrends",
      "metadataBrokerList" : "localhost:9092",
      "zookeeperConnect" : "localhost:2181",
      "messageAcking" : "false",
      "charset" : "UTF-8"
    },
    "fromQueue" : "webtrends-content",
    "toQueue" : ""
  } ]
}