-
Notifications
You must be signed in to change notification settings - Fork 0
suryasev/SparkCarrot
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
Library for creating reliable Spark RabbitMQ (AMQP) receivers. Reliable receivers should be able to avoid data loss, as the signal to delete a message from the queue only happens after the data is stored. Simple example: implicit val ssc = new StreamingContext(sparkConfForStreaming, Seconds(10)) class LocalRMQConfig(queue: String) extends RMQConfig(queue, "localhost", 5672, "127.0.0.1") class StringDecoder extends Decoder[String] { override def fromBytes(bytes: Array[Byte]): String = new String(bytes) } val config = new LocalRMQConfig("funQueue") val stream : ReceiverInputDStream[String] = RabbitUtils.createStream[String, StringDecoder](ssc, config) TODOs: Create tests More detailed documentation/examples Break Avro decoders into separate library to eliminate avro dependency Bump spark/spark-streaming versions
About
RabbitMQ receiver for a Spark implemented in same way as Spark KafkaUtils
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published