v0.20.0: Batch account access!

Now you can manage common Batch.sh account tasks directly from Plumber!

Command	Description
plumber batch login	Login and save credentials to ~/.batchsh/config.json
plumber batch logout	Logout and clear saved credentials
plumber batch list collection	List collections
plumber batch create collection --name "My events" --schema-id "0e31f330-aca0-468c-a186-a26e889a19e9"	Create collection
plumber batch search --collection-id="..." --query="*"	Search a collection
plumber batch list schema	List schemas
plumber batch list replay	List replays
plumber batch list destination	List destinations
plumber batch create destination kafka	Create kafka destination
plumber batch create destination aws-sqs	Create SQS destination
plumber batch create destination rabbit	Create RabbitMQ destination
plumber batch create destination http	Create HTTP destination
plumber batch create replay	Create and start a new replay

v0.19.0

Minor relay fixes:

• Fix redis-streams stats display (wasn't increasing stats or muting on error)
• Fix bug where redis-streams relay incorrectly exits on error (should run forever and just log error)
• Kafka relay stats now get muted when read results in an error (+ retry in 5s)
• Add RetryReadInterval on error for gcp pubsub relay
• Add RetryReadInterval on error for aws-sqs relay

v0.18.0

• Added read, write and relay support for redis streams
• Renamed redis to redis-pubsub (to follow existing conventions such as aws-sqs and aws-sns)
• write redis-pubsub now writes to all specified channels (originally was just the first)
• use semantic versioning in --version .. turns out, having a short sha is not terribly intuitive 😄

Install

Download the appropriate release binary or install via brew:

brew tap batchcorp/public
brew install plumber

Redis relay

This release adds support for relaying Redis pubsub channels to Batch.

To run plumber relay with redis in Docker:

docker run -d --name plumber-redis -p 8080:8080 \
    -e PLUMBER_RELAY_REDIS_ADDRESS=localhost:6379 \
    -e PLUMBER_RELAY_REDIS_CHANNELS=channel1,channel2 \
    -e PLUMBER_RELAY_TYPE=redis \
    -e PLUMBER_RELAY_TOKEN=$YOUR-BATCHSH-TOKEN-HERE \
    batchcorp/plumber

Dockerhub: docker pull batchcorp/plumber:a934947

Relay mode fixes and cli flag improvements

Breaking changes:

• remove --output-type for reading; remove --input-type for writing. Both are unnecessary and the intent can be gathered from other flags.

Features:

• Kafka: Ability to specify an offset to begin reading from
• GCP: Environment variable PUBSUB_EMULATOR_HOST can now be specified to connect to a pubsub emulator instead of GCP
• Retries for GRPC calls in relay mode

Bug fixes:

• --queue-declare now actually declares the queue for rabbit
• Increasing max record size for relay payloads

Batching for all relays, stats, improved kafka

This is a fairly large (and awesome) release.

In this release:

• New --stats flag (and PLUMBER_STATS env var) which will cause plumber to periodically dump consume/produce speed stats.
• Lots of kafka improvements to improve read and relay throughput
  • Several new env vars that can be tweaked to further improve throughput and/or reliability
• Batching support for relay
• Pretty config printing - all relay modes will now print out the config they're using on console
  • plumber now has a lot of options - some messaging platforms (such as kafka) have 20+ flags/options - it is nice to be able to see what options are actually being used at runtime.

Enjoy!

Fixing quiet flag

v0.14.1

Fixing quiet flag

Support for relaying GCP PubSub messages to Batch collections

v0.14.0

Support for GCP PubSub relaying

Support for Azure relay, improved Kafka read and relay

1. Support for relaying Azure Service Bus to Batch.sh
2. Improved Kafka read and relay speeds; works with Confluent cloud!
   1. Kafka read and relay is highly configurable for improved throughput
3. Improved docs (including env var docs!)

Docker: https://hub.docker.com/r/batchcorp/plumber/

Updating GRPC token name

v0.12.1

Modifying grpc token name