Code

Transaction batching Kafka producer#735

Code

Merged

bryanlb merged 18 commits intomasterslackhq/astra:masterfrom bburkholder/exp-transactionsslackhq/astra:bburkholder/exp-transactionsCopy head branch name to clipboardJan 8, 2024

+1,065-572

Commits

Commits on Jan 2, 2024

• Add experimental Kafka batch producer

  bryanlb

  committed
  36d95cd

• Initial refactor of experimental batch producer

  bryanlb

  committed
  3119f04

• Cleanup & rename refactor

  bryanlb

  committed
  abdd9de

• Additional cleanup, documentation, error handling, and metrics added

  bryanlb

  committed
  5fa8cdf

• Rework exception handling to better catch errors

  bryanlb

  committed
  200af94

• Adjust producer linger/batch configs

  bryanlb

  committed
  ad0f220

• Add bulk ingest edge instrumentation

  bryanlb

  committed
  1d92605

• Add support for additional kafka props on bulk ingest

  bryanlb

  committed
  51379e9

• Remove default configs in favor of env settings

  bryanlb

  committed
  12db93c

• Add ability to override blocking task executor limits for bulk ingest

  bryanlb

  committed
  faf5c7f

Commits on Jan 3, 2024

• Add timer for bulk ingest

  bryanlb

  committed
  8b2f6d3

• Add a stall counter for bulk ingest

  bryanlb

  committed
  5129539

• Test pooling kafka producers

  bryanlb

  committed
  1d4aeef

Commits on Jan 4, 2024

• Increase default kafka producer pool, make configurable

  bryanlb

  committed
  412a86f

• Move bulk ingest to non-blocking virtual threads

  bryanlb

  committed
  e2c2727

Commits on Jan 5, 2024

• Remove pooled producer logic, unused metrics, early exit

  bryanlb

  committed
  b17b820

• Remove unused blocking config

  bryanlb

  committed
  e242efd

Commits on Jan 8, 2024

• PR feedback

  bryanlb

  committed
  16683f9