Skip to content

Commit

Permalink
Added documentation about batch hints
Browse files Browse the repository at this point in the history
  • Loading branch information
maheshwarip committed Jan 27, 2025
1 parent 4511e58 commit 4c940f7
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 2 deletions.
3 changes: 3 additions & 0 deletions src/content/docs/pipelines/configuration/batching.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,13 @@ There are three ways to define how requests are batched:
2. `batch-max-rows`: The maximum number of rows or events in a batch before data is written. Default, and maximum, is 10,000 rows.
3. `batch-max-seconds`: The maximum duration of a batch before data is written, in seconds. Default is 15 seconds, maximum is 600 seconds.

Pipelines batch definitions are hints. A pipeline will follow these hints closely, but batches will not be exact.

All three batch definitions work together. Whichever limit is reached first triggers the delivery of a batch.

For example, a `batch-max-mb` = 100 MB and a `batch-max-seconds` = 600 means that if 100 MB of events are posted to the Pipeline, the batch will be delivered. However, if it takes longer than 600 seconds for 100 MB of events to be posted, a batch of all the messages that were posted during those 600 seconds will be created and delivered.


## Batch settings

You can configure the following batch-level settings to adjust how Pipelines create a batch:
Expand Down
4 changes: 2 additions & 2 deletions src/content/docs/pipelines/reference/limits.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,5 @@ Many of these limits will increase during Pipelines' public beta period. [Follow
| Requests per second | 10,000 |
| Maximum payload per request | 1 MB |
| Maximum batch size | 100 MB |
| Maximum batch records | 10,000 |
| Maximum batch duration | 600s |
| Maximum batch records | 10,000,000 |
| Maximum batch duration | 300s |

0 comments on commit 4c940f7

Please sign in to comment.