Scaling sinks based on partitions

[ppadovani]

So I'm still working through evaluating numaflow, and have an additional question related to my potential data flow. Consider multiple tenant data streams (100s) flowing into a single pipeline.

• Take the source URI and tenantid key a msg and forward to next step
• process URI and emit possibly 1000s of results per tenant
• for each tenant reduce across the emitted msgs
• send result to sink

There could be 10s or 100s of tenant data inflight at once.

Since reduce vertices do not autoscale, how would a pipeline that has these requirements scale? I have no wish to create a pipeline per tenant if at all possible.

[whynowy]

Reduce vertices can not autoscale at this moment, but you can specify partitions for each vertex, which reflects in a reduce vertex will be N number of pods.

[vigith]

e.g., https://github.com/numaproj/numaflow/blob/main/examples/6-reduce-fixed-window.yaml#L29

[ppadovani]

So, if I have to scale the reduce to 100 partitions, I'll end up with 100 pods even if they are idle. In other words, the reduce vertex pod will not handle more than a single partition of data. This reduces the ability to downscale during low traffic times and save $$.

Do I have that right?

[vigith]

that is correct, at the moment we cannot autoscale reduce.