You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Given 10 million small messages, Persist processed them at around 2,000 messages a second. This is 1/5 the speed of receive. While Persist does not bottleneck the system, this does mean it will get backed up over time if it can't keep up with other parts of the system. This also could reflect on the number of presto workers etc
AC
Persist should be at least as fast as the next slowest part of the upstream system so that it does not back up over time
Tech Notes
This is probably more a presto tuning / adding presto workers more than a Persist thing
The text was updated successfully, but these errors were encountered:
And per brian this happens for every batch. So while we don't write an insert statement that inserts 1MB of rows, we write 1MB of rows, then run an insert into select from of the staging table, which practically means presto is hit almost as many times per my understanding.
Given 10 million small messages, Persist processed them at around 2,000 messages a second. This is 1/5 the speed of receive. While Persist does not bottleneck the system, this does mean it will get backed up over time if it can't keep up with other parts of the system. This also could reflect on the number of presto workers etc
AC
Tech Notes
The text was updated successfully, but these errors were encountered: