[Feature Request]: Reduce expense of calculating quantiles

[scwhittle]

What would you like to happen?

The algorigthm used by ApproximateQuantiles ends up calculating all percentiles (ie 1-100) even if you just want 99%. The combiner state is quite large, by default ~35K if using doubles, which can add a lot of overhead if it is modified by a few elements at a time.

Some possible improvements:

• add compression and diff-based encoding for the buffers in ApproximateQuantiles
• investigate other streaming algorithms to calculate quantiles, such as t-digests which may be more space efficient

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner