Use larger docstore blocks in quickwit

[fulmicoton]

Tantivy's docstore blocks have been picked using the Lucene value.

Increasing this value should improve compression rate, espeically on logs.
Loki for instance, compresses 2x better (with snappy and larger blocks).

It should reduce the size of the hotcache a little.
On the other hand, it will impact the amount of data read and decompressed to fetch one doc.

In quickwit every fetch comes with a large latency which justifies larger blocks.

• study the size of the docstore on the hdfs dataset for different block sizes.
• make the block size configurable in tantivy.
• change the block size used in quickwit.
• (optional) consider other compression algorithm.
• (optional) be column oriented at the scale of a block? (would that help?)

[PSeitz]

block sizes and total doc store size

16KB

140M    total

160KB

136M    total

1,6MB

135M    total

[fulmicoton]

bummer. :p

What was the dataset?
Can you monkey write some code to test what happens if we are column oriented at the scale of a block?

[PSeitz]

The dataset was hdfs.

I can make a column oriented test. I think column oriented storage would be a good fit in combination with entropy encoding.

[PSeitz]

The comparison is on a quite hacky version (ignoring 1:n fields and merge segments)

Dataset: 1GB hdfs

160KB block-size row-store

9,9M    total

160KB block-size column-store

8,2M   total

[fulmicoton]

that's a bummer. >.<

My hunch was wrong then.
We need to go deeper. The original problem is that in my tests, we did not compress as well as loki or clickhouse. Can you broadly investigate what could improve compression of the docstore, and possibly check if my original observation (loki compresses better) was correct or not.

(Aggregation in quickwit is higher priority )

[fulmicoton]

quickwit-oss/tantivy#1374 suggests block size has a big influence.

[guilload]

We resumed this work in the last few weeks and did observe that increasing the block size improves the compression rate. After analysis, we settle on the following default compression parameters:

• compression algo: zstd
• compression level: 8
• input blocksize: 1M

This was implemented in #1646.

[guilload]

Closed by #1646