[Enhancement](merge-on-write) add correctness check for the calculation of delete bitmap

[bobhan1]

Proposed changes

Currently, for merge-on-write unique table, the delete bitmap of a rowset will be calculated during flush phase, commit phase and publish phase. In this PR, we add a special mark in every rowset considered when we calculate delete bitmap in these three phases. Before we finally merge the delete bitmap to the table meta's delete bitmap, we will check if all the rowsets contain the special mark to check if we have considered all the rowsets during the above three phases.
Because the executor can not fail in publish phase if the coordinator have received successful commits info from all the executors, we just print logs if this correctness check failed rather than report a failure.

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

[bobhan1]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.24 seconds
stream load tsv: 505 seconds loaded 74807831229 Bytes, about 141 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17164434966 Bytes

[bobhan1]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.87 seconds
stream load tsv: 506 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 30.0 seconds inserted 10000000 Rows, about 333K ops/s
storage size: 17164169142 Bytes

[bobhan1]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[bobhan1]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[bobhan1]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.73 seconds
stream load tsv: 501 seconds loaded 74807831229 Bytes, about 142 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17167839689 Bytes

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.78 seconds
stream load tsv: 504 seconds loaded 74807831229 Bytes, about 141 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17163042439 Bytes

[bobhan1]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.77 seconds
stream load tsv: 505 seconds loaded 74807831229 Bytes, about 141 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17169261124 Bytes

[bobhan1]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[liaoxin01]

LGTM

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[bobhan1]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.99 seconds
stream load tsv: 513 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162282284 Bytes

[zhannngchen]

LGTM

[zhannngchen]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[bobhan1]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.46 seconds
stream load tsv: 513 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17161955253 Bytes

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.65 seconds
stream load tsv: 512 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162510251 Bytes

[bobhan1]

run arm

[zhannngchen]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.