storage: Support vector index and ANN hint

[Lloyd-Pottiger]

What problem does this PR solve?

Issue Number: ref #9032

Problem Summary:

What is changed and how it works?

Pick https://github.com/tidbcloud/tiflash-cse/pull/156, https://github.com/tidbcloud/tiflash-cse/pull/162, https://github.com/tidbcloud/tiflash-cse/pull/163, https://github.com/tidbcloud/tiflash-cse/pull/164

Changes:

• Support one or no vector index on the vector column
• Enable ExtendColumnStat in DMFile meta for storing the meta of vector index
• Write: When vector index exist, DMFileWriter::addStreams will generate the index while writing down a new DMFile
  • class VectorIndex generates the index by HNSW algorithm
• Read: DMFileBlockInputStreamBuilder will try to generate a DMFileWithVectorIndexBlockInputStream. If vector index is unavailable, the read fallback to normal read and filter. In DMFileWithVectorIndexBlockInputStream, it
  • load vector index and apply ANN vector search algorithm, this will generate a rowid bitmap for later reading
  • load the vector column data (from vector index) and other column data from disk

TODO:

• Support building vector index in background pool
• Support building multiple vector index on the same vector column
• Support adding/dropping vector index following TiDB DDL
• Confirm the following logic
  

  tiflash/dbms/src/Storages/DeltaMerge/File/DMFileBlockInputStream.cpp

  Lines 145 to 147 in 9fab7b2

  	// Fast Scan and Clean Read does not affect our behavior. (TODO: Confirm?)
  	// if (is_fast_scan || enable_del_clean_read || enable_handle_clean_read)
  	// return fallback();

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

[Lloyd-Pottiger]

/build

[Lloyd-Pottiger]

@JaySon-Huang Please confirm those changes.

[JaySon-Huang]

Confirm. It is OK because we need to bump the STORAGE_FORMAT_CURRENT in later PR

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: breezewish, JaySon-Huang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [JaySon-Huang,breezewish]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-08-07 06:57:58.525180002 +0000 UTC m=+421608.392279075: ☑️ agreed by breezewish.
• 2024-08-12 03:40:53.066618827 +0000 UTC m=+152937.770088471: ☑️ agreed by JaySon-Huang.