[refactor] Refactored into multiple modules to be compatible with different spark version features

[gnehil]

Proposed changes

Issue Number: close #xxx

Problem Summary:

In the past, all codes were in the same module. In order to be compatible with different Spark versions, many features available in higher versions were abandoned, which brought inconvenience to use.
So refactoring is imperative.
After reconstruction, it is divided into the following modules:

├── spark-doris-connector-base
├── spark-doris-connector-dist
├── spark-doris-connector-it
├── spark-doris-connector-shims
├── spark-doris-connector-spark-2
└── spark-doris-connector-spark-3

• spark-doris-connector-base
  This module contains the basic implementation of interaction with Doris, such as reading, writing and other API access.
  In this module, there are several important abstractions, DorisReader, DorisWriter and DorisCommitter

  • DorisReader
    This interface provides hashNext and next methods to determine whether the reading is finished and to obtain the next piece of data. Currently, two reading methods based on Thrift and Arrow Flight SQL are implemented.
  • DorisWriter
    This interface mainly provides the load and stop methods for writing a piece of data and stopping an import task.
  • DorisCommitter
    This interface mainly provides commit and abort methods for committing and rolling back transactions.
    Considering the consistency of writing and transaction processing, the write processor based on stream load or copy into implements both DorisWriter and DorisCommitter in the same class.

• spark-doris-connector-dist
  This module is used to package.

• spark-doris-connector-it
  This module is for end-to-end testing.

• spark-doris-connector-shims
  This module is a shim layer used to implement functions of different spark versions.
  Currently, the version split is only based on spark 3.

├── spark-doris-connector-shims_spark-3.0
├── spark-doris-connector-shims_spark-3.1
├── spark-doris-connector-shims_spark-3.2
├── spark-doris-connector-shims_spark-3.3
├── spark-doris-connector-shims_spark-3.4
├── spark-doris-connector-shims_spark-3.5
└── spark-doris-connector-shims_spark-3-base

For the future higher spark 4 version, you can also add modules here to implement the function of minor version differences.

• spark-doris-connector-spark-2
  This module is implemented based on spark 2.
• spark-doris-connector-spark-3
  This module is implemented based on spark 3. It relies on different shim layers for different minor versions, which provides better compatibility.

In addition, a new feature has been added: converting and reading Bitmap type data through Arrow Flight SQL
Example

val df = spark.read
.format("doris")
.option("doris.fenodes", "192.168.0.1:8030")
.option("doris.read.arrow-flight-sql.port", "11110")
.option("doris.table.identifier", "test.t_bitmap_read")
.option("user", "root")
.option("password", "")
.option("doris.read.mode", "arrow")
.option("doris.read.bitmap-to-string", "true")
.load

Result

+------+----------------------+
| k1   | v1                   |
+------+----------------------+
|    1 | 243                  |
|    2 | 1,2,3,4,5,434543     |
|    3 | 287667876573         |
+------+----------------------+

Convert a bitmap into a comma-delimited string containing all the set BIT bits. If the input is null, null will be returned.

Checklist(Required)

1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

[JNSimba]

Good Job, LGTM