[Feature] Bypass sdk read and write decoupling #45918

Jay-ju · 2024-05-20T07:33:18Z

Why I'm doing:

What I'm doing:

You can learn more about the background from pr's #38466.

This PR is mainly used to solve the problem that when BE directly reads and writes SR file format on S3, the code cannot be encapsulated into SDK form. This PR is mainly to decouple some code implementation and provide some compilation scripts.

build format lib

BUILD_TYPE=Debug ./build.sh --format-lib  -j32

will install directory:

-- Installing: /data00/code/community/starrocks/be/output/format-lib/
-- Installing: /data00/code/community/starrocks/be/output/format-lib/libstarrocks_format.so
Skip Building Java Extensions
***************************************
Successfully build StarRocks  √ Format Lib ; StartTime:2024-05-20 09:52:29, EndTime:2024-05-20 09:58:21, TotalTime:352s

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

be/src/fs/fs_s3.cpp

build.sh

be/src/storage/lake/rowset.cpp

be/src/fs/CMakeLists.txt

be/src/io/s3_input_stream.cpp

tracymacding · 2024-06-27T21:59:41Z

be/src/runtime/descriptors.cpp

+            (*tbl)->_slot_with_column_name_map[tdesc.id] = slot_d;
+        }
+
+        if (!slot_d->col_name().empty()) {


duplicated？

tracymacding · 2024-06-27T22:06:58Z

be/src/runtime/fragment_mgr.cpp

@@ -873,7 +872,7 @@ Status FragmentMgr::exec_external_plan_fragment(const TScanOpenParams& params, c
            LOG(WARNING) << "tuple descriptor is null. id: " << slot_ref.tuple_id;
            return Status::InvalidArgument("tuple descriptor is null");
        }
-        auto* slot_desc = desc_tbl->get_slot_descriptor(slot_ref.slot_id);
+        auto* slot_desc = desc_tbl->get_slot_descriptor_with_column(slot_ref.slot_id);


why change to this new api?

when spark connector using filter push down, be can't recognize slot column name

tracymacding · 2024-06-27T22:10:31Z

be/src/starrocks_format/starrocks_lib.h

+namespace starrocks::lake {
+
+extern lake::TabletManager* _lake_tablet_manager;
+// extern lake::UpdateManager* _lake_update_manager;


remove this?

tracymacding · 2024-06-27T22:11:22Z

be/src/storage/CMakeLists.txt

+
+if (NOT BUILD_FORMAT_LIB)
+    list(APPEND STORAGE_FILES  lake/starlet_location_provider.cpp)
+    message(STATUS "include starlet_location_provider.cpp when build Storage.")


remove this?

tracymacding · 2024-06-27T22:15:00Z

be/src/storage/lake/tablet.cpp

@@ -103,7 +104,13 @@ const std::shared_ptr<const TabletSchema> Tablet::tablet_schema() const {
 }

 StatusOr<std::shared_ptr<const TabletSchema>> Tablet::get_schema() {
-    return _mgr->get_tablet_schema(_id, &_version_hint);
+    if (_tablet_schema) {


when will _table_schema not be nullptr?

here Tablet Construct Function has been modified

tracymacding · 2024-06-27T22:17:46Z

be/src/storage/lake/tablet_manager.cpp

@@ -239,11 +244,17 @@ StatusOr<TabletMetadataPtr> TabletManager::get_tablet_metadata(int64_t tablet_id
 }

 StatusOr<TabletMetadataPtr> TabletManager::get_tablet_metadata(const string& path, bool fill_cache) {
+    std::shared_ptr<FileSystem> fs;
+    return get_tablet_metadata(fs, path, fill_cache);


why pass a invalid fs pointer as a parameter?

Signed-off-by: jukejian <[email protected]> Signed-off-by: plotor <[email protected]>

sonarqubecloud · 2024-09-26T04:18:11Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-09-26T07:27:24Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-09-26T07:30:59Z

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-09-26T07:37:43Z

[BE Incremental Coverage Report]

✅ pass : 262 / 314 (83.44%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/runtime/fragment_mgr.cpp	0	1	00.00%	[878]
🔵	be/src/runtime/descriptors.cpp	2	6	33.33%	[820, 822, 824, 827]
🔵	be/src/storage/lake/general_tablet_writer.cpp	12	22	54.55%	[74, 98, 99, 266, 267, 268, 270, 272, 293, 294]
🔵	be/src/storage/lake/pk_tablet_writer.cpp	4	6	66.67%	[67, 68]
🔵	be/src/fs/fs_s3.cpp	43	61	70.49%	[138, 303, 379, 382, 383, 393, 403, 408, 415, 525, 526, 527, 528, 529, 530, 555, 556, 557]
🔵	be/src/fs/fs.cpp	4	5	80.00%	[93]
🔵	be/src/storage/protobuf_file.cpp	17	20	85.00%	[60, 77, 155]
🔵	be/src/storage/lake/rowset.cpp	33	38	86.84%	[177, 438, 460, 461, 472]
🔵	be/src/storage/lake/lake_delvec_loader.cpp	14	16	87.50%	[28, 31]
🔵	be/src/storage/lake/tablet_manager.cpp	15	17	88.24%	[72, 73]
🔵	be/src/storage/lake/tablet.cpp	12	13	92.31%	[113]
🔵	be/src/io/s3_input_stream.cpp	42	45	93.33%	[70, 75, 111]
🔵	be/src/types/timestamp_value.cpp	7	7	100.00%	[]
🔵	be/src/storage/lake/meta_file.cpp	9	9	100.00%	[]
🔵	be/src/storage/lake/lake_delvec_loader.h	5	5	100.00%	[]
🔵	be/src/storage/rowset/indexed_column_reader.cpp	2	2	100.00%	[]
🔵	be/src/runtime/exec_env.cpp	2	2	100.00%	[]
🔵	be/src/storage/rowset/scalar_column_iterator.cpp	3	3	100.00%	[]
🔵	be/src/storage/lake/horizontal_compaction_task.cpp	2	2	100.00%	[]
🔵	be/src/storage/lake/vertical_compaction_task.cpp	2	2	100.00%	[]
🔵	be/src/storage/lake/tablet_reader.cpp	2	2	100.00%	[]
🔵	be/src/storage/lake/update_manager.cpp	9	9	100.00%	[]
🔵	be/src/storage/lake/tablet_manager.h	1	1	100.00%	[]
🔵	be/src/storage/rowset/segment.cpp	1	1	100.00%	[]
🔵	be/src/storage/lake/lake_primary_key_compaction_conflict_resolver.h	1	1	100.00%	[]
🔵	be/src/storage/protobuf_file.h	1	1	100.00%	[]
🔵	be/src/storage/lake/tablet.h	4	4	100.00%	[]
🔵	be/src/fs/fs.h	2	2	100.00%	[]
🔵	be/src/storage/lake/lake_primary_key_compaction_conflict_resolver.cpp	3	3	100.00%	[]
🔵	be/src/fs/hdfs/fs_hdfs.cpp	2	2	100.00%	[]
🔵	be/src/io/s3_input_stream.h	6	6	100.00%	[]

Samrose-Ahmed · 2024-09-26T14:27:27Z

I am trying to understand this PR, it seems to aim to achieve being able to directly read and write SR files. But what is the 'format-lib' that is being compiled and built, can you explain?

kevincai · 2024-09-27T07:41:45Z

@Samrose-Ahmed yes, this PR will provide the ability to build the library format-lib, and then other application can leverage the library to interact with Frontend and has the ability to write data to remote storage directly without proxying through backend RPC.

Samrose-Ahmed · 2024-09-27T13:28:38Z

Awesome thanks.

wangsimo0 · 2024-10-09T11:58:04Z

ignore backport check: 3.3.5

introduced by StarRocks#45918 Signed-off-by: stdpain <[email protected]>

libhdfs depend on libjvm. introduced by #45918 Signed-off-by: stdpain <[email protected]>

libhdfs depend on libjvm. introduced by StarRocks#45918 Signed-off-by: stdpain <[email protected]>

Signed-off-by: jukejian <[email protected]> Signed-off-by: plotor <[email protected]> Signed-off-by: zhiminr.ren <[email protected]>

libhdfs depend on libjvm. introduced by StarRocks#45918 Signed-off-by: stdpain <[email protected]> Signed-off-by: zhiminr.ren <[email protected]>

Jay-ju requested review from a team as code owners May 20, 2024 07:33

starrocks-cr bot reviewed May 20, 2024

View reviewed changes

be/src/fs/fs_s3.cpp Show resolved Hide resolved

starrocks-cr bot reviewed May 20, 2024

View reviewed changes

build.sh Show resolved Hide resolved

starrocks-cr bot reviewed May 20, 2024

View reviewed changes

be/src/storage/lake/rowset.cpp Show resolved Hide resolved

mergify bot assigned Jay-ju May 20, 2024

Jay-ju force-pushed the bypass_sdk_read_and_write_decoupling branch 8 times, most recently from e0c2c85 to 61d5aad Compare May 27, 2024 02:09

wyb requested review from decster, kevincai and wyb June 19, 2024 05:10

Jay-ju force-pushed the bypass_sdk_read_and_write_decoupling branch 6 times, most recently from d74163f to 18ed0a7 Compare June 24, 2024 08:07

tracymacding reviewed Jun 27, 2024

View reviewed changes

plotor force-pushed the bypass_sdk_read_and_write_decoupling branch 5 times, most recently from c1215bf to cafa2b3 Compare September 25, 2024 10:01

decster approved these changes Sep 26, 2024

View reviewed changes

kevincai approved these changes Sep 26, 2024

View reviewed changes

plotor force-pushed the bypass_sdk_read_and_write_decoupling branch from cafa2b3 to 5fdd31b Compare September 26, 2024 04:09

plotor force-pushed the bypass_sdk_read_and_write_decoupling branch from 5fdd31b to 4097009 Compare September 26, 2024 04:10

wyb approved these changes Sep 26, 2024

View reviewed changes

trueeyu approved these changes Sep 26, 2024

View reviewed changes

trueeyu merged commit 3fbe518 into StarRocks:main Sep 26, 2024
48 of 49 checks passed

github-actions bot added the version:3.4 label Sep 26, 2024

wanpengfei-git mentioned this pull request Sep 26, 2024

Bypass read/write SR segment files in storage separation mode #47367

Open

1 task

stdpain added a commit to stdpain/starrocks-2 that referenced this pull request Oct 15, 2024

[BugFix] Fix ld link error in libhdfs

96336bd

introduced by StarRocks#45918 Signed-off-by: stdpain <[email protected]>

stdpain mentioned this pull request Oct 15, 2024

[BugFix] Fix ld link error in libhdfs #51913

Merged

24 tasks

alvin-celerdata pushed a commit that referenced this pull request Oct 15, 2024

[BugFix] Fix ld link error in libhdfs (#51913)

b03de73

libhdfs depend on libjvm. introduced by #45918 Signed-off-by: stdpain <[email protected]>

ZiheLiu pushed a commit to ZiheLiu/starrocks that referenced this pull request Oct 31, 2024

[BugFix] Fix ld link error in libhdfs (StarRocks#51913)

737ec01

libhdfs depend on libjvm. introduced by StarRocks#45918 Signed-off-by: stdpain <[email protected]>

plotor mentioned this pull request Nov 6, 2024

[Feature] Supports reading and writing data files bypass BE Server #52682

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Bypass sdk read and write decoupling #45918

[Feature] Bypass sdk read and write decoupling #45918

Jay-ju commented May 20, 2024 •

edited by wangsimo0

Loading

tracymacding Jun 27, 2024

Jay-ju Jun 28, 2024

tracymacding Jun 27, 2024

Jay-ju Jun 28, 2024

tracymacding Jun 27, 2024

tracymacding Jun 27, 2024

tracymacding Jun 27, 2024

Jay-ju Jun 28, 2024 •

edited

Loading

tracymacding Jun 27, 2024

Jay-ju Jun 28, 2024

sonarqubecloud bot commented Sep 26, 2024

github-actions bot commented Sep 26, 2024

github-actions bot commented Sep 26, 2024

github-actions bot commented Sep 26, 2024

Samrose-Ahmed commented Sep 26, 2024

kevincai commented Sep 27, 2024

Samrose-Ahmed commented Sep 27, 2024

wangsimo0 commented Oct 9, 2024

[Feature] Bypass sdk read and write decoupling #45918

[Feature] Bypass sdk read and write decoupling #45918

Conversation

Jay-ju commented May 20, 2024 • edited by wangsimo0 Loading

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jay-ju Jun 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Sep 26, 2024

Quality Gate passed

github-actions bot commented Sep 26, 2024

[Java-Extensions Incremental Coverage Report]

github-actions bot commented Sep 26, 2024

[FE Incremental Coverage Report]

github-actions bot commented Sep 26, 2024

[BE Incremental Coverage Report]

file detail

Samrose-Ahmed commented Sep 26, 2024

kevincai commented Sep 27, 2024

Samrose-Ahmed commented Sep 27, 2024

wangsimo0 commented Oct 9, 2024

Jay-ju commented May 20, 2024 •

edited by wangsimo0

Loading

Jay-ju Jun 28, 2024 •

edited

Loading