[Feature] Support Cluster Snapshot Backup: SQL Interface and meta data (part 1) #54447

srlch · 2024-12-27T03:31:10Z

What I'm doing:

This is part 1 for support backup cluster snapshot:

Introduce SQL Interface for TURN ON and TRUN OFF the automated cluster snapshot task:

ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON
[STORAGE VOLUME <storage_volume_name>]

ADMIN SET AUTOMATED CLUSTER SNAPSHOT OFF

Fixes #53867
#53867

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

starrocks-cr · 2024-12-27T03:31:43Z

fe/fe-core/src/main/java/com/starrocks/lake/snapshot/ClusterSnapshotJob.java

+    public void write(DataOutput out) throws IOException {
+        Text.writeString(out, GsonUtils.GSON.toJson(this));
+    }
+}


The most risky bug in this code is:
Typo in the method call createAutomatedSnaphot, which should be createAutomatedSnapshot.

You can modify the code like this:

private void createSnapshotIfJobIsFinished() { GlobalStateMgr.getCurrentState().getClusterSnapshotMgr().createAutomatedSnapshot(this); }

starrocks-cr · 2024-12-27T03:31:45Z

fe/fe-core/src/main/java/com/starrocks/lake/snapshot/ClusterSnapshotMgr.java

+    @Override
+    public void gsonPostProcess() throws IOException {
+    }
+}


The most risky bug in this code is:
In the load method, the deserialized object data is not used to update the state of the current ClusterSnapshotMgr. This means that the data read from the SRMetaBlockReader, which presumably represents a saved state, does nothing to restore or set the internal state of the object using it.

You can modify the code like this:

public void load(SRMetaBlockReader reader) throws SRMetaBlockEOFException, IOException, SRMetaBlockException { ClusterSnapshotMgr data = reader.readJson(ClusterSnapshotMgr.class); this.automatedSnapshotSvName = data.automatedSnapshotSvName; this.automatedSnapshot = data.automatedSnapshot; this.historyAutomatedSnapshotJobs = data.historyAutomatedSnapshotJobs; this.remoteStorage = data.remoteStorage; this.locationWithServiceId = data.locationWithServiceId; }

starrocks-cr · 2024-12-27T03:31:45Z

fe/fe-core/src/main/java/com/starrocks/persist/ClusterSnapshotLog.java

+        String json = GsonUtils.GSON.toJson(this);
+        Text.writeString(out, json);
+    }
+}


The most risky bug in this code is:
The read method may fail if the input JSON does not match the expected structure or if required fields are missing or null.

You can modify the code like this:

public static ClusterSnapshotLog read(DataInput in) throws IOException { try { String json = Text.readString(in); ClusterSnapshotLog log = GsonUtils.GSON.fromJson(json, ClusterSnapshotLog.class); // Add checks to ensure required fields are present and valid if (log.type == null) { throw new IOException("ClusterSnapshotLog type cannot be null"); } return log; } catch (Exception e) { throw new IOException("Failed to read ClusterSnapshotLog from input", e); } }

xiangguangyxg · 2024-12-27T06:44:39Z

fe/fe-core/src/main/java/com/starrocks/sql/parser/StarRocks.g4

+adminSetAutomatedSnapshotStatement
+    : ADMIN SET AUTOMATED CLUSTER SNAPSHOT (ON (STORAGE VOLUME svName=identifier)? | OFF)
+    ;


Suggested change

adminSetAutomatedSnapshotStatement

: ADMIN SET AUTOMATED CLUSTER SNAPSHOT (ON (STORAGE VOLUME svName=identifier)? | OFF)

;

adminSetAutomatedSnapshotOnStatement

: ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON (STORAGE VOLUME svName=identifier)?

;

adminSetAutomatedSnapshotOffStatement

: ADMIN SET AUTOMATED CLUSTER SNAPSHOT OFF

;

xiangguangyxg · 2024-12-27T07:37:43Z

fe/fe-core/src/main/java/com/starrocks/lake/snapshot/ClusterSnapshotMgr.java

+
+    public ClusterSnapshotMgr() {}
+
+    public void addAutomatedSnapshotRequest(String storageVolumeName, boolean isReplay) {


Suggested change

public void addAutomatedSnapshotRequest(String storageVolumeName, boolean isReplay) {

// Turn on automated snapshot, use stmt for extension in future

public void setAutomatedSnapshotOn(AdminSetAutomatedSnapshotOnStmt stmt) {

xiangguangyxg · 2024-12-27T07:38:45Z

fe/fe-core/src/main/java/com/starrocks/lake/snapshot/ClusterSnapshotMgr.java

+        return getAutomatedSnapshot() != null;
+    }
+
+    public boolean containAutomatedSnapshotRequest() {


Suggested change

public boolean containAutomatedSnapshotRequest() {

public boolean isAutomatedSnapshotOn() {

xiangguangyxg · 2024-12-27T07:40:38Z

fe/fe-core/src/main/java/com/starrocks/lake/snapshot/ClusterSnapshotMgr.java

+        return snapshot.getSnapshotName();
+    }
+
+    public void dropAutomatedSnapshotRequest(boolean isReplay) {


Suggested change

public void dropAutomatedSnapshotRequest(boolean isReplay) {

// Turn off automated snapshot, use stmt for extension in future

public void setAutomatedSnapshotOff(AdminSetAutomatedSnapshotOffStmt stmt) {

xiangguangyxg · 2024-12-27T07:41:47Z

fe/fe-core/src/main/java/com/starrocks/lake/snapshot/ClusterSnapshotMgr.java

+    @Override
+    public void write(DataOutput out) throws IOException {
+        Text.writeString(out, GsonUtils.GSON.toJson(this));
+    }


xiangguangyxg · 2024-12-27T07:45:04Z

fe/fe-core/src/main/java/com/starrocks/lake/snapshot/ClusterSnapshotMgr.java

+    @SerializedName(value = "automatedSnapshot")
+    private ClusterSnapshot automatedSnapshot = null;
+    @SerializedName(value = "historyAutomatedSnapshotJobs")
+    private List<ClusterSnapshotJob> historyAutomatedSnapshotJobs = Lists.newArrayList();


not thread-safe ?

xiangguangyxg · 2024-12-30T06:50:42Z

fe/fe-core/src/main/java/com/starrocks/lake/snapshot/ClusterSnapshotMgr.java

+    public void setAutomatedSnapshotOn(AdminSetAutomatedSnapshotStmt stmt) {
+        setAutomatedSnapshotOn(stmt, false);
+    }
+
+    protected void setAutomatedSnapshotOn(AdminSetAutomatedSnapshotStmt stmt, boolean isReplay) {
+        String storageVolumeName = stmt.getStorageVolumeName();
+        if (!isReplay) {
+            ClusterSnapshotLog log = new ClusterSnapshotLog();
+            log.setCreateSnapshotNamePrefix(AUTOMATED_NAME_PREFIX, storageVolumeName);
+            GlobalStateMgr.getCurrentState().getEditLog().logClusterSnapshotLog(log);
+        }
+        automatedSnapshotSvName = storageVolumeName;
+    }


Suggested change

public void setAutomatedSnapshotOn(AdminSetAutomatedSnapshotStmt stmt) {

setAutomatedSnapshotOn(stmt, false);

}

protected void setAutomatedSnapshotOn(AdminSetAutomatedSnapshotStmt stmt, boolean isReplay) {

String storageVolumeName = stmt.getStorageVolumeName();

if (!isReplay) {

ClusterSnapshotLog log = new ClusterSnapshotLog();

log.setCreateSnapshotNamePrefix(AUTOMATED_NAME_PREFIX, storageVolumeName);

GlobalStateMgr.getCurrentState().getEditLog().logClusterSnapshotLog(log);

}

automatedSnapshotSvName = storageVolumeName;

}

public void setAutomatedSnapshotOn(AdminSetAutomatedSnapshotStmt stmt) {

setAutomatedSnapshotOn(stmt.getStorageVolumeName());

ClusterSnapshotLog log = new ClusterSnapshotLog();

log.setCreateSnapshotNamePrefix(AUTOMATED_NAME_PREFIX, storageVolumeName);

GlobalStateMgr.getCurrentState().getEditLog().logClusterSnapshotLog(log);

}

protected void setAutomatedSnapshotOn(String storageVolumeName) {

automatedSnapshotSvName = storageVolumeName;

}

xiangguangyxg · 2024-12-30T06:52:11Z

fe/fe-core/src/main/java/com/starrocks/persist/metablock/SRMetaBlockID.java

@@ -97,6 +97,8 @@ public int getId() {

    public static final SRMetaBlockID PIPE_MGR = new SRMetaBlockID(32);

+    public static final SRMetaBlockID CLOULD_NATIVE_SNAPSHOT_MGR = new SRMetaBlockID(33);


CLUSTER_SNAPSHOT_MGR

xiangguangyxg · 2024-12-30T06:52:35Z

fe/fe-core/src/main/java/com/starrocks/sql/ast/AdminSetAutomatedSnapshotStmt.java

+
+import com.starrocks.sql.parser.NodePosition;
+
+public class AdminSetAutomatedSnapshotStmt extends DdlStmt {


Suggested change

public class AdminSetAutomatedSnapshotStmt extends DdlStmt {

public class AdminSetAutomatedSnapshotOnStmt extends DdlStmt {

xiangguangyxg · 2024-12-30T06:52:58Z

fe/fe-core/src/main/java/com/starrocks/sql/ast/AdminSetOffAutomatedSnapshotStmt.java

+
+import com.starrocks.sql.parser.NodePosition;
+
+public class AdminSetOffAutomatedSnapshotStmt extends DdlStmt {


Suggested change

public class AdminSetOffAutomatedSnapshotStmt extends DdlStmt {

public class AdminSetAutomatedSnapshotOffStmt extends DdlStmt {

xiangguangyxg · 2024-12-30T06:53:18Z

fe/fe-core/src/main/java/com/starrocks/sql/parser/StarRocks.g4

@@ -734,6 +736,14 @@ syncStatement
    : SYNC
    ;

+adminSetAutomatedSnapshotStatement


Suggested change

adminSetAutomatedSnapshotStatement

adminSetAutomatedSnapshotOnStatement

xiangguangyxg · 2024-12-30T06:53:34Z

fe/fe-core/src/main/java/com/starrocks/sql/parser/StarRocks.g4

+    : ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON (STORAGE VOLUME svName=identifier)?
+    ;
+
+adminSetOffAutomatedSnapshotStatement


Suggested change

adminSetOffAutomatedSnapshotStatement

adminSetAutomatedSnapshotOffStatement

xiangguangyxg · 2024-12-30T06:57:00Z

fe/fe-core/src/main/java/com/starrocks/lake/snapshot/ClusterSnapshotMgr.java

+    public void setAutomatedSnapshotOff(AdminSetOffAutomatedSnapshotStmt stmt) {
+        setAutomatedSnapshotOff(stmt, false);
+    }
+
+    protected void setAutomatedSnapshotOff(AdminSetOffAutomatedSnapshotStmt stmt, boolean isReplay) {
+        if (!isReplay) {
+            ClusterSnapshotLog log = new ClusterSnapshotLog();
+            log.setDropSnapshot(AUTOMATED_NAME_PREFIX);
+            GlobalStateMgr.getCurrentState().getEditLog().logClusterSnapshotLog(log);
+        }
+
+        // drop AUTOMATED snapshot
+        automatedSnapshotSvName = "";
+    }


same as above

…a (part 1) Signed-off-by: srlch <[email protected]>

Signed-off-by: srlch <[email protected]>

sonarqubecloud · 2024-12-30T08:08:28Z

Quality Gate passed

Issues
35 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2024-12-30T09:38:16Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-12-30T09:38:54Z

[FE Incremental Coverage Report]

❌ fail : 79 / 115 (68.70%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/persist/EditLog.java	0	5	00.00%	[1114, 1115, 1116, 1967, 1968]
🔵	com/starrocks/sql/ast/AstVisitor.java	0	2	00.00%	[430, 434]
🔵	com/starrocks/journal/JournalEntity.java	0	2	00.00%	[780, 781]
🔵	com/starrocks/qe/DDLStmtExecutor.java	0	8	00.00%	[1194, 1195, 1196, 1197, 1203, 1204, 1205, 1206]
🔵	com/starrocks/server/GlobalStateMgr.java	1	3	33.33%	[1068, 1557]
🔵	com/starrocks/lake/snapshot/ClusterSnapshotMgr.java	29	38	76.32%	[59, 99, 105, 106, 107, 108, 112, 113, 117]
🔵	com/starrocks/persist/ClusterSnapshotLog.java	17	22	77.27%	[69, 70, 75, 76, 77]
🔵	com/starrocks/sql/analyzer/ClusterSnapshotAnalyzer.java	14	17	82.35%	[27, 45, 46]
🔵	com/starrocks/sql/ast/AdminSetAutomatedSnapshotOffStmt.java	5	5	100.00%	[]
🔵	com/starrocks/sql/analyzer/Analyzer.java	4	4	100.00%	[]
🔵	com/starrocks/persist/metablock/SRMetaBlockID.java	1	1	100.00%	[]
🔵	com/starrocks/sql/ast/AdminSetAutomatedSnapshotOnStmt.java	8	8	100.00%	[]

github-actions · 2024-12-30T09:42:56Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

srlch · 2024-12-30T11:17:40Z

https://github.com/Mergifyio backport branch-3.4

mergify · 2024-12-30T11:17:51Z

backport branch-3.4

✅ Backports have been created

#54503 [Feature] Support Cluster Snapshot Backup: SQL Interface and meta data (part 1) (backport #54447) has been created for branch branch-3.4 but encountered conflicts

…a (part 1) (#54447) Signed-off-by: srlch <[email protected]> (cherry picked from commit be70af0) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/qe/DDLStmtExecutor.java

…a (part 1) (backport #54447) (#54505) Signed-off-by: srlch <[email protected]>

…a (part 1) (StarRocks#54447) Signed-off-by: srlch <[email protected]>

srlch requested review from a team as code owners December 27, 2024 03:31

wanpengfei-git added the META-REVIEW label Dec 27, 2024

wanpengfei-git requested a review from a team December 27, 2024 03:31

starrocks-cr bot reviewed Dec 27, 2024

View reviewed changes

mergify bot assigned srlch Dec 27, 2024

starrocks-cr bot reviewed Dec 27, 2024

View reviewed changes

xiangguangyxg reviewed Dec 27, 2024

View reviewed changes

srlch force-pushed the cluster_snapshot_part_1 branch from 38fa362 to 0f400f3 Compare December 30, 2024 06:40

xiangguangyxg reviewed Dec 30, 2024

View reviewed changes

[Feature] Support Cluster Snapshot Backup: SQL Interface and meta dat…

94b2e67

…a (part 1) Signed-off-by: srlch <[email protected]>

srlch force-pushed the cluster_snapshot_part_1 branch from 0891fe8 to 3bea59e Compare December 30, 2024 07:51

fix

46d5c78

Signed-off-by: srlch <[email protected]>

srlch force-pushed the cluster_snapshot_part_1 branch from 3bea59e to 46d5c78 Compare December 30, 2024 08:00

gengjun-git approved these changes Dec 30, 2024

View reviewed changes

xiangguangyxg approved these changes Dec 30, 2024

View reviewed changes

kangkaisen approved these changes Dec 30, 2024

View reviewed changes

andyziye merged commit be70af0 into StarRocks:main Dec 30, 2024
53 of 54 checks passed

github-actions bot added the version:3.5 label Dec 30, 2024

wanpengfei-git mentioned this pull request Dec 30, 2024

shared-data mode support backup restore though automated snapshot #53867

Open

9 tasks

mergify bot mentioned this pull request Dec 30, 2024

[Feature] Support Cluster Snapshot Backup: SQL Interface and meta data (part 1) (backport #54447) #54503

Closed

18 tasks

wanpengfei-git pushed a commit that referenced this pull request Dec 31, 2024

[Feature] Support Cluster Snapshot Backup: SQL Interface and meta dat…

1b0a9a5

…a (part 1) (backport #54447) (#54505) Signed-off-by: srlch <[email protected]>

github-actions bot added the 3.4-merged label Dec 31, 2024

maggie-zhu pushed a commit to maggie-zhu/starrocks that referenced this pull request Jan 6, 2025

[Feature] Support Cluster Snapshot Backup: SQL Interface and meta dat…

7663d66

…a (part 1) (StarRocks#54447) Signed-off-by: srlch <[email protected]>


		public ClusterSnapshotMgr() {}

		public void addAutomatedSnapshotRequest(String storageVolumeName, boolean isReplay) {

	public void addAutomatedSnapshotRequest(String storageVolumeName, boolean isReplay) {
	// Turn on automated snapshot, use stmt for extension in future
	public void setAutomatedSnapshotOn(AdminSetAutomatedSnapshotOnStmt stmt) {

	public boolean containAutomatedSnapshotRequest() {
	public boolean isAutomatedSnapshotOn() {

	public void dropAutomatedSnapshotRequest(boolean isReplay) {
	// Turn off automated snapshot, use stmt for extension in future
	public void setAutomatedSnapshotOff(AdminSetAutomatedSnapshotOffStmt stmt) {

		@@ -97,6 +97,8 @@ public int getId() {

		public static final SRMetaBlockID PIPE_MGR = new SRMetaBlockID(32);

		public static final SRMetaBlockID CLOULD_NATIVE_SNAPSHOT_MGR = new SRMetaBlockID(33);


		import com.starrocks.sql.parser.NodePosition;

		public class AdminSetAutomatedSnapshotStmt extends DdlStmt {

	public class AdminSetAutomatedSnapshotStmt extends DdlStmt {
	public class AdminSetAutomatedSnapshotOnStmt extends DdlStmt {


		import com.starrocks.sql.parser.NodePosition;

		public class AdminSetOffAutomatedSnapshotStmt extends DdlStmt {

	public class AdminSetOffAutomatedSnapshotStmt extends DdlStmt {
	public class AdminSetAutomatedSnapshotOffStmt extends DdlStmt {

	adminSetAutomatedSnapshotStatement
	adminSetAutomatedSnapshotOnStatement

	adminSetOffAutomatedSnapshotStatement
	adminSetAutomatedSnapshotOffStatement

[Feature] Support Cluster Snapshot Backup: SQL Interface and meta data (part 1) #54447

[Feature] Support Cluster Snapshot Backup: SQL Interface and meta data (part 1) #54447

Conversation

srlch commented Dec 27, 2024 • edited Loading

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

starrocks-cr bot Dec 27, 2024

Choose a reason for hiding this comment

starrocks-cr bot Dec 27, 2024

Choose a reason for hiding this comment

starrocks-cr bot Dec 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Dec 30, 2024

Quality Gate passed

github-actions bot commented Dec 30, 2024

[Java-Extensions Incremental Coverage Report]

github-actions bot commented Dec 30, 2024

[FE Incremental Coverage Report]

file detail

github-actions bot commented Dec 30, 2024

[BE Incremental Coverage Report]

srlch commented Dec 30, 2024

mergify bot commented Dec 30, 2024 • edited Loading

✅ Backports have been created

srlch commented Dec 27, 2024 •

edited

Loading

mergify bot commented Dec 30, 2024 •

edited

Loading