Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(clustertest): add basic MiniCluster module for distributed system #2615

Merged
merged 51 commits into from
Oct 15, 2024
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
1c3794f
fix(server): Random generate default jwt secret key
HJ-Young Jul 2, 2024
1460469
fix(server): random generate default jwt secret key
HJ-Young Jul 5, 2024
9840c7c
Merge branch 'apache:master' into master
HJ-Young Jul 5, 2024
85c2b6d
Apply suggestions from code review
imbajin Jul 13, 2024
f3199fa
Merge branch 'apache:master' into master
HJ-Young Jul 19, 2024
d43da0b
Merge branch 'apache:master' into master
HJ-Young Aug 4, 2024
0d50758
feat(it): add basic miniCluster supports SimpleEnv
HJ-Young Aug 4, 2024
d7d3fc7
feat(it): add dist to generate configuration files
HJ-Young Aug 7, 2024
c21c936
chore(it): move the path in Env Object to ClusterConstant
HJ-Young Aug 8, 2024
ab08c68
chore(it): move the path in Env Object to ClusterConstant
HJ-Young Aug 8, 2024
1c44444
Merge branch 'apache:master' into master
HJ-Young Aug 8, 2024
efa6c9c
chore(it): move the path in Env Object to ClusterConstant
HJ-Young Aug 8, 2024
92a8001
Merge remote-tracking branch 'origin/master'
HJ-Young Aug 8, 2024
1ef2cd0
feat(it): add the config and implement of multiEnv
HJ-Young Aug 9, 2024
530f3ea
feat(it): amend the implement of reading configuration files
HJ-Young Aug 16, 2024
ce20f44
fix(it): fix the bug of coping wrong node dir
HJ-Young Aug 17, 2024
31c58f4
fix(it): add configuration files in dist
HJ-Young Aug 17, 2024
3476109
Merge branch 'apache:master' into master
HJ-Young Aug 17, 2024
9f02e27
fix(it): get hugegraph.properties through template file
HJ-Young Aug 17, 2024
1714213
fix(it): add 5s intervals between generation of pd nodes
HJ-Young Aug 18, 2024
16a4347
Merge branch 'apache:master' into master
HJ-Young Aug 18, 2024
7c57f67
Merge branch 'apache:master' into master
HJ-Young Sep 9, 2024
a23ab6f
feat(ct): add test and ci to CT module
HJ-Young Sep 9, 2024
deedb7d
feat(ct): add CI yml file
HJ-Young Sep 9, 2024
7275b67
fix(ct): add dependency to ct-test pom
HJ-Young Sep 9, 2024
4dfea30
fix(ct): delete unused pom dependencies
HJ-Young Sep 9, 2024
2d1896a
fix(ct): fix ci file
HJ-Young Sep 9, 2024
17f96a4
fix(ct): fix ci file
HJ-Young Sep 9, 2024
94a60ba
fix(ct): fix ci file
HJ-Young Sep 11, 2024
956226e
fix(ct): update dependency file
HJ-Young Sep 11, 2024
4fa78c2
fix(ct): add Multi cluster test
HJ-Young Sep 12, 2024
48cfc6e
chore(ct): reformat code & update config files
HJ-Young Sep 23, 2024
80835a3
chore(ct): reformat code style based on code review comment
HJ-Young Sep 25, 2024
1f593d8
fix(ct): resolve dependency problems & add exegesis in BaseTest class
HJ-Young Sep 25, 2024
2f769d7
Merge branch 'apache:master' into master
HJ-Young Sep 25, 2024
dd0bbd6
fix(ct): use different Logger to record log
HJ-Young Oct 6, 2024
6d391f6
Merge branch 'apache:master' into master
HJ-Young Oct 6, 2024
57d39ad
fix(ct): sync new commit
HJ-Young Oct 6, 2024
7ca4c8a
Apply suggestions from code review
HJ-Young Oct 8, 2024
69b7623
fix(ct): update code based on comments
HJ-Young Oct 8, 2024
359b832
fix(ct): fix pom file problem
HJ-Young Oct 9, 2024
82d3dd3
fix(ct): update pom.xml
HJ-Young Oct 9, 2024
981d572
fix code
imbajin Oct 9, 2024
ab3d76c
Apply suggestions from code review
HJ-Young Oct 10, 2024
436c2ac
fix(ct): fix code style problems
HJ-Young Oct 10, 2024
d916ad8
Merge branch 'apache:master' into master
HJ-Young Oct 10, 2024
2896c07
fix(ct): update hg-client version
HJ-Young Oct 13, 2024
584c79b
fix(ct): simplify the dist
HJ-Young Oct 14, 2024
cf6994d
fix(ct): simplify the dist
HJ-Young Oct 14, 2024
e677136
fix(ct): simplify the dist
HJ-Young Oct 14, 2024
b4f451d
fix(ct): fix server config file path
HJ-Young Oct 14, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions hugegraph-cluster-test/hg-ct-dist/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->

<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://maven.apache.org/POM/4.0.0"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.hugegraph</groupId>
<artifactId>hugegraph-cluster-test</artifactId>
<version>${revision}</version>
<relativePath>../pom.xml</relativePath>
</parent>

<artifactId>hg-ct-dist</artifactId>
HJ-Young marked this conversation as resolved.
Show resolved Hide resolved

<properties>
<dist.dir>${project.parent.basedir}</dist.dir>
<shell-executable>bash</shell-executable>
<assembly.dir>${project.basedir}/src/assembly</assembly.dir>
<assembly.descriptor.dir>${assembly.dir}/descriptor</assembly.descriptor.dir>
<assembly.static.dir>${assembly.dir}/static</assembly.static.dir>
<executable.jar.name>hg-ct</executable.jar.name>
</properties>

<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version>
<executions>
<execution>
<id>assembly-hugegraph-ct</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<attach>false</attach>
<appendAssemblyId>false</appendAssemblyId>
<outputDirectory>${dist.dir}</outputDirectory>
<descriptors>
<descriptor>
${assembly.descriptor.dir}/assembly.xml
</descriptor>
</descriptors>
<finalName>${final.name}</finalName>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

<dependencies>
<dependency>
<groupId>org.apache.hugegraph</groupId>
<artifactId>hg-ct-minicluster</artifactId>
<version>${revision}</version>
</dependency>
</dependencies>

</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->

<assembly>
<id>distribution</id>
<includeBaseDirectory>false</includeBaseDirectory>

<formats>
<format>dir</format>
</formats>

<fileSets>
<!-- <fileSet>-->
<!-- <directory>${assembly.static.dir}/bin</directory>-->
<!-- <outputDirectory>bin</outputDirectory>-->
<!-- <includes>-->
<!-- <include>*</include>-->
<!-- </includes>-->
<!-- <fileMode>755</fileMode>-->
<!-- </fileSet>-->
<fileSet>
<directory>${assembly.static.dir}/simple</directory>
<outputDirectory>simple</outputDirectory>
<includes>
<include>**/*</include>
</includes>
</fileSet>
<fileSet>
<directory>${assembly.static.dir}/multi</directory>
<outputDirectory>multi</outputDirectory>
<includes>
<include>**/*</include>
</includes>
</fileSet>
</fileSets>

<dependencySets>
<!-- code jars -->
<dependencySet>
<outputDirectory>/lib</outputDirectory>
<unpack>false</unpack>
<scope>runtime</scope>
<useProjectArtifact>false</useProjectArtifact>
<includes>
<include>org.apache.hugegraph:${executable.jar.name}:jar:*</include>
</includes>
</dependencySet>
</dependencySets>

</assembly>
77 changes: 77 additions & 0 deletions hugegraph-cluster-test/hg-ct-minicluster/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->

<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://maven.apache.org/POM/4.0.0"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>hg-ct-minicluster</artifactId>

<parent>
<groupId>org.apache.hugegraph</groupId>
<artifactId>hugegraph-cluster-test</artifactId>
<version>${revision}</version>
</parent>


<properties>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<log4j2.version>2.17.0</log4j2.version>
</properties>

<dependencies>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.13.0</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.12.0</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>2.0.9</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.24</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.freemarker</groupId>
<artifactId>freemarker</artifactId>
<version>2.3.33</version>
</dependency>
<dependency>
<groupId>org.yaml</groupId>
<artifactId>snakeyaml</artifactId>
<version>2.2</version>
<scope>compile</scope>
</dependency>
</dependencies>

</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.hugegraph.ct.base;

import java.io.File;

import org.apache.commons.lang3.SystemUtils;

public class ClusterConstant {

public static final String USER_DIR = "user.dir";
public static final String LOG = "logs";

public static final String LIB_DIR = "lib";
public static final String EXT_DIR = "ext";
public static final String PLUGINS_DIR = "plugins";
public static final String BIN_DIR = "bin";
public static final String DIST_DIR = "dist";
public static final String CONF_DIR = "conf";

public static final String PD_PACKAGE_PREFIX = "hugegraph-pd";
public static final String PD_JAR_PREFIX = "hg-pd-service";
public static final String STORE_PACKAGE_PREFIX = "hugegraph-store";
public static final String STORE_JAR_PREFIX = "hg-store-node";
public static final String SERVER_PACKAGE_PREFIX = "apache-hugegraph-incubating";
public static final String CT_PACKAGE_PREFIX = "apache-hugegraph-incubating-ct";

public static final String APPLICATION_FILE = "application.yml";
public static final String SERVER_PROPERTIES = "rest-server.properties";
public static final String HUGEGRAPH_PROPERTIES = "hugegraph.properties";
public static final String LOG4J_FILE = "log4j2.xml";
public static final String VERIFY_LICENSE_FILE = "verify-license.json";
public static final String LICENSE_FILE = "hugegraph.license";
public static final String PD_TEMPLATE_FILE = "pd-application.yml.template";
public static final String STORE_TEMPLATE_FILE = "store-application.yml.template";
public static final String SERVER_TEMPLATE_FILE = "rest-server.properties.template";

public static final String JAVA_CMD =
System.getProperty("java.home")
+ File.separator
+ BIN_DIR
+ File.separator
+ (SystemUtils.IS_OS_WINDOWS ? "java.exe" : "java");

public static final String PD_DIST_PATH =
System.getProperty(USER_DIR)
+ File.separator
+ "hugegraph-pd"
+ File.separator
+ DIST_DIR
+ File.separator;

public static final String PD_LIB_PATH =
getFileInDir(PD_DIST_PATH, PD_PACKAGE_PREFIX)
+ File.separator
+ LIB_DIR
+ File.separator;

public static final String STORE_DIST_PATH =
System.getProperty(USER_DIR)
+ File.separator
+ "hugegraph-store"
+ File.separator
+ DIST_DIR
+ File.separator;

public static final String STORE_LIB_PATH =
getFileInDir(STORE_DIST_PATH, STORE_PACKAGE_PREFIX)
+ File.separator
+ LIB_DIR
+ File.separator;

public static final String SERVER_DIST_PATH =
System.getProperty(USER_DIR)
+ File.separator
+ "hugegraph-server"
+ File.separator;

public static final String SERVER_LIB_PATH =
getFileInDir(SERVER_DIST_PATH, SERVER_PACKAGE_PREFIX)
+ File.separator;

public static final String CT_DIST_PATH =
System.getProperty(USER_DIR)
+ File.separator
+ "hugegraph-cluster-test"
+ File.separator;

public static final String CT_PACKAGE_PATH =
getFileInDir(CT_DIST_PATH, CT_PACKAGE_PREFIX)
+ File.separator;

public static final String IT_LOG_PATH =
CT_PACKAGE_PATH
+ LOG
+ File.separator;

private ClusterConstant() {
throw new IllegalStateException("Utility class");
}

public static String getFileInDir(String path, String fileName) {
File dir = new File(path);
if (dir.exists() && dir.isDirectory()) {
for (File file : dir.listFiles()) {
if (file.getName().startsWith(fileName) && !file.getName().endsWith(".gz")) {
return path + file.getName();
}
}
}
return "";
}

public static boolean isJava11OrHigher() {
String version = System.getProperty("java.version");
if (version.startsWith("1.")) {
version = version.substring(2, 3);
} else {
int dot = version.indexOf(".");
if (dot != -1) {
version = version.substring(0, dot);
}
}
int versionNumber = Integer.parseInt(version);
return versionNumber >= 11;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.hugegraph.ct.base;

public enum EnvType {
Simple,
MultiNode;

public static EnvType getSystemEnvType() {
String envType = System.getProperty("TestEnv", Simple.name());
return EnvType.valueOf(envType);
}
}
Loading
Loading