Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

netty as an independent module, add new feature about collector list #1244

Merged
merged 48 commits into from
Sep 14, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
49dd4d4
add offline and online collector interface
Ceilzcx Aug 24, 2023
5340a37
update protoc class
tomsun28 Aug 24, 2023
077df18
Merge branch 'master' into issue_1194_collector
Ceilzcx Aug 25, 2023
542ab0b
add close collector interface
Ceilzcx Aug 25, 2023
55e3553
clear jobs when shutdown
tomsun28 Aug 27, 2023
94026c5
update netty to remoting module
Ceilzcx Aug 28, 2023
b8b23dc
update netty to remoting module
Ceilzcx Aug 28, 2023
20ca044
ManageServer field update to collectorAndJobScheduler
Ceilzcx Aug 29, 2023
f493520
impl netty client
Ceilzcx Aug 29, 2023
caace27
impl netty client
Ceilzcx Aug 29, 2023
e98a9d3
impl netty client
Ceilzcx Aug 29, 2023
1f3366a
impl netty client
Ceilzcx Aug 29, 2023
e742325
add direction field in cluster_msg.proto
tomsun28 Aug 30, 2023
7c47440
perfect remoting module
Ceilzcx Aug 30, 2023
73f6065
add remoting readme
Ceilzcx Aug 30, 2023
94775d8
ManageServer update to use RemotingServer
Ceilzcx Sep 4, 2023
35c2799
fix test bug
Ceilzcx Sep 5, 2023
0b86383
CollectServer change to use NettyRemotingClient
Ceilzcx Sep 6, 2023
5a0ddea
Merge branch 'master' into issue_1194_collector
tomsun28 Sep 7, 2023
40c2b15
add license header
tomsun28 Sep 7, 2023
6064bc7
ignore some pmd
tomsun28 Sep 8, 2023
ed88304
init collector cluster ui
tomsun28 Sep 8, 2023
a4765fd
collector cluster ui
tomsun28 Sep 8, 2023
5b81767
update collector cluster ui
tomsun28 Sep 9, 2023
0207784
update collector cluster ui
tomsun28 Sep 9, 2023
6f6f4b6
Merge branch 'master' into issue_1194_collector
tomsun28 Sep 9, 2023
af4296b
delete collector
tomsun28 Sep 10, 2023
48ea46b
bugfix dispatch jobs error when delete collector
tomsun28 Sep 10, 2023
ccd889e
support delete collector
tomsun28 Sep 11, 2023
42e3e5e
refactor thread factory
tomsun28 Sep 11, 2023
c46cb75
reformat code style
tomsun28 Sep 11, 2023
87a1994
format code, and send offline, online response msg
Ceilzcx Sep 11, 2023
bbcbff7
use CommonConstants.SUCCESS_CODE
tomsun28 Sep 11, 2023
731794f
Merge branch 'master' into issue_1194_collector
tomsun28 Sep 12, 2023
6d7ff95
merge
tomsun28 Sep 12, 2023
5f4c288
Merge branch 'master' into issue_1194_collector
Ceilzcx Sep 12, 2023
789fe93
Merge remote-tracking branch 'origin/issue_1194_collector' into issue…
Ceilzcx Sep 12, 2023
d8fc36a
bugfix number range
tomsun28 Sep 12, 2023
1252ea9
fix unit test
tomsun28 Sep 12, 2023
64a0260
fix unit test
tomsun28 Sep 13, 2023
48ccedc
fix unit test
tomsun28 Sep 13, 2023
ec8d319
fix unit test
tomsun28 Sep 13, 2023
83153aa
fix unit test
tomsun28 Sep 13, 2023
b9fa371
fix unit test
tomsun28 Sep 13, 2023
566bdce
Merge branch 'master' into issue_1194_collector
tomsun28 Sep 13, 2023
59935b0
fix unit test
tomsun28 Sep 13, 2023
0912626
Merge remote-tracking branch 'origin/issue_1194_collector' into issue…
Ceilzcx Sep 14, 2023
f142c83
change CollectJobService and invoke collectServer sendMsg method
Ceilzcx Sep 14, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/maven-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,4 @@ jobs:
with:
java-version: 11
- name: Build with Maven
run: mvn -B package --file pom.xml
run: mvn clean -B package --file pom.xml
4 changes: 2 additions & 2 deletions alerter/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
### alerter
### Alerter

Processing data according to threshold rules to determine alarms, and alarm distribution functions.
根据告警规则配置信息,处理指标数据判断告警,告警分发。
根据告警规则配置信息,处理指标数据判断告警,告警分发。
5 changes: 5 additions & 0 deletions collector/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,11 @@
<groupId>org.dromara.hertzbeat</groupId>
<artifactId>hertzbeat-common</artifactId>
</dependency>
<!-- remoting -->
<dependency>
<groupId>org.dromara.hertzbeat</groupId>
<artifactId>hertzbeat-remoting</artifactId>
</dependency>
<!-- validation -->
<dependency>
<groupId>org.springframework.boot</groupId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import com.alibaba.fastjson.JSONObject;
import com.google.common.collect.Lists;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.collections.CollectionUtils;
import org.apache.commons.lang3.StringUtils;
Expand Down Expand Up @@ -77,14 +78,14 @@ public RocketmqSingleCollectImpl() {
Runtime runtime = Runtime.getRuntime();
int corePoolSize = Math.max(8, runtime.availableProcessors());
int maximumPoolSize = Math.max(16, runtime.availableProcessors());
ThreadFactory threadFactory = new ThreadFactory() {
private final AtomicLong threadIndex = new AtomicLong(0);

@Override
public Thread newThread(@NotNull Runnable r) {
return new Thread(r, "RocketMQCollectGroup_" + this.threadIndex.incrementAndGet());
}
};
ThreadFactory threadFactory = new ThreadFactoryBuilder()
.setUncaughtExceptionHandler((thread, throwable) -> {
log.error("RocketMQCollectGroup has uncaughtException.");
log.error(throwable.getMessage(), throwable);
})
.setDaemon(true)
.setNameFormat("rocketMQ-collector-%d")
.build();
this.executorService = new ThreadPoolExecutor(corePoolSize, maximumPoolSize, 60L, TimeUnit.SECONDS,
new LinkedBlockingQueue<>(5000), threadFactory, new ThreadPoolExecutor.DiscardOldestPolicy());
}
Expand Down Expand Up @@ -131,6 +132,7 @@ public String supportProtocol() {

/**
* 采集前置条件, 入参判断
*
* @param metrics 数据指标
*/
private void preCheck(Metrics metrics) {
Expand All @@ -144,6 +146,7 @@ private void preCheck(Metrics metrics) {

/**
* 创建DefaultMQAdminExt实体类; 这里有个小问题, 是否需要每次都重新创建
*
* @param metrics 数据指标
* @return DefaultMQAdminExt
*/
Expand All @@ -162,7 +165,8 @@ private DefaultMQAdminExt createMqAdminExt(Metrics metrics) {

/**
* 采集rocketmq数据
* @param mqAdminExt rocketmq提供的远程调用类
*
* @param mqAdminExt rocketmq提供的远程调用类
* @param rocketmqCollectData rocketmq数据采集类
* @throws Exception 远程调用异常
*/
Expand All @@ -174,7 +178,8 @@ private void collectData(DefaultMQAdminExt mqAdminExt, RocketmqCollectData rocke

/**
* 采集rocketmq的集群数据
* @param mqAdminExt rocketmq提供的远程调用类
*
* @param mqAdminExt rocketmq提供的远程调用类
* @param rocketmqCollectData rocketmq数据采集类
* @throws Exception 远程调用异常
*/
Expand Down Expand Up @@ -244,7 +249,8 @@ private void collectClusterData(DefaultMQAdminExt mqAdminExt, RocketmqCollectDat

/**
* 采集rocketmq的消费者数据
* @param mqAdminExt rocketmq提供的远程调用类
*
* @param mqAdminExt rocketmq提供的远程调用类
* @param rocketmqCollectData rocketmq数据采集类
* @throws Exception 远程调用异常
*/
Expand Down Expand Up @@ -273,8 +279,7 @@ private void collectConsumerData(DefaultMQAdminExt mqAdminExt, RocketmqCollectDa
ConsumeStats consumeStats = null;
try {
consumeStats = mqAdminExt.examineConsumeStats(consumerGroup);
}
catch (Exception e) {
} catch (Exception e) {
log.warn("examineConsumeStats exception to consumerGroup {}, response [{}]", consumerGroup, e.getMessage());
}
if (consumeStats != null) {
Expand All @@ -285,8 +290,7 @@ private void collectConsumerData(DefaultMQAdminExt mqAdminExt, RocketmqCollectDa
ConsumerConnection consumerConnection = null;
try {
consumerConnection = mqAdminExt.examineConsumerConnectionInfo(consumerGroup);
}
catch (Exception e) {
} catch (Exception e) {
log.warn("examineConsumeStats exception to consumerGroup {}, response [{}]", consumerGroup, e.getMessage());
}
if (consumerConnection != null) {
Expand All @@ -312,8 +316,7 @@ private void collectConsumerData(DefaultMQAdminExt mqAdminExt, RocketmqCollectDa
}

/**
*
* @param mqAdminExt rocketmq提供的远程调用类
* @param mqAdminExt rocketmq提供的远程调用类
* @param rocketmqCollectData rocketmq数据采集类
* @throws Exception 远程调用异常
*/
Expand Down Expand Up @@ -343,10 +346,11 @@ private void collectTopicData(DefaultMQAdminExt mqAdminExt, RocketmqCollectData

/**
* 采集数据填充到builder
*
* @param rocketmqCollectData rocketmq数据采集类
* @param builder metrics data builder
* @param aliasFields 字段别名
* @param parseScript JSON的base path
* @param builder metrics data builder
* @param aliasFields 字段别名
* @param parseScript JSON的base path
*/
private void fillBuilder(RocketmqCollectData rocketmqCollectData, CollectRep.MetricsData.Builder builder, List<String> aliasFields, String parseScript) {
String dataJson = JSONObject.toJSONString(rocketmqCollectData);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,6 @@
/**
* Collection data scheduler interface
* 采集数据调度器接口
*
* @author tomsun28
*
*/
public interface CollectDataDispatch {

Expand Down
Loading
Loading