Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][zeta]Support exposing monitoring metrics by prometheus exporter protocol #7564

Merged
merged 8 commits into from
Sep 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions config/seatunnel.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,6 @@ seatunnel:
namespace: /tmp/seatunnel/checkpoint_snapshot
storage.type: hdfs
fs.defaultFS: file:///tmp/ # Ensure that the directory has written permission
telemetry:
metric:
enabled: false
152 changes: 152 additions & 0 deletions docs/en/seatunnel-engine/telemetry.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions docs/en/seatunnel-engine/telemetry/grafana-dashboard.json

Large diffs are not rendered by default.

296 changes: 296 additions & 0 deletions docs/en/seatunnel-engine/telemetry/metrics.txt

Large diffs are not rendered by default.

295 changes: 295 additions & 0 deletions docs/en/seatunnel-engine/telemetry/openmetrics.txt

Large diffs are not rendered by default.

Binary file added docs/images/grafana.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 2 additions & 1 deletion docs/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,8 @@ const sidebars = {
"seatunnel-engine/tcp",
"seatunnel-engine/resource-isolation",
"seatunnel-engine/rest-api",
"seatunnel-engine/user-command"
"seatunnel-engine/user-command",
"seatunnel-engine/telemetry"
]
},
{
Expand Down
151 changes: 151 additions & 0 deletions docs/zh/seatunnel-engine/telemetry.md

Large diffs are not rendered by default.

Empty file.
296 changes: 296 additions & 0 deletions docs/zh/seatunnel-engine/telemetry/metrics.txt

Large diffs are not rendered by default.

295 changes: 295 additions & 0 deletions docs/zh/seatunnel-engine/telemetry/openmetrics.txt

Large diffs are not rendered by default.

22 changes: 22 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,9 @@
<e2e.dependency.skip>true</e2e.dependency.skip>
<skip.spotless>false</skip.spotless>

<!-- prometheus simpleclient -->
<prometheus.simpleclient.version>0.16.0</prometheus.simpleclient.version>

</properties>

<dependencyManagement>
Expand Down Expand Up @@ -541,6 +544,25 @@
<scope>test</scope>
</dependency>

<!-- The prometheus simpleclient -->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient</artifactId>
<version>${prometheus.simpleclient.version}</version>
</dependency>
<!-- Hotspot JVM metrics-->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_hotspot</artifactId>
<version>${prometheus.simpleclient.version}</version>
</dependency>
<!-- Exposition HTTPServer-->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_httpserver</artifactId>
<version>${prometheus.simpleclient.version}</version>
</dependency>

</dependencies>

<build>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,10 @@
import org.apache.seatunnel.engine.common.config.EngineConfig;
import org.apache.seatunnel.engine.common.config.SeaTunnelConfig;
import org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException;
import org.apache.seatunnel.engine.server.SeaTunnelNodeContext;
import org.apache.seatunnel.engine.server.SeaTunnelServerStarter;

import org.apache.commons.lang3.StringUtils;

import com.hazelcast.instance.impl.HazelcastInstanceFactory;

/** This command is used to execute the SeaTunnel engine job by SeaTunnel API. */
public class ServerExecuteCommand implements Command<ServerCommandArgs> {

Expand Down Expand Up @@ -59,9 +57,7 @@ public void execute() {
.setClusterRole(EngineConfig.ClusterRole.MASTER_AND_WORKER);
}

HazelcastInstanceFactory.newHazelcastInstance(
seaTunnelConfig.getHazelcastConfig(),
Thread.currentThread().getName(),
new SeaTunnelNodeContext(seaTunnelConfig));
SeaTunnelServerStarter.createHazelcastInstance(
seaTunnelConfig, Thread.currentThread().getName());
}
}
7 changes: 7 additions & 0 deletions seatunnel-dist/release-docs/LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,13 @@ The text of each license is the standard Apache 2.0 license.
(Apache-2.0) accessors-smart (net.minidev:accessors-smart:2.4.7 - https://mvnrepository.com/artifact/net.minidev/accessors-smart)
(Apache-2.0) json-smart (net.minidev:json-smart:2.4.7 - https://mvnrepository.com/artifact/net.minidev/json-smart)
(Apache-2.0) json-path (com.jayway.jsonpath:json-path:2.7.0 - https://mvnrepository.com/artifact/com.jayway.jsonpath/json-path)
(The Apache Software License, Version 2.0) Prometheus Java Simpleclient (io.prometheus:simpleclient:0.16.0 - https://mvnrepository.com/artifact/io.prometheus/simpleclient/0.16.0)
(The Apache Software License, Version 2.0) Prometheus Java Simpleclient Common (io.prometheus:simpleclient_common:0.16.0 - https://mvnrepository.com/artifact/io.prometheus/simpleclient_common/0.16.0)
(The Apache Software License, Version 2.0) Prometheus Java Simpleclient Hotspot (io.prometheus:simpleclient_hotspot:0.16.0 - https://mvnrepository.com/artifact/io.prometheus/simpleclient_hotspot/0.16.0)
(The Apache Software License, Version 2.0) Prometheus Java Simpleclient Httpserver (io.prometheus:simpleclient_httpserver:0.16.0 - https://mvnrepository.com/artifact/io.prometheus/simpleclient_httpserver/0.16.0)
(The Apache Software License, Version 2.0) Prometheus Java Span Context Supplier - Common (io.prometheus:simpleclient_tracer_common:0.16.0 - https://mvnrepository.com/artifact/io.prometheus/simpleclient_tracer_common/0.16.0)
(The Apache Software License, Version 2.0) Prometheus Java Span Context Supplier - OpenTelemetry (io.prometheus:simpleclient_tracer_otel:0.16.0 - https://mvnrepository.com/artifact/io.prometheus/simpleclient_tracer_otel/0.16.0)
(The Apache Software License, Version 2.0) Prometheus Java Span Context Supplier - OpenTelemetry Agent (io.prometheus:simpleclient_tracer_otel_agent:0.16.0 - https://mvnrepository.com/artifact/io.prometheus/simpleclient_tracer_otel_agent/0.16.0)

========================================================================
MOZILLA PUBLIC LICENSE License
Expand Down
17 changes: 17 additions & 0 deletions seatunnel-dist/release-docs/NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -806,4 +806,21 @@ LongAdder), which was released with the following comments:

=========================================================================

Prometheus NOTICE

=========================================================================
Prometheus instrumentation library for JVM applications
Copyright 2012-2015 The Prometheus Authors

This product includes software developed at
Boxever Ltd. (http://www.boxever.com/).

This product includes software developed at
SoundCloud Ltd. (http://soundcloud.com/).

This product includes software developed as part of the
Ocelli project by Netflix Inc. (https://github.com/Netflix/ocelli/).

=========================================================================


Loading
Loading