Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Shade] Add seatunnel hadoop3 uber #3755

Merged
merged 13 commits into from
Dec 22, 2022
Merged

[Feature][Shade] Add seatunnel hadoop3 uber #3755

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
c9f185a
improve config file
EricJoy2048 Dec 19, 2022
5924665
remove copy engine yaml config file from engine module
EricJoy2048 Dec 19, 2022
b2b1c90
add seatunnel hadoop3 uber jar
EricJoy2048 Dec 19, 2022
a58f6e3
add license header
EricJoy2048 Dec 19, 2022
7ca66d4
add seatunnel hadoop shade uber jar
EricJoy2048 Dec 19, 2022
2288f94
merge from dev
EricJoy2048 Dec 19, 2022
c71f153
fix seatunnel hadoop shaded uber jar dependency
EricJoy2048 Dec 19, 2022
26d2dfc
improve version manager
EricJoy2048 Dec 19, 2022
8ce8a79
merge from dev
EricJoy2048 Dec 20, 2022
5bf995a
fix error
EricJoy2048 Dec 20, 2022
1695a28
bind guava version
CalvinKirs Dec 20, 2022
f20b434
fix check style
CalvinKirs Dec 20, 2022
784a33a
trigger
CalvinKirs Dec 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@
<module>seatunnel-engine</module>
<module>seatunnel-examples</module>
<module>seatunnel-e2e</module>
<module>seatunnel-shade</module>
</modules>

<profiles>
Expand Down Expand Up @@ -496,6 +497,14 @@
<scope>provided</scope>
</dependency>

<dependency>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-hadoop3-3.1.4-uber</artifactId>
<version>${project.version}</version>
<classifier>optional</classifier>
<scope>provided</scope>
</dependency>

</dependencies>

</dependencyManagement>
Expand Down
14 changes: 8 additions & 6 deletions seatunnel-connectors-v2/connector-file/connector-file-base/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,16 +34,16 @@
<orc.version>1.5.6</orc.version>
<commons.collecton4.version>4.4</commons.collecton4.version>
<commons.lang3.version>3.4</commons.lang3.version>
<flink.hadoop.version>2.7.5-7.0</flink.hadoop.version>
<parquet-avro.version>1.12.3</parquet-avro.version>
</properties>

<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
<version>${flink.hadoop.version}</version>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-hadoop3-3.1.4-uber</artifactId>
<version>${project.version}</version>
<classifier>optional</classifier>
<scope>provided</scope>
</dependency>
</dependencies>
Expand Down Expand Up @@ -117,8 +117,10 @@
</dependency>

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-hadoop3-3.1.4-uber</artifactId>
<version>${project.version}</version>
<classifier>optional</classifier>
<scope>provided</scope>
<exclusions>
<exclusion>
Expand Down
6 changes: 4 additions & 2 deletions seatunnel-connectors-v2/connector-file/connector-file-ftp/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,10 @@
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-hadoop3-3.1.4-uber</artifactId>
<version>${project.version}</version>
<classifier>optional</classifier>
<scope>provided</scope>
<exclusions>
<exclusion>
Expand Down
6 changes: 4 additions & 2 deletions seatunnel-connectors-v2/connector-file/connector-file-hadoop/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,10 @@
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-hadoop3-3.1.4-uber</artifactId>
<version>${project.version}</version>
<classifier>optional</classifier>
<scope>provided</scope>
<exclusions>
<exclusion>
Expand Down
6 changes: 4 additions & 2 deletions seatunnel-connectors-v2/connector-file/connector-file-local/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,10 @@
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-hadoop3-3.1.4-uber</artifactId>
<version>${project.version}</version>
<classifier>optional</classifier>
<scope>provided</scope>
<exclusions>
<exclusion>
Expand Down
6 changes: 4 additions & 2 deletions seatunnel-connectors-v2/connector-file/connector-file-s3/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,10 @@
</dependency>

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-hadoop3-3.1.4-uber</artifactId>
<version>${project.version}</version>
<classifier>optional</classifier>
<scope>provided</scope>
<exclusions>
<exclusion>
Expand Down
11 changes: 9 additions & 2 deletions seatunnel-connectors-v2/connector-file/connector-file-sftp/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,17 @@
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-hadoop3-3.1.4-uber</artifactId>
<version>${project.version}</version>
<classifier>optional</classifier>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.jcraft</groupId>
<artifactId>jsch</artifactId>
<version>0.1.55</version>
</dependency>
</dependencies>

</project>
10 changes: 6 additions & 4 deletions seatunnel-connectors-v2/connector-hive/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,14 +41,16 @@
<version>${project.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-hadoop3-3.1.4-uber</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-hadoop3-3.1.4-uber</artifactId>
<version>${project.version}</version>
<classifier>optional</classifier>
<scope>provided</scope>
<exclusions>
<exclusion>
Expand Down
1 change: 1 addition & 0 deletions seatunnel-core/seatunnel-flink-starter/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@
<exclude>org.apache.logging.log4j:log4j-slf4j-impl</exclude>
<exclude>org.apache.logging.log4j:log4j-1.2-api</exclude>
<exclude>org.apache.logging.log4j:log4j-to-slf4j</exclude>
<exclude>org.apache.seatunnel:seatunnel-hadoop3-3.1.4-uber</exclude>
</excludes>
</artifactSet>
</configuration>
Expand Down
1 change: 1 addition & 0 deletions seatunnel-core/seatunnel-spark-starter/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,7 @@
<exclude>org.apache.logging.log4j:log4j-slf4j-impl</exclude>
<!-- spark3.x use slf4j + log4j2.x -->
<exclude>org.apache.logging.log4j:log4j-to-slf4j</exclude>
<exclude>org.apache.seatunnel:seatunnel-hadoop3-3.1.4-uber</exclude>
</excludes>
</artifactSet>
</configuration>
Expand Down
1 change: 1 addition & 0 deletions seatunnel-core/seatunnel-starter/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@
<exclude>commons-logging:*</exclude>
<exclude>ch.qos.logback:*</exclude>
<exclude>org.apache.logging.log4j:log4j-to-slf4j</exclude>
<exclude>org.apache.seatunnel:seatunnel-hadoop3-3.1.4-uber</exclude>
</excludes>
</artifactSet>
</configuration>
Expand Down
29 changes: 28 additions & 1 deletion seatunnel-dist/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@
</property>
</activation>
<properties>
<mysql.version>8.0.16</mysql.version>
<mysql.version>8.0.27</mysql.version>
<postgresql.version>42.4.3</postgresql.version>
<dm-jdbc.version>8.1.2.141</dm-jdbc.version>
<sqlserver.version>9.2.1.jre8</sqlserver.version>
Expand All @@ -97,6 +97,12 @@
<teradata.version>17.20.00.12</teradata.version>
<redshift.version>2.1.0.9</redshift.version>
</properties>
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<dependencies>
<!-- starters -->
<dependency>
Expand Down Expand Up @@ -496,6 +502,27 @@
<scope>provided</scope>
</dependency>
<!-- jdbc driver end -->

<!-- hadoop jar -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version>3.1.4</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-bundle</artifactId>
<version>1.11.271</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-hadoop3-3.1.4-uber</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<!-- hadoop jar end -->
</dependencies>
</profile>
<profile>
Expand Down
5 changes: 4 additions & 1 deletion seatunnel-dist/src/main/assembly/assembly-bin-ci.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,13 +144,16 @@
<scope>provided</scope>
</dependencySet>

<!-- ============ SeaTunnel Transforms V2 Jars ============ -->
<!-- ============ SeaTunnel Transforms V2 Jars And SeaTunnel Hadoop3 Uber Jar ============ -->
<dependencySet>
<useProjectArtifact>false</useProjectArtifact>
<useTransitiveDependencies>true</useTransitiveDependencies>
<unpack>false</unpack>
<includes>
<include>org.apache.seatunnel:seatunnel-transforms-v2:jar</include>
<include>org.apache.hadoop:hadoop-aws:jar</include>
<include>com.amazonaws:aws-java-sdk-bundle:jar</include>
<include>org.apache.seatunnel:seatunnel-hadoop3-3.1.4-uber:jar</include>
</includes>
<outputFileNameMapping>${artifact.file.name}</outputFileNameMapping>
<outputDirectory>/lib</outputDirectory>
Expand Down
2 changes: 1 addition & 1 deletion seatunnel-dist/src/main/assembly/assembly-bin.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@
<scope>provided</scope>
</dependencySet>

<!-- ============ SeaTunnel Transforms V2 Jars ============ -->
<!-- ============ SeaTunnel Transforms V2 Jars And SeaTunnel Hadoop3 Uber Jar============ -->
<dependencySet>
<useProjectArtifact>false</useProjectArtifact>
<useTransitiveDependencies>true</useTransitiveDependencies>
Expand Down
6 changes: 4 additions & 2 deletions seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,10 @@
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-hadoop3-3.1.4-uber</artifactId>
<version>${project.version}</version>
<classifier>optional</classifier>
<scope>test</scope>
EricJoy2048 marked this conversation as resolved.
Show resolved Hide resolved
<exclusions>
<exclusion>
Expand Down
2 changes: 1 addition & 1 deletion ...l-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/resources/hazelcast.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,4 +30,4 @@ hazelcast:
properties:
hazelcast.invocation.max.retry.count: 20
hazelcast.tcp.join.port.try.count: 30
hazelcast.logging.type: log4j2
hazelcast.logging.type: log4j2
6 changes: 4 additions & 2 deletions seatunnel-e2e/seatunnel-flink-connector-v2-e2e/connector-iceberg-flink-e2e/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,10 @@
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-hadoop3-3.1.4-uber</artifactId>
<version>${project.version}</version>
<classifier>optional</classifier>
EricJoy2048 marked this conversation as resolved.
Show resolved Hide resolved
<scope>test</scope>
EricJoy2048 marked this conversation as resolved.
Show resolved Hide resolved
</dependency>
</dependencies>
Expand Down
6 changes: 4 additions & 2 deletions seatunnel-engine/seatunnel-engine-client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,10 @@
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-hadoop3-3.1.4-uber</artifactId>
<version>${project.version}</version>
<classifier>optional</classifier>
EricJoy2048 marked this conversation as resolved.
Show resolved Hide resolved
<scope>test</scope>
EricJoy2048 marked this conversation as resolved.
Show resolved Hide resolved
</dependency>

Expand Down
8 changes: 7 additions & 1 deletion seatunnel-engine/seatunnel-engine-server/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,13 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-hadoop3-3.1.4-uber</artifactId>
<version>${project.version}</version>
<classifier>optional</classifier>
EricJoy2048 marked this conversation as resolved.
Show resolved Hide resolved
<scope>test</scope>
EricJoy2048 marked this conversation as resolved.
Show resolved Hide resolved
</dependency>
</dependencies>

</project>
13 changes: 4 additions & 9 deletions ...ngine/seatunnel-engine-storage/checkpoint-storage-plugins/checkpoint-storage-hdfs/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,16 +31,11 @@
<artifactId>checkpoint-storage-hdfs</artifactId>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version>3.0.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>3.0.0</version>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-hadoop3-3.1.4-uber</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
<classifier>optional</classifier>
EricJoy2048 marked this conversation as resolved.
Show resolved Hide resolved
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
Expand Down
10 changes: 6 additions & 4 deletions seatunnel-engine/seatunnel-engine-storage/imap-storage-file/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,13 +37,15 @@
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-hadoop3-3.1.4-uber</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
<classifier>optional</classifier>
<exclusions>
<exclusion>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang</artifactId>
<artifactId>*</artifactId>
<groupId>*</groupId>
</exclusion>
</exclusions>
</dependency>
Expand Down
Loading