Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Connector] Trino Pulsar Plugin #23566

Draft
wants to merge 127 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
127 commits
Select commit Hold shift + click to select a range
7ca60ce
Updates
eaba Sep 24, 2024
f1083c2
More Fixes
eaba Sep 24, 2024
c6e4273
More fixes
eaba Sep 25, 2024
82560f9
More fixes
eaba Sep 25, 2024
3572360
More fixes
eaba Sep 25, 2024
3a0efbe
Fixed scope `provided`
eaba Sep 25, 2024
b6c9898
More fixes
eaba Sep 26, 2024
9fd7468
fix
eaba Sep 26, 2024
f3e197a
updates
eaba Sep 28, 2024
c45b2bb
update
eaba Sep 28, 2024
096a1ea
fixes
eaba Sep 28, 2024
dddc458
fixes versions
eaba Sep 28, 2024
33ee3bb
Added `opentelemetry-api` provided
eaba Sep 28, 2024
b57e51a
Added `opentelemetry-context` provided
eaba Sep 28, 2024
6a8510b
Added `*` provided
eaba Sep 28, 2024
38c575c
Fix
eaba Sep 28, 2024
5d56380
fix
eaba Sep 28, 2024
b3238c1
fixed: `trino-record-decoder` must not have scope `provided`
eaba Sep 28, 2024
a71d64c
[Update][PULSAR][4.0.0-preview.1][managed-ledger] `PositionImpl` to `…
eaba Sep 28, 2024
e8040f5
Added `jackson-annotations` provided
eaba Sep 28, 2024
f89f377
Added `jol-core` provided
eaba Sep 28, 2024
a497096
update
eaba Sep 28, 2024
d3e1e85
org.junit.Assert
eaba Sep 29, 2024
b0aadda
fixes
eaba Sep 29, 2024
b2f8cd5
Fixes `LF`
eaba Sep 30, 2024
fb90bf5
Changes
eaba Sep 30, 2024
ca61c59
Removed `rewrite-maven-plugin`
eaba Sep 30, 2024
623cf6b
change `<version>460-SNAPSHOT</version>`
eaba Sep 30, 2024
992738d
change `<version>459-SNAPSHOT</version>`
eaba Sep 30, 2024
edad049
change `<version>458</version>`
eaba Sep 30, 2024
2ab373f
Reformat Code
eaba Sep 30, 2024
7bc6cb7
[FIX][TEST][PulsarServer] Checkstyle violations
eaba Oct 1, 2024
89cd7c0
[FIX][TEST][TestPulsarConnectorTest] Checkstyle violations
eaba Oct 1, 2024
e534a6c
[FIX][TEST][TestPulsarConnectorConfig] Checkstyle violations
eaba Oct 1, 2024
73f0b4d
[FIX][TEST][PulsarQueryRunner] Checkstyle violations
eaba Oct 1, 2024
3091405
[FIX][TEST][TestMsg] Checkstyle violations
eaba Oct 1, 2024
7379bad
Merge branch 'master' into trino-plugin-pulsar-sql
eaba Oct 2, 2024
4b12daa
fix
eaba Oct 2, 2024
1314a0a
[FIX][TEST][TestMsg] Checkstyle violations
eaba Oct 2, 2024
eb74557
[FIX][TEST][ProtobufNativeDecoderTestUtil] Checkstyle violations
eaba Oct 2, 2024
b13357c
[FIX][TEST][TestPrimitiveDecoder] Checkstyle violations
eaba Oct 2, 2024
349f998
[FIX][TEST][primitive.*, json.*] Checkstyle violations
eaba Oct 2, 2024
4c97885
[FIX][TEST][DecoderTestUtil] Checkstyle violations
eaba Oct 2, 2024
50f5c41
[FIX][TEST][DecoderTestMessage] Checkstyle violations
eaba Oct 2, 2024
d34eff8
[FIX][TEST][TestAvroDecoder] Checkstyle violations
eaba Oct 2, 2024
aa66e84
[FIX][TEST][AvroDecoderTestUtil] Checkstyle violations
eaba Oct 2, 2024
f38f80f
[FIX][TEST][AbstractDecoderTester] Checkstyle violations
eaba Oct 2, 2024
2f30237
Updates
eaba Sep 24, 2024
e754540
More Fixes
eaba Sep 24, 2024
d39fef9
More fixes
eaba Sep 25, 2024
d5c4b78
More fixes
eaba Sep 25, 2024
5693962
More fixes
eaba Sep 25, 2024
032cd9f
Fixed scope `provided`
eaba Sep 25, 2024
9316c70
More fixes
eaba Sep 26, 2024
0867349
fix
eaba Sep 26, 2024
30f9745
updates
eaba Sep 28, 2024
143702b
update
eaba Sep 28, 2024
1f7bb99
fixes
eaba Sep 28, 2024
5c416f6
fixes versions
eaba Sep 28, 2024
4b1c5f3
Added `opentelemetry-api` provided
eaba Sep 28, 2024
cb4b3f2
Added `opentelemetry-context` provided
eaba Sep 28, 2024
538f319
Added `*` provided
eaba Sep 28, 2024
eb81f41
Fix
eaba Sep 28, 2024
48cd89e
fix
eaba Sep 28, 2024
38bfaaa
fixed: `trino-record-decoder` must not have scope `provided`
eaba Sep 28, 2024
c6e8fbc
[Update][PULSAR][4.0.0-preview.1][managed-ledger] `PositionImpl` to `…
eaba Sep 28, 2024
ae1456b
Added `jackson-annotations` provided
eaba Sep 28, 2024
1add2e0
Added `jol-core` provided
eaba Sep 28, 2024
1f20f52
update
eaba Sep 28, 2024
4111741
org.junit.Assert
eaba Sep 29, 2024
6901be2
fixes
eaba Sep 29, 2024
25fbed3
Fixes `LF`
eaba Sep 30, 2024
1fe3c27
Changes
eaba Sep 30, 2024
ddbe782
Removed `rewrite-maven-plugin`
eaba Sep 30, 2024
06cd61d
change `<version>460-SNAPSHOT</version>`
eaba Sep 30, 2024
da37d21
change `<version>459-SNAPSHOT</version>`
eaba Sep 30, 2024
b1180f1
change `<version>458</version>`
eaba Sep 30, 2024
cd29b4e
Reformat Code
eaba Sep 30, 2024
4100463
[FIX][TEST][PulsarServer] Checkstyle violations
eaba Oct 1, 2024
05f64a7
[FIX][TEST][TestPulsarConnectorTest] Checkstyle violations
eaba Oct 1, 2024
1c1bdf7
[FIX][TEST][TestPulsarConnectorConfig] Checkstyle violations
eaba Oct 1, 2024
9e9c2c5
[FIX][TEST][PulsarQueryRunner] Checkstyle violations
eaba Oct 1, 2024
a5b4427
[FIX][TEST][TestMsg] Checkstyle violations
eaba Oct 1, 2024
d0e42ee
fix
eaba Oct 2, 2024
36aa193
[FIX][TEST][TestMsg] Checkstyle violations
eaba Oct 2, 2024
889be68
[FIX][TEST][ProtobufNativeDecoderTestUtil] Checkstyle violations
eaba Oct 2, 2024
71186c9
[FIX][TEST][TestPrimitiveDecoder] Checkstyle violations
eaba Oct 2, 2024
ffc062b
[FIX][TEST][primitive.*, json.*] Checkstyle violations
eaba Oct 2, 2024
bd295f4
[FIX][TEST][DecoderTestUtil] Checkstyle violations
eaba Oct 2, 2024
ddcff43
[FIX][TEST][DecoderTestMessage] Checkstyle violations
eaba Oct 2, 2024
685a902
[FIX][TEST][TestAvroDecoder] Checkstyle violations
eaba Oct 2, 2024
312b50e
[FIX][TEST][AvroDecoderTestUtil] Checkstyle violations
eaba Oct 2, 2024
dfd3dea
[FIX][TEST][AbstractDecoderTester] Checkstyle violations
eaba Oct 2, 2024
e10c6a6
Merge branch 'trino-plugin-pulsar-sql' of https://github.com/eaba/tri…
eaba Oct 4, 2024
5d1b0cf
[FIX][TEST][`OffloadPoliciesImpl`] Checkstyle violations
eaba Oct 4, 2024
75e8317
[FIX][MAIN][util.*] Checkstyle violations
eaba Oct 4, 2024
4e29893
[FIX][MAIN][`PulsarTopicDescription`, `PulsarTableHandle`, `PulsarSql…
eaba Oct 4, 2024
bbf8d36
[FIX][MAIN][`PulsarSplit`] Checkstyle violations
eaba Oct 4, 2024
d6a848b
[FIX][MAIN][`PulsarSchemaInfoProvider`] Checkstyle violations
eaba Oct 4, 2024
215289e
[FIX][MAIN][`PulsarRowDecoder*`] Checkstyle violations
eaba Oct 4, 2024
bcc7af4
[FIX][MAIN][`PulsarRecord`] Checkstyle violations
eaba Oct 4, 2024
80d2422
Added `ci pulsar`
eaba Oct 4, 2024
0c9c612
[FIX][MAIN][`PulsarReadOnlyCursor*`] Checkstyle violations
eaba Oct 5, 2024
96ce3d5
[FIX][MAIN][`PulsarPlugin`, `PulsarMetadata`] Checkstyle violations
eaba Oct 5, 2024
9cd8495
[FIX][MAIN][`PulsarInternalColumn`] Checkstyle violations
eaba Oct 5, 2024
f96a4ad
[FIX][MAIN][`PulsarFieldValueProviders`] Checkstyle violations
eaba Oct 5, 2024
7457610
[FIX][MAIN][`PulsarDispatchingRowDecoderFactory`] Checkstyle violations
eaba Oct 5, 2024
eba734e
[FIX][MAIN][`PulsarConnectorUtils`] Checkstyle violations
eaba Oct 5, 2024
7e7e75a
[FIX][MAIN][`PulsarConnectorModule`] Checkstyle violations
eaba Oct 5, 2024
bbadebb
[FIX][MAIN][`PulsarConnectorMetricsTracker`] Checkstyle violations
eaba Oct 5, 2024
2141f89
[FIX][MAIN][`PulsarConnector*`] Checkstyle violations
eaba Oct 5, 2024
0d86aca
[FIX][MAIN][`PulsarColumn*`] Checkstyle violations
eaba Oct 5, 2024
b853238
[FIX][MAIN][`PulsarAuth`] Checkstyle violations
eaba Oct 5, 2024
ad14d5d
[FIX][MAIN][`PulsarAdminClientProvider`] Checkstyle violations
eaba Oct 5, 2024
e84a0c2
[FIX][MAIN][`protobufnative.PulsarProtobufNative8`] Checkstyle violat…
eaba Oct 5, 2024
4bd765a
[FIX][MAIN][`primitive.PulsarPrimitiveRowDecoder*`] Checkstyle violat…
eaba Oct 5, 2024
456c94b
[FIX][MAIN][`json.PulsarJson*`] Checkstyle violations
eaba Oct 5, 2024
e0de330
[FIX][MAIN][`avro.PulsarAvro*`] Checkstyle violations
eaba Oct 5, 2024
7e1fae5
Fixes
eaba Oct 5, 2024
eafb1ed
Fixes
eaba Oct 9, 2024
cba7e22
Merge branch 'master' into trino-plugin-pulsar-sql
eaba Oct 9, 2024
f1f4886
[FIX] `com.google.common.base.Predicate` to Prefer `java.util.functio…
eaba Oct 10, 2024
5eb18d6
461-SNAPSHOT
eaba Oct 10, 2024
66a07b5
Fixed Test part 1
eaba Oct 10, 2024
ab87157
Removed `ObjectMapperFactory`
eaba Oct 11, 2024
c1d216c
Fixed `Non-Test`
eaba Oct 17, 2024
b871d2b
[FAILURE] io.trino.plugin.pulsar.TestPulsarConnectorTest
eaba Oct 17, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -370,6 +370,7 @@ jobs:
!:trino-phoenix5,
!:trino-pinot,
!:trino-postgresql,
!:trino-pulsar,
!:trino-redis,
!:trino-redshift,
!:trino-resource-group-managers,
Expand Down Expand Up @@ -475,6 +476,7 @@ jobs:
- { modules: plugin/trino-phoenix5 }
- { modules: plugin/trino-pinot }
- { modules: plugin/trino-postgresql }
- { modules: plugin/trino-pulsar }
- { modules: plugin/trino-redis }
- { modules: plugin/trino-redshift }
- { modules: plugin/trino-redshift, profile: cloud-tests }
Expand Down Expand Up @@ -902,6 +904,7 @@ jobs:
- suite-snowflake
- suite-hudi
- suite-ignite
- suite-pulsar
exclude:
- suite: suite-azure
ignore exclusion if: >-
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
"react-dom": "^18.3.1",
"react-router-dom": "^6.25.1",
"sass": "^1.77.8",
"webapp-preview": "file:",
"zustand": "^4.5.4"
},
"devDependencies": {
Expand Down
278 changes: 278 additions & 0 deletions plugin/trino-pulsar/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,278 @@
<!--

Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.

-->

![logo](https://pulsar.apache.org/img/pulsar.svg)

[![docker pull](https://img.shields.io/docker/pulls/apachepulsar/pulsar-all.svg)](https://hub.docker.com/r/apachepulsar/pulsar)
[![contributors](https://img.shields.io/github/contributors-anon/apache/pulsar)](https://github.com/apache/pulsar/graphs/contributors)
[![last commit](https://img.shields.io/github/last-commit/apache/pulsar)](https://github.com/apache/pulsar/commits/master)
[![release](https://img.shields.io/github/v/release/apache/pulsar?sort=semver)](https://pulsar.apache.org/download/)
[![downloads](https://img.shields.io/github/downloads/apache/pulsar/total)](https://pulsar.apache.org/download/)

Pulsar is a distributed pub-sub messaging platform with a very
flexible messaging model and an intuitive client API.

Learn more about Pulsar at https://pulsar.apache.org

## Main features

- Horizontally scalable (Millions of independent topics and millions
of messages published per second)
- Strong ordering and consistency guarantees
- Low latency durable storage
- Topic and queue semantics
- Load balancer
- Designed for being deployed as a hosted service:
- Multi-tenant
- Authentication
- Authorization
- Quotas
- Support mixing very different workloads
- Optional hardware isolation
- Keeps track of consumer cursor position
- REST API for provisioning, admin and stats
- Geo replication
- Transparent handling of partitioned topics
- Transparent batching of messages

## Repositories

This repository is the main repository of Apache Pulsar. Pulsar PMC also maintains other repositories for
components in the Pulsar ecosystem, including connectors, adapters, and other language clients.

- [Pulsar Core](https://github.com/apache/pulsar)

### Helm Chart

- [Pulsar Helm Chart](https://github.com/apache/pulsar-helm-chart)

### Ecosystem

- [Pulsar Adapters](https://github.com/apache/pulsar-adapters)

### Clients

- [.NET/C# Client](https://github.com/apache/pulsar-dotpulsar)
- [C++ Client](https://github.com/apache/pulsar-client-cpp)
- [Go Client](https://github.com/apache/pulsar-client-go)
- [NodeJS Client](https://github.com/apache/pulsar-client-node)
- [Python Client](https://github.com/apache/pulsar-client-python)
- [Reactive Java Client](https://github.com/apache/pulsar-client-reactive)

### Dashboard & Management Tools

- [Pulsar Manager](https://github.com/apache/pulsar-manager)

### Website

- [Pulsar Site](https://github.com/apache/pulsar-site)

### CI/CD

- [Pulsar CI](https://github.com/apache/pulsar-test-infra)

### Archived/Halted

- [Pulsar Connectors](https://github.com/apache/pulsar-connectors) (bundled as [pulsar-io](pulsar-io))
- [Pulsar Translation](https://github.com/apache/pulsar-translation)
- [Pulsar SQL (Pulsar Presto Connector)](https://github.com/apache/pulsar-presto) (bundled as [pulsar-sql](pulsar-sql))
- [Ruby Client](https://github.com/apache/pulsar-client-ruby)

## Pulsar Runtime Java Version Recommendation

- pulsar ver > 2.10 and master branch

| Components | Java Version |
|----------------|:-------------:|
| Broker | 17 |
| Functions / IO | 17 |
| CLI | 17 |
| Java Client | 8 or 11 or 17 |

- 2.8 <= pulsar ver <= 2.10

| Components | Java Version |
|----------------|:------------:|
| Broker | 11 |
| Functions / IO | 11 |
| CLI | 8 or 11 |
| Java Client | 8 or 11 |

- pulsar ver < 2.8

| Components | Java Version |
|------------|:------------:|
| All | 8 or 11 |

## Build Pulsar

### Requirements

- JDK

| Pulsar Version | JDK Version |
|-------------------|:-------------------------------------------------:|
| master and 2.11 + | [JDK 17](https://adoptium.net/?variant=openjdk17) |
| 2.8 / 2.9 / 2.10 | [JDK 11](https://adoptium.net/?variant=openjdk11) |
| 2.7 - | [JDK 8](https://adoptium.net/?variant=openjdk8) |

- Maven 3.6.1+
- zip

> **Note**:
>
> This project includes a [Maven Wrapper](https://maven.apache.org/wrapper/) that can be used instead of a
> system-installed Maven.
> Use it by replacing `mvn` by `./mvnw` on Linux and `mvnw.cmd` on Windows in the commands below.
>
> It's better to use CMD rather than Powershell on Windows. Because maven will activate the `windows` profile which runs
`rename-netty-native-libs.cmd`.

### Build

Compile and install:

```bash
$ mvn install -DskipTests
```

Compile and install individual module

```bash
$ mvn -pl module-name (e.g: pulsar-broker) install -DskipTests
```

### Minimal build (This skips most of external connectors and tiered storage handlers)

```bash
mvn install -Pcore-modules,-main -DskipTests
```

Run Unit Tests:

```bash
$ mvn test
```

Run Individual Unit Test:

```bash
$ mvn -pl module-name (e.g: pulsar-client) test -Dtest=unit-test-name (e.g: ConsumerBuilderImplTest)
```

Run Selected Test packages:

```bash
$ mvn test -pl module-name (for example, pulsar-broker) -Dinclude=org/apache/pulsar/**/*.java
```

Start standalone Pulsar service:

```bash
$ bin/pulsar standalone
```

Check https://pulsar.apache.org for documentation and examples.

## Build custom docker images

* Docker images must be built with Java 8 for `branch-2.7` or previous branches because
of [ISSUE-8445](https://github.com/apache/pulsar/issues/8445).
* Java 11 is the recommended JDK version in `branch-2.8`, `branch-2.9` and `branch-2.10`.
* Java 17 is the recommended JDK version in `master`.

The following command builds the docker images `apachepulsar/pulsar-all:latest` and `apachepulsar/pulsar:latest`:

```bash
mvn clean install -DskipTests
mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests
```

After the images are built, they can be tagged and pushed to your custom repository. Here's an example of a bash script
that tags the docker images with the current version and git revision and pushes them to `localhost:32000/apachepulsar`.

```bash
image_repo_and_project=localhost:32000/apachepulsar
pulsar_version=$(mvn initialize help:evaluate -Dexpression=project.version -pl . -q -DforceStdout)
gitrev=$(git rev-parse HEAD | colrm 10)
tag="${pulsar_version}-${gitrev}"
echo "Using tag $tag"
docker tag apachepulsar/pulsar-all:latest ${image_repo_and_project}/pulsar-all:$tag
docker push ${image_repo_and_project}/pulsar-all:$tag
docker tag apachepulsar/pulsar:latest ${image_repo_and_project}/pulsar:$tag
docker push ${image_repo_and_project}/pulsar:$tag
```

## Setting up your IDE

Read https://pulsar.apache.org/contribute/setup-ide for setting up IntelliJ IDEA or Eclipse for developing Pulsar.

## Documentation

> **Note**:
>
> For how to make contributions to Pulsar documentation,
> see [Pulsar Documentation Contribution Guide](https://pulsar.apache.org/contribute/document-intro/).

## Contact

##### Mailing lists

| Name | Scope | Subscribe | Unsubscribe | Archives |
|:----------------------------------------------------------|:--------------------------------|:------------------------------------------------------|:----------------------------------------------------------|:-------------------------------------------------------------------|
| [[email protected]](mailto:[email protected]) | User-related discussions | [Subscribe](mailto:[email protected]) | [Unsubscribe](mailto:[email protected]) | [Archives](http://mail-archives.apache.org/mod_mbox/pulsar-users/) |
| [[email protected]](mailto:[email protected]) | Development-related discussions | [Subscribe](mailto:[email protected]) | [Unsubscribe](mailto:[email protected]) | [Archives](http://mail-archives.apache.org/mod_mbox/pulsar-dev/) |

##### Slack

Pulsar slack channel at https://apache-pulsar.slack.com/

You can self-register at https://communityinviter.com/apps/apache-pulsar/apache-pulsar

##### Report a security vulnerability

To report a vulnerability for Pulsar, contact the [Apache Security Team](https://www.apache.org/security/). When
reporting a vulnerability to [[email protected]](mailto:[email protected]), you can copy your email
to [[email protected]](mailto:[email protected]) to send your report to the Apache Pulsar Project
Management Committee. This is a private mailing list.

https://github.com/apache/pulsar/security/policy contains more details.

## License

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

## Crypto Notice

This distribution includes cryptographic software. The country in which you currently reside may have restrictions on
the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption
software, please check your country's laws, regulations and policies concerning the import, possession, or use, and
re-export of encryption software, to see if this is permitted.
See [The Wassenaar Arrangement](http://www.wassenaar.org/) for more information.

The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as
Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing
cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution
makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the
BIS Export Administration Regulations, Section 740.13) for both object code and source code.

The following provides more details on the included cryptographic software: Pulsar uses the SSL library from Bouncy
Castle written by http://www.bouncycastle.org.
Loading
Loading