Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add kafka connector #1771

Merged
merged 2 commits into from
May 12, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions extensions/kafka-connect-jdbc/.github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* @confluentinc/connect
19 changes: 19 additions & 0 deletions extensions/kafka-connect-jdbc/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Build products
target/

# IntelliJ data
*.iml
.idea/
.ipr

# Documentation build output
/docs/_build

.DS_Store

derby.log

# Test databases
__test_database_*
test.sqlite
*.db-journal
188 changes: 188 additions & 0 deletions extensions/kafka-connect-jdbc/CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,188 @@
# Contributing

In order for us to consider merging a contribution, you will need to sign our
**C**ontributor **L**icense **A**greement.

> The purpose of a CLA is to ensure that the guardian of a project's outputs has the necessary ownership or grants of rights over all contributions to allow them to distribute under the chosen licence.
> [Wikipedia](http://en.wikipedia.org/wiki/Contributor_License_Agreement)

You can read and sign our full Contributor License Agreement [here](http://clabot.confluent.io/cla).

## Reporting Bugs and Issues

Report bugs and issues by creating a new GitHub issue. Prior to creating an issue, please search
through existing issues so that you are not creating duplicate ones. If a pull request exists that
corresponds to the issue, mention this pull request on the GitHub issue.

## Guidelines for Contributing Code, Examples, Documentation

Code changes are submitted via a pull request (PR). When submitting a PR use the following
guidelines:

* Follow the style guide below
* Add/update documentation appropriately for the change you are making.
* Non-trivial changes should include unit tests covering the new functionality and potentially integration tests.
* Bug fixes should include unit tests and/or integration tests proving the issue is fixed.
* Try to keep pull requests short and submit separate ones for unrelated features.
* Keep formatting changes in separate commits to make code reviews easier and distinguish them from actual code changes.

### Code Style
This connector is using a coding style that generally follows the [Google Java coding standard guide](https://google.github.io/styleguide/javaguide.html).

Some conventions worth mentioning are:

* Indentation (single tab) is 2 spaces.
* All import statements are listed explicitly. The wildcard (*) is not used in imports.
* Imports are groups as follows:
```
import all packages not listed below (all other imports)
<blank line>
import all javax.* packages
import all java.* packages
<blank line>
import all io.confluent.* packages
<blank line>
import static packages
```
* Javadoc is highly recommended and often required during reviews in interfaces and public or protected classes and methods.

### Titles and changelogs

The title of a pull request is used as an entry on the release notes (aka changelogs) of the
connector in every release.

For this reason, please use a brief but descriptive title for your pull request. If GitHub shortens
your pull request title when you issue the pull request adding the excessive part to the pull
request description, make sure that you correct the title before or after you issue the pull
request.

If the fix is a minor fix you are encouraged to use the tag `MINOR:` followed by your pull request
title. You may link the corresponding issue to the description of your pull request but adding it to
the title will not be useful during changelog generation.

When reverting a previous commit, use the prefix `Revert ` on the pull request title (automatically
added by GitHub when a pull request is created to revert an existing commit).

### Tests
Every pull request should contain a sufficient amount of tests that assess your suggested code
changes. It’s highly recommended that you also check the code coverage of the production code you
are adding to make sure that your changes are covered sufficiently by the test code.

### Description
Including a good description when you issue your pull requests helps significantly with reviews.
Feel free to follow the template that is when issuing a pull request and mention how your changes
are tested.

### Backporting Commits
If your code changes are essentially bug fixes that make sense to backport to existing releases make sure to target the earliest release branch (e.g. 2.0.x) that should contain your changes. When selecting the release branch you should also consider how easy it will be to resolve any conflicts in newer release branches, including the `master` branch.

## Github Workflow

1. Fork the connector repository into your GitHub account: https://github.com/confluentinc/kafka-connect-jdbc/fork

2. Clone your fork of the GitHub repository, replacing `<username>` with your GitHub username.

Use ssh (recommended):

```bash
git clone [email protected]:<username>/kafka-connect-jdbc.git
```

Or https:

```bash
git clone https://github.com/<username>/kafka-connect-jdbc.git
```

3. Add a remote to keep up with upstream changes.

```bash
git remote add upstream https://github.com/confluentinc/kafka-connect-jdbc.git
```

If you already have a copy, fetch upstream changes.

```bash
git fetch upstream
```

or

```bash
git remote update
```

4. Create a feature branch to work in.

```bash
git checkout -b feature-xyz upstream/master
```

5. Work in your feature branch.

```bash
git commit -a --verbose
```

6. Periodically rebase your changes

```bash
git pull --rebase
```

7. When done, combine ("squash") related commits into a single one

```bash
git rebase -i upstream/master
```

This will open your editor and allow you to re-order commits and merge them:
- Re-order the lines to change commit order (to the extent possible without creating conflicts)
- Prefix commits using `s` (squash) or `f` (fixup) to merge extraneous commits.

8. Submit a pull-request

```bash
git push origin feature-xyz
```

Go to your fork main page

```bash
https://github.com/<username>/kafka-connect-jdbc.git
```

If you recently pushed your changes GitHub will automatically pop up a `Compare & pull request`
button for any branches you recently pushed to. If you click that button it will automatically
offer you to submit your pull-request to the `confluentinc` connector repository.

- Give your pull-request a meaningful title as described [above](#titles-and-changelogs).
- In the description, explain your changes and the problem they are solving.

9. Addressing code review comments

Repeat steps 5. through 7. to address any code review comments and rebase your changes if necessary.

Push your updated changes to update the pull request

```bash
git push origin [--force] feature-xyz
```

`--force` may be necessary to overwrite your existing pull request in case your
commit history was changed when performing the rebase.

Note: Be careful when using `--force` since you may lose data if you are not careful.

```bash
git push origin --force feature-xyz
```

## Useful Resources for Developers

1. Connector Developer Guide: https://docs.confluent.io/platform/current/connect/devguide.html
2. A Guide to the Confluent Verified Integrations Program: https://www.confluent.io/blog/guide-to-confluent-verified-integrations-program/
3. Verification Guide for Confluent Platform Integrations: https://cdn.confluent.io/wp-content/uploads/Verification-Guide-Confluent-Platform-Connectors-Integrations.pdf
4. From Zero to Hero with Kafka Connect: https://www.confluent.io/kafka-summit-lon19/from-zero-to-hero-with-kafka-connect/
5. 4 Steps to Creating Apache Kafka Connectors with the Kafka Connect API: https://www.confluent.io/blog/create-dynamic-kafka-connect-source-connectors/
6. How to Write a Connector for Kafka Connect – Deep Dive into Configuration Handling: https://www.confluent.io/blog/write-a-kafka-connect-connector-with-configuration-handling/
8 changes: 8 additions & 0 deletions extensions/kafka-connect-jdbc/Jenkinsfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#!/usr/bin/env groovy
common {
slackChannel = '#connect-warn'
nodeLabel = 'docker-debian-jdk8'
upstreamProjects = 'confluentinc/common'
pintMerge = true
downStreamValidate = false
}
114 changes: 114 additions & 0 deletions extensions/kafka-connect-jdbc/LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
Confluent Community License Agreement
Copy link
Collaborator

@aceforeverd aceforeverd May 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is license compatible ?

Copy link
Collaborator

@lumianph lumianph May 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have checked with the legal team, it should be okay if we are not the competitor of their commercial products

Version 1.0

This Confluent Community License Agreement Version 1.0 (the “Agreement”) sets
forth the terms on which Confluent, Inc. (“Confluent”) makes available certain
software made available by Confluent under this Agreement (the “Software”).  BY
INSTALLING, DOWNLOADING, ACCESSING, USING OR DISTRIBUTING ANY OF THE SOFTWARE,
YOU AGREE TO THE TERMS AND CONDITIONS OF THIS AGREEMENT. IF YOU DO NOT AGREE TO
SUCH TERMS AND CONDITIONS, YOU MUST NOT USE THE SOFTWARE.  IF YOU ARE RECEIVING
THE SOFTWARE ON BEHALF OF A LEGAL ENTITY, YOU REPRESENT AND WARRANT THAT YOU
HAVE THE ACTUAL AUTHORITY TO AGREE TO THE TERMS AND CONDITIONS OF THIS
AGREEMENT ON BEHALF OF SUCH ENTITY. “Licensee” means you, an individual, or
the entity on whose behalf you are receiving the Software.

1. LICENSE GRANT AND CONDITIONS.

1.1 License.  Subject to the terms and conditions of this Agreement,
Confluent hereby grants to Licensee a non-exclusive, royalty-free,
worldwide, non-transferable, non-sublicenseable license during the term
of this Agreement to: (a) use the Software; (b) prepare modifications and
derivative works of the Software; (c) distribute the Software (including
without limitation in source code or object code form); and (d) reproduce
copies of the Software (the “License”).  Licensee is not granted the
right to, and Licensee shall not, exercise the License for an Excluded
Purpose.  For purposes of this Agreement, “Excluded Purpose” means making
available any software-as-a-service, platform-as-a-service,
infrastructure-as-a-service or other similar online service that competes
with Confluent products or services that provide the Software.

1.2 Conditions.  In consideration of the License, Licensee’s distribution
of the Software is subject to the following conditions:

(a) Licensee must cause any Software modified by Licensee to carry
prominent notices stating that Licensee modified the Software.

(b) On each Software copy, Licensee shall reproduce and not remove or
alter all Confluent or third party copyright or other proprietary
notices contained in the Software, and Licensee must provide the
notice below with each copy.  

“This software is made available by Confluent, Inc., under the
terms of the Confluent Community License Agreement, Version 1.0
located at http://www.confluent.io/confluent-community-license.  BY
INSTALLING, DOWNLOADING, ACCESSING, USING OR DISTRIBUTING ANY OF
THE SOFTWARE, YOU AGREE TO THE TERMS OF SUCH LICENSE AGREEMENT.”

1.3 Licensee Modifications.  Licensee may add its own copyright notices
to modifications made by Licensee and may provide additional or different
license terms and conditions for use, reproduction, or distribution of
Licensee’s modifications.  While redistributing the Software or
modifications thereof, Licensee may choose to offer, for a fee or free of
charge, support, warranty, indemnity, or other obligations. Licensee, and
not Confluent, will be responsible for any such obligations.

1.4 No Sublicensing.  The License does not include the right to
sublicense the Software, however, each recipient to which Licensee
provides the Software may exercise the Licenses so long as such recipient
agrees to the terms and conditions of this Agreement.  

2. TERM AND TERMINATION.  This Agreement will continue unless and until
earlier terminated as set forth herein.  If Licensee breaches any of its
conditions or obligations under this Agreement, this Agreement will
terminate automatically and the License will terminate automatically and
permanently.

3. INTELLECTUAL PROPERTY.  As between the parties, Confluent will retain all
right, title, and interest in the Software, and all intellectual property
rights therein.  Confluent hereby reserves all rights not expressly granted
to Licensee in this Agreement. Confluent hereby reserves all rights in its
trademarks and service marks, and no licenses therein are granted in this
Agreement.

4. DISCLAIMER.  CONFLUENT HEREBY DISCLAIMS ANY AND ALL WARRANTIES AND
CONDITIONS, EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, AND SPECIFICALLY
DISCLAIMS ANY WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR
PURPOSE, WITH RESPECT TO THE SOFTWARE.  

5. LIMITATION OF LIABILITY.  CONFLUENT WILL NOT BE LIABLE FOR ANY DAMAGES OF
ANY KIND, INCLUDING BUT NOT LIMITED TO, LOST PROFITS OR ANY CONSEQUENTIAL,
SPECIAL, INCIDENTAL, INDIRECT, OR DIRECT DAMAGES, HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, ARISING OUT OF THIS AGREEMENT.  THE FOREGOING SHALL
APPLY TO THE EXTENT PERMITTED BY APPLICABLE LAW.

6.GENERAL.

6.1 Governing Law. This Agreement will be governed by and interpreted in
accordance with the laws of the state of California, without reference to
its conflict of laws principles.  If Licensee is located within the
United States, all disputes arising out of this Agreement are subject to
the exclusive jurisdiction of courts located in Santa Clara County,
California. USA.  If Licensee is located outside of the United States,
any dispute, controversy or claim arising out of or relating to this
Agreement will be referred to and finally determined by arbitration in
accordance with the JAMS International Arbitration Rules.  The tribunal
will consist of one arbitrator. The place of arbitration will be Palo
Alto, California. The language to be used in the arbitral proceedings
will be English. Judgment upon the award rendered by the arbitrator may
be entered in any court having jurisdiction thereof.

6.2 Assignment.  Licensee is not authorized to assign its rights under
this Agreement to any third party. Confluent may freely assign its rights
under this Agreement to any third party.

6.3 Other.  This Agreement is the entire agreement between the parties
regarding the subject matter hereof.  No amendment or modification of
this Agreement will be valid or binding upon the parties unless made in
writing and signed by the duly authorized representatives of both
parties.  In the event that any provision, including without limitation
any condition, of this Agreement is held to be unenforceable, this
Agreement and all licenses and rights granted hereunder will immediately
terminate.  Waiver by Confluent of a breach of any provision of this
Agreement or the failure by Confluent to exercise any right hereunder
will not be construed as a waiver of any subsequent breach of that right
or as a waiver of any other right.
20 changes: 20 additions & 0 deletions extensions/kafka-connect-jdbc/NOTICE
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
kafka-connect-jdbc
Copyright (c) 2015 Confluent Inc.

The following libraries are included in packaged versions of this project:

* SQLite JDBC Driver
* COPYRIGHT: Copyright Taro L. Saito, David Crenshaw
* LICENSE: licenses/LICENSE.apache2.txt
* NOTICE: licenses/NOTICE.sqlite-jdbc.txt
* HOMEPAGE: https://github.com/xerial/sqlite-jdbc

* PostgreSQL JDBC Driver
* COPYRIGHT: Copyright 1997-2011, PostgreSQL Global Development Group
* LICENSE: licenses/LICENSE.bsd.txt
* HOMEPAGE: https://jdbc.postgresql.org/

* MariaDB JDBC Driver
* COPYRIGHT: Copyright 2012 Monty Program Ab., 2009-2011, Marcus Eriksson
* LICENSE: licenses/LICENSE.lgpl.txt
* HOMEPAGE: https://mariadb.com/kb/en/mariadb/about-mariadb-connector-j/
35 changes: 35 additions & 0 deletions extensions/kafka-connect-jdbc/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Kafka Connect JDBC Connector

kafka-connect-jdbc is a [Kafka Connector](http://kafka.apache.org/documentation.html#connect)
aceforeverd marked this conversation as resolved.
Show resolved Hide resolved
for loading data to and from any JDBC-compatible database.

Documentation for this connector can be found [here](http://docs.confluent.io/current/connect/connect-jdbc/docs/index.html).

# Development

To build a development version you'll need a recent version of Kafka as well as a set of upstream Confluent projects, which you'll have to build from their appropriate snapshot branch. See the [FAQ](https://github.com/confluentinc/kafka-connect-jdbc/wiki/FAQ)
for guidance on this process.

You can build kafka-connect-jdbc with Maven using the standard lifecycle phases.

# FAQ

Refer frequently asked questions on Kafka Connect JDBC here -
https://github.com/confluentinc/kafka-connect-jdbc/wiki/FAQ

# Contribute

Contributions can only be accepted if they contain appropriate testing. For example, adding a new dialect of JDBC will require an integration test.

- Source Code: https://github.com/confluentinc/kafka-connect-jdbc
- Issue Tracker: https://github.com/confluentinc/kafka-connect-jdbc/issues
- Learn how to work with the connector's source code by reading our [Development and Contribution guidelines](CONTRIBUTING.md).

# Information

For more information, check the documentation for the JDBC connector on the [confluent.io](https://docs.confluent.io/current/connect/kafka-connect-jdbc/index.html) website. Questions related to the connector can be asked on [Community Slack](https://launchpass.com/confluentcommunity) or the [Confluent Platform Google Group](https://groups.google.com/forum/#!topic/confluent-platform/).

# License

This project is licensed under the [Confluent Community License](LICENSE).

Loading