-
Notifications
You must be signed in to change notification settings - Fork 322
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add kafka connector #1771
Merged
vagetablechicken
merged 2 commits into
4paradigm:main
from
vagetablechicken:kafka-connector
May 12, 2022
Merged
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Validating CODEOWNERS rules …
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
* @confluentinc/connect |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
# Build products | ||
target/ | ||
|
||
# IntelliJ data | ||
*.iml | ||
.idea/ | ||
.ipr | ||
|
||
# Documentation build output | ||
/docs/_build | ||
|
||
.DS_Store | ||
|
||
derby.log | ||
|
||
# Test databases | ||
__test_database_* | ||
test.sqlite | ||
*.db-journal |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,188 @@ | ||
# Contributing | ||
|
||
In order for us to consider merging a contribution, you will need to sign our | ||
**C**ontributor **L**icense **A**greement. | ||
|
||
> The purpose of a CLA is to ensure that the guardian of a project's outputs has the necessary ownership or grants of rights over all contributions to allow them to distribute under the chosen licence. | ||
> [Wikipedia](http://en.wikipedia.org/wiki/Contributor_License_Agreement) | ||
|
||
You can read and sign our full Contributor License Agreement [here](http://clabot.confluent.io/cla). | ||
|
||
## Reporting Bugs and Issues | ||
|
||
Report bugs and issues by creating a new GitHub issue. Prior to creating an issue, please search | ||
through existing issues so that you are not creating duplicate ones. If a pull request exists that | ||
corresponds to the issue, mention this pull request on the GitHub issue. | ||
|
||
## Guidelines for Contributing Code, Examples, Documentation | ||
|
||
Code changes are submitted via a pull request (PR). When submitting a PR use the following | ||
guidelines: | ||
|
||
* Follow the style guide below | ||
* Add/update documentation appropriately for the change you are making. | ||
* Non-trivial changes should include unit tests covering the new functionality and potentially integration tests. | ||
* Bug fixes should include unit tests and/or integration tests proving the issue is fixed. | ||
* Try to keep pull requests short and submit separate ones for unrelated features. | ||
* Keep formatting changes in separate commits to make code reviews easier and distinguish them from actual code changes. | ||
|
||
### Code Style | ||
This connector is using a coding style that generally follows the [Google Java coding standard guide](https://google.github.io/styleguide/javaguide.html). | ||
|
||
Some conventions worth mentioning are: | ||
|
||
* Indentation (single tab) is 2 spaces. | ||
* All import statements are listed explicitly. The wildcard (*) is not used in imports. | ||
* Imports are groups as follows: | ||
``` | ||
import all packages not listed below (all other imports) | ||
<blank line> | ||
import all javax.* packages | ||
import all java.* packages | ||
<blank line> | ||
import all io.confluent.* packages | ||
<blank line> | ||
import static packages | ||
``` | ||
* Javadoc is highly recommended and often required during reviews in interfaces and public or protected classes and methods. | ||
|
||
### Titles and changelogs | ||
|
||
The title of a pull request is used as an entry on the release notes (aka changelogs) of the | ||
connector in every release. | ||
|
||
For this reason, please use a brief but descriptive title for your pull request. If GitHub shortens | ||
your pull request title when you issue the pull request adding the excessive part to the pull | ||
request description, make sure that you correct the title before or after you issue the pull | ||
request. | ||
|
||
If the fix is a minor fix you are encouraged to use the tag `MINOR:` followed by your pull request | ||
title. You may link the corresponding issue to the description of your pull request but adding it to | ||
the title will not be useful during changelog generation. | ||
|
||
When reverting a previous commit, use the prefix `Revert ` on the pull request title (automatically | ||
added by GitHub when a pull request is created to revert an existing commit). | ||
|
||
### Tests | ||
Every pull request should contain a sufficient amount of tests that assess your suggested code | ||
changes. It’s highly recommended that you also check the code coverage of the production code you | ||
are adding to make sure that your changes are covered sufficiently by the test code. | ||
|
||
### Description | ||
Including a good description when you issue your pull requests helps significantly with reviews. | ||
Feel free to follow the template that is when issuing a pull request and mention how your changes | ||
are tested. | ||
|
||
### Backporting Commits | ||
If your code changes are essentially bug fixes that make sense to backport to existing releases make sure to target the earliest release branch (e.g. 2.0.x) that should contain your changes. When selecting the release branch you should also consider how easy it will be to resolve any conflicts in newer release branches, including the `master` branch. | ||
|
||
## Github Workflow | ||
|
||
1. Fork the connector repository into your GitHub account: https://github.com/confluentinc/kafka-connect-jdbc/fork | ||
|
||
2. Clone your fork of the GitHub repository, replacing `<username>` with your GitHub username. | ||
|
||
Use ssh (recommended): | ||
|
||
```bash | ||
git clone [email protected]:<username>/kafka-connect-jdbc.git | ||
``` | ||
|
||
Or https: | ||
|
||
```bash | ||
git clone https://github.com/<username>/kafka-connect-jdbc.git | ||
``` | ||
|
||
3. Add a remote to keep up with upstream changes. | ||
|
||
```bash | ||
git remote add upstream https://github.com/confluentinc/kafka-connect-jdbc.git | ||
``` | ||
|
||
If you already have a copy, fetch upstream changes. | ||
|
||
```bash | ||
git fetch upstream | ||
``` | ||
|
||
or | ||
|
||
```bash | ||
git remote update | ||
``` | ||
|
||
4. Create a feature branch to work in. | ||
|
||
```bash | ||
git checkout -b feature-xyz upstream/master | ||
``` | ||
|
||
5. Work in your feature branch. | ||
|
||
```bash | ||
git commit -a --verbose | ||
``` | ||
|
||
6. Periodically rebase your changes | ||
|
||
```bash | ||
git pull --rebase | ||
``` | ||
|
||
7. When done, combine ("squash") related commits into a single one | ||
|
||
```bash | ||
git rebase -i upstream/master | ||
``` | ||
|
||
This will open your editor and allow you to re-order commits and merge them: | ||
- Re-order the lines to change commit order (to the extent possible without creating conflicts) | ||
- Prefix commits using `s` (squash) or `f` (fixup) to merge extraneous commits. | ||
|
||
8. Submit a pull-request | ||
|
||
```bash | ||
git push origin feature-xyz | ||
``` | ||
|
||
Go to your fork main page | ||
|
||
```bash | ||
https://github.com/<username>/kafka-connect-jdbc.git | ||
``` | ||
|
||
If you recently pushed your changes GitHub will automatically pop up a `Compare & pull request` | ||
button for any branches you recently pushed to. If you click that button it will automatically | ||
offer you to submit your pull-request to the `confluentinc` connector repository. | ||
|
||
- Give your pull-request a meaningful title as described [above](#titles-and-changelogs). | ||
- In the description, explain your changes and the problem they are solving. | ||
|
||
9. Addressing code review comments | ||
|
||
Repeat steps 5. through 7. to address any code review comments and rebase your changes if necessary. | ||
|
||
Push your updated changes to update the pull request | ||
|
||
```bash | ||
git push origin [--force] feature-xyz | ||
``` | ||
|
||
`--force` may be necessary to overwrite your existing pull request in case your | ||
commit history was changed when performing the rebase. | ||
|
||
Note: Be careful when using `--force` since you may lose data if you are not careful. | ||
|
||
```bash | ||
git push origin --force feature-xyz | ||
``` | ||
|
||
## Useful Resources for Developers | ||
|
||
1. Connector Developer Guide: https://docs.confluent.io/platform/current/connect/devguide.html | ||
2. A Guide to the Confluent Verified Integrations Program: https://www.confluent.io/blog/guide-to-confluent-verified-integrations-program/ | ||
3. Verification Guide for Confluent Platform Integrations: https://cdn.confluent.io/wp-content/uploads/Verification-Guide-Confluent-Platform-Connectors-Integrations.pdf | ||
4. From Zero to Hero with Kafka Connect: https://www.confluent.io/kafka-summit-lon19/from-zero-to-hero-with-kafka-connect/ | ||
5. 4 Steps to Creating Apache Kafka Connectors with the Kafka Connect API: https://www.confluent.io/blog/create-dynamic-kafka-connect-source-connectors/ | ||
6. How to Write a Connector for Kafka Connect – Deep Dive into Configuration Handling: https://www.confluent.io/blog/write-a-kafka-connect-connector-with-configuration-handling/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
#!/usr/bin/env groovy | ||
common { | ||
slackChannel = '#connect-warn' | ||
nodeLabel = 'docker-debian-jdk8' | ||
upstreamProjects = 'confluentinc/common' | ||
pintMerge = true | ||
downStreamValidate = false | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,114 @@ | ||
Confluent Community License Agreement | ||
Version 1.0 | ||
|
||
This Confluent Community License Agreement Version 1.0 (the “Agreement”) sets | ||
forth the terms on which Confluent, Inc. (“Confluent”) makes available certain | ||
software made available by Confluent under this Agreement (the “Software”). BY | ||
INSTALLING, DOWNLOADING, ACCESSING, USING OR DISTRIBUTING ANY OF THE SOFTWARE, | ||
YOU AGREE TO THE TERMS AND CONDITIONS OF THIS AGREEMENT. IF YOU DO NOT AGREE TO | ||
SUCH TERMS AND CONDITIONS, YOU MUST NOT USE THE SOFTWARE. IF YOU ARE RECEIVING | ||
THE SOFTWARE ON BEHALF OF A LEGAL ENTITY, YOU REPRESENT AND WARRANT THAT YOU | ||
HAVE THE ACTUAL AUTHORITY TO AGREE TO THE TERMS AND CONDITIONS OF THIS | ||
AGREEMENT ON BEHALF OF SUCH ENTITY. “Licensee” means you, an individual, or | ||
the entity on whose behalf you are receiving the Software. | ||
|
||
1. LICENSE GRANT AND CONDITIONS. | ||
|
||
1.1 License. Subject to the terms and conditions of this Agreement, | ||
Confluent hereby grants to Licensee a non-exclusive, royalty-free, | ||
worldwide, non-transferable, non-sublicenseable license during the term | ||
of this Agreement to: (a) use the Software; (b) prepare modifications and | ||
derivative works of the Software; (c) distribute the Software (including | ||
without limitation in source code or object code form); and (d) reproduce | ||
copies of the Software (the “License”). Licensee is not granted the | ||
right to, and Licensee shall not, exercise the License for an Excluded | ||
Purpose. For purposes of this Agreement, “Excluded Purpose” means making | ||
available any software-as-a-service, platform-as-a-service, | ||
infrastructure-as-a-service or other similar online service that competes | ||
with Confluent products or services that provide the Software. | ||
|
||
1.2 Conditions. In consideration of the License, Licensee’s distribution | ||
of the Software is subject to the following conditions: | ||
|
||
(a) Licensee must cause any Software modified by Licensee to carry | ||
prominent notices stating that Licensee modified the Software. | ||
|
||
(b) On each Software copy, Licensee shall reproduce and not remove or | ||
alter all Confluent or third party copyright or other proprietary | ||
notices contained in the Software, and Licensee must provide the | ||
notice below with each copy. | ||
|
||
“This software is made available by Confluent, Inc., under the | ||
terms of the Confluent Community License Agreement, Version 1.0 | ||
located at http://www.confluent.io/confluent-community-license. BY | ||
INSTALLING, DOWNLOADING, ACCESSING, USING OR DISTRIBUTING ANY OF | ||
THE SOFTWARE, YOU AGREE TO THE TERMS OF SUCH LICENSE AGREEMENT.” | ||
|
||
1.3 Licensee Modifications. Licensee may add its own copyright notices | ||
to modifications made by Licensee and may provide additional or different | ||
license terms and conditions for use, reproduction, or distribution of | ||
Licensee’s modifications. While redistributing the Software or | ||
modifications thereof, Licensee may choose to offer, for a fee or free of | ||
charge, support, warranty, indemnity, or other obligations. Licensee, and | ||
not Confluent, will be responsible for any such obligations. | ||
|
||
1.4 No Sublicensing. The License does not include the right to | ||
sublicense the Software, however, each recipient to which Licensee | ||
provides the Software may exercise the Licenses so long as such recipient | ||
agrees to the terms and conditions of this Agreement. | ||
|
||
2. TERM AND TERMINATION. This Agreement will continue unless and until | ||
earlier terminated as set forth herein. If Licensee breaches any of its | ||
conditions or obligations under this Agreement, this Agreement will | ||
terminate automatically and the License will terminate automatically and | ||
permanently. | ||
|
||
3. INTELLECTUAL PROPERTY. As between the parties, Confluent will retain all | ||
right, title, and interest in the Software, and all intellectual property | ||
rights therein. Confluent hereby reserves all rights not expressly granted | ||
to Licensee in this Agreement. Confluent hereby reserves all rights in its | ||
trademarks and service marks, and no licenses therein are granted in this | ||
Agreement. | ||
|
||
4. DISCLAIMER. CONFLUENT HEREBY DISCLAIMS ANY AND ALL WARRANTIES AND | ||
CONDITIONS, EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, AND SPECIFICALLY | ||
DISCLAIMS ANY WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR | ||
PURPOSE, WITH RESPECT TO THE SOFTWARE. | ||
|
||
5. LIMITATION OF LIABILITY. CONFLUENT WILL NOT BE LIABLE FOR ANY DAMAGES OF | ||
ANY KIND, INCLUDING BUT NOT LIMITED TO, LOST PROFITS OR ANY CONSEQUENTIAL, | ||
SPECIAL, INCIDENTAL, INDIRECT, OR DIRECT DAMAGES, HOWEVER CAUSED AND ON ANY | ||
THEORY OF LIABILITY, ARISING OUT OF THIS AGREEMENT. THE FOREGOING SHALL | ||
APPLY TO THE EXTENT PERMITTED BY APPLICABLE LAW. | ||
|
||
6.GENERAL. | ||
|
||
6.1 Governing Law. This Agreement will be governed by and interpreted in | ||
accordance with the laws of the state of California, without reference to | ||
its conflict of laws principles. If Licensee is located within the | ||
United States, all disputes arising out of this Agreement are subject to | ||
the exclusive jurisdiction of courts located in Santa Clara County, | ||
California. USA. If Licensee is located outside of the United States, | ||
any dispute, controversy or claim arising out of or relating to this | ||
Agreement will be referred to and finally determined by arbitration in | ||
accordance with the JAMS International Arbitration Rules. The tribunal | ||
will consist of one arbitrator. The place of arbitration will be Palo | ||
Alto, California. The language to be used in the arbitral proceedings | ||
will be English. Judgment upon the award rendered by the arbitrator may | ||
be entered in any court having jurisdiction thereof. | ||
|
||
6.2 Assignment. Licensee is not authorized to assign its rights under | ||
this Agreement to any third party. Confluent may freely assign its rights | ||
under this Agreement to any third party. | ||
|
||
6.3 Other. This Agreement is the entire agreement between the parties | ||
regarding the subject matter hereof. No amendment or modification of | ||
this Agreement will be valid or binding upon the parties unless made in | ||
writing and signed by the duly authorized representatives of both | ||
parties. In the event that any provision, including without limitation | ||
any condition, of this Agreement is held to be unenforceable, this | ||
Agreement and all licenses and rights granted hereunder will immediately | ||
terminate. Waiver by Confluent of a breach of any provision of this | ||
Agreement or the failure by Confluent to exercise any right hereunder | ||
will not be construed as a waiver of any subsequent breach of that right | ||
or as a waiver of any other right. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
kafka-connect-jdbc | ||
Copyright (c) 2015 Confluent Inc. | ||
|
||
The following libraries are included in packaged versions of this project: | ||
|
||
* SQLite JDBC Driver | ||
* COPYRIGHT: Copyright Taro L. Saito, David Crenshaw | ||
* LICENSE: licenses/LICENSE.apache2.txt | ||
* NOTICE: licenses/NOTICE.sqlite-jdbc.txt | ||
* HOMEPAGE: https://github.com/xerial/sqlite-jdbc | ||
|
||
* PostgreSQL JDBC Driver | ||
* COPYRIGHT: Copyright 1997-2011, PostgreSQL Global Development Group | ||
* LICENSE: licenses/LICENSE.bsd.txt | ||
* HOMEPAGE: https://jdbc.postgresql.org/ | ||
|
||
* MariaDB JDBC Driver | ||
* COPYRIGHT: Copyright 2012 Monty Program Ab., 2009-2011, Marcus Eriksson | ||
* LICENSE: licenses/LICENSE.lgpl.txt | ||
* HOMEPAGE: https://mariadb.com/kb/en/mariadb/about-mariadb-connector-j/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
# Kafka Connect JDBC Connector | ||
|
||
kafka-connect-jdbc is a [Kafka Connector](http://kafka.apache.org/documentation.html#connect) | ||
aceforeverd marked this conversation as resolved.
Show resolved
Hide resolved
|
||
for loading data to and from any JDBC-compatible database. | ||
|
||
Documentation for this connector can be found [here](http://docs.confluent.io/current/connect/connect-jdbc/docs/index.html). | ||
|
||
# Development | ||
|
||
To build a development version you'll need a recent version of Kafka as well as a set of upstream Confluent projects, which you'll have to build from their appropriate snapshot branch. See the [FAQ](https://github.com/confluentinc/kafka-connect-jdbc/wiki/FAQ) | ||
for guidance on this process. | ||
|
||
You can build kafka-connect-jdbc with Maven using the standard lifecycle phases. | ||
|
||
# FAQ | ||
|
||
Refer frequently asked questions on Kafka Connect JDBC here - | ||
https://github.com/confluentinc/kafka-connect-jdbc/wiki/FAQ | ||
|
||
# Contribute | ||
|
||
Contributions can only be accepted if they contain appropriate testing. For example, adding a new dialect of JDBC will require an integration test. | ||
|
||
- Source Code: https://github.com/confluentinc/kafka-connect-jdbc | ||
- Issue Tracker: https://github.com/confluentinc/kafka-connect-jdbc/issues | ||
- Learn how to work with the connector's source code by reading our [Development and Contribution guidelines](CONTRIBUTING.md). | ||
|
||
# Information | ||
|
||
For more information, check the documentation for the JDBC connector on the [confluent.io](https://docs.confluent.io/current/connect/kafka-connect-jdbc/index.html) website. Questions related to the connector can be asked on [Community Slack](https://launchpass.com/confluentcommunity) or the [Confluent Platform Google Group](https://groups.google.com/forum/#!topic/confluent-platform/). | ||
|
||
# License | ||
|
||
This project is licensed under the [Confluent Community License](LICENSE). | ||
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is license compatible ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we have checked with the legal team, it should be okay if we are not the competitor of their commercial products