Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GOBBLIN-1747] add job.name and job.id to kafka and compaction workunits #3607

Merged
merged 3 commits into from
Nov 23, 2022

Conversation

arjun4084346
Copy link
Contributor

@arjun4084346 arjun4084346 commented Nov 23, 2022

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

Description

  • Here are some details about my PR, including screenshots (if applicable):
    adding job.name and job.id to workunits created by kafka and compaction source. these configs can be used by dataset handlers or other part of code flow

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:
    trivial changes. tested manually

Commits

  • My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

@arjun4084346 arjun4084346 changed the title add job.name and job.id to kafka workunits [GOBBLIN-1747] add job.name and job.id to kafka workunits Nov 23, 2022
@codecov-commenter
Copy link

codecov-commenter commented Nov 23, 2022

Codecov Report

Merging #3607 (aeb4e94) into master (94c40dc) will increase coverage by 2.00%.
The diff coverage is n/a.

❗ Current head aeb4e94 differs from pull request most recent head d010096. Consider uploading reports for the commit d010096 to get more accurate results

@@             Coverage Diff              @@
##             master    #3607      +/-   ##
============================================
+ Coverage     49.36%   51.37%   +2.00%     
+ Complexity     9199     7145    -2054     
============================================
  Files          1741     1335     -406     
  Lines         67495    49882   -17613     
  Branches       7715     5563    -2152     
============================================
- Hits          33322    25626    -7696     
+ Misses        31055    21995    -9060     
+ Partials       3118     2261     -857     
Impacted Files Coverage Δ
...he/gobblin/source/PartitionAwareFileRetriever.java 48.14% <0.00%> (-7.41%) ⬇️
...ache/gobblin/metastore/MysqlDataSourceFactory.java
...lin/metastore/metadata/StateStoreEntryManager.java
...rg/apache/gobblin/cluster/GobblinHelixMetrics.java
...etastore/database/DatabaseJobHistoryStoreV101.java
...pache/gobblin/cluster/GobblinClusterException.java
...pache/gobblin/cluster/GobblinHelixTaskMetrics.java
...ache/gobblin/cluster/SingleFailInCreationTask.java
...c/main/java/org/apache/gobblin/util/PortUtils.java
.../apache/gobblin/util/concurrent/TaskScheduler.java
... and 403 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@arjun4084346 arjun4084346 changed the title [GOBBLIN-1747] add job.name and job.id to kafka workunits [GOBBLIN-1747] add job.name and job.id to kafka and compaction workunits Nov 23, 2022
@Will-Lo Will-Lo merged commit c6d6c1b into apache:master Nov 23, 2022
phet pushed a commit to phet/gobblin that referenced this pull request Dec 3, 2022
…its (apache#3607)

* add job.name and job.id to kafka workunits

* fix unit test

* add job.name and job.id to compaction source workunits also

Co-authored-by: umustafi <[email protected]>
phet added a commit to phet/gobblin that referenced this pull request Dec 5, 2022
* upstream/master:
  [GOBBLIN-1747] add job.name and job.id to kafka and compaction workunits (apache#3607)
  [GOBBLIN-1746] Add fs.uri to FsDatasetDescriptor to support copy between volumes in GaaS (apache#3605)
  [GOBBLIN-1743] Ensure GobblinTaskRunner works without Yarn use (apache#3602)
  [GOBBLIN-1745] Fix bug in SimpleKafkaSpecProducer (apache#3604)
  [GOBBLIN-1739]Define Datanodes and Dataset Descriptor for Iceberg (apache#3596)
  do not close DestinationDatasetHandlerService prematurely (apache#3601)
  [GOBBLIN-1720]Add ancestors owner permissions preservations for iceberg distcp (apache#3577)
  [HOTFIX] Fix checkstyleMain (apache#3600)
  [GOBBLIN-1736] Add metrics for change stream monitor and mysql quota manager (apache#3593)
  [GOBBLIN-1737] Fix bug when using mysql user quota manager (apache#3595)
  Correct a log line and GTE with currect number of total task count (apache#3591)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants