Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GOBBLIN-1559] Support wildcard for input paths #3410

Merged
merged 4 commits into from
Oct 18, 2021

Conversation

umustafi
Copy link
Contributor

@umustafi umustafi commented Oct 7, 2021

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

Description

  • Here are some details about my PR, including screenshots (if applicable):
    Adding a special case for wildcard GaaS paths in the input, where glob patterns are accepted if the paths match exactly, and that should allow glob patterns as input paths for the self serve edges.

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:
    unit test for exact matching glob path

Commits

  • My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

@umustafi
Copy link
Contributor Author

umustafi commented Oct 8, 2021

@jack-moseley @arjun4084346 can you review?

@codecov-commenter
Copy link

codecov-commenter commented Oct 8, 2021

Codecov Report

Merging #3410 (b345c15) into master (384dc09) will increase coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master    #3410   +/-   ##
=========================================
  Coverage     46.52%   46.52%           
  Complexity    10246    10246           
=========================================
  Files          2063     2063           
  Lines         80475    80477    +2     
  Branches       8989     8990    +1     
=========================================
+ Hits          37443    37445    +2     
+ Misses        39567    39563    -4     
- Partials       3465     3469    +4     
Impacted Files Coverage Δ
...n/service/modules/dataset/FSDatasetDescriptor.java 78.00% <100.00%> (+0.91%) ⬆️
...a/org/apache/gobblin/util/limiter/NoopLimiter.java 40.00% <0.00%> (-20.00%) ⬇️
...lin/util/filesystem/FileSystemInstrumentation.java 92.85% <0.00%> (-7.15%) ⬇️
...a/org/apache/gobblin/cluster/GobblinHelixTask.java 60.21% <0.00%> (-2.16%) ⬇️
...a/management/copy/publisher/CopyDataPublisher.java 74.17% <0.00%> (-1.33%) ⬇️
.../org/apache/gobblin/cluster/GobblinTaskRunner.java 63.46% <0.00%> (-0.33%) ⬇️
...in/java/org/apache/gobblin/cluster/HelixUtils.java 38.01% <0.00%> (+5.78%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 384dc09...b345c15. Read the comment docs.

@umustafi
Copy link
Contributor Author

@phet can you merge when you get the chance

Copy link
Contributor

@phet phet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@ZihanLi58 ZihanLi58 merged commit 0d8503e into apache:master Oct 18, 2021
phet added a commit to phet/gobblin that referenced this pull request Oct 29, 2021
* upstream/master:
  Refactor `MysqlSpecStore` into a generalization, `MysqlNonFlowSpecStore` (not limited to `FlowSpec`s), also useable for `TopologySpec`s (apache#3414)
  [GOBBLIN-1563]Collect more information to analyze the RC for some job cannot emit kafka events to update job status (apache#3416)
  [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard (apache#3372)
  [GOBBLIN-1559] Support wildcard for input paths (apache#3410)
  [GOBBLIN-1561]Improve error message when flow compilation fails (apache#3412)
  [GOBBLIN-1556]Add shutdown logic in FsJobConfigurationManager (apache#3407)
  [GOBBLIN-1542] Integrate with Helix API to add/remove task from a running helix job (apache#3393)
arjun4084346 pushed a commit to arjun4084346/gobblin that referenced this pull request Nov 4, 2021
* [GOBBLIN-1559] Support wildcard for input paths

* [GOBBLIN-1559] Support wildcard for input paths

* remove new check and allow 'other' to be glob

* go back to adding special case for exact match of this & other

Co-authored-by: Urmi Mustafi <[email protected]>
jack-moseley pushed a commit to jack-moseley/gobblin that referenced this pull request Aug 24, 2022
* [GOBBLIN-1559] Support wildcard for input paths

* [GOBBLIN-1559] Support wildcard for input paths

* remove new check and allow 'other' to be glob

* go back to adding special case for exact match of this & other

Co-authored-by: Urmi Mustafi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants