-
Notifications
You must be signed in to change notification settings - Fork 751
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[GOBBLIN-1559] Support wildcard for input paths #3410
Conversation
@jack-moseley @arjun4084346 can you review? |
...in-service/src/main/java/org/apache/gobblin/service/modules/dataset/FSDatasetDescriptor.java
Outdated
Show resolved
Hide resolved
Codecov Report
@@ Coverage Diff @@
## master #3410 +/- ##
=========================================
Coverage 46.52% 46.52%
Complexity 10246 10246
=========================================
Files 2063 2063
Lines 80475 80477 +2
Branches 8989 8990 +1
=========================================
+ Hits 37443 37445 +2
+ Misses 39567 39563 -4
- Partials 3465 3469 +4
Continue to review full report at Codecov.
|
...in-service/src/main/java/org/apache/gobblin/service/modules/dataset/FSDatasetDescriptor.java
Show resolved
Hide resolved
@phet can you merge when you get the chance |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good
* upstream/master: Refactor `MysqlSpecStore` into a generalization, `MysqlNonFlowSpecStore` (not limited to `FlowSpec`s), also useable for `TopologySpec`s (apache#3414) [GOBBLIN-1563]Collect more information to analyze the RC for some job cannot emit kafka events to update job status (apache#3416) [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard (apache#3372) [GOBBLIN-1559] Support wildcard for input paths (apache#3410) [GOBBLIN-1561]Improve error message when flow compilation fails (apache#3412) [GOBBLIN-1556]Add shutdown logic in FsJobConfigurationManager (apache#3407) [GOBBLIN-1542] Integrate with Helix API to add/remove task from a running helix job (apache#3393)
* [GOBBLIN-1559] Support wildcard for input paths * [GOBBLIN-1559] Support wildcard for input paths * remove new check and allow 'other' to be glob * go back to adding special case for exact match of this & other Co-authored-by: Urmi Mustafi <[email protected]>
* [GOBBLIN-1559] Support wildcard for input paths * [GOBBLIN-1559] Support wildcard for input paths * remove new check and allow 'other' to be glob * go back to adding special case for exact match of this & other Co-authored-by: Urmi Mustafi <[email protected]>
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA
Description
Adding a special case for wildcard GaaS paths in the input, where glob patterns are accepted if the paths match exactly, and that should allow glob patterns as input paths for the self serve edges.
Tests
unit test for exact matching glob path
Commits