[GOBBLIN-1732] Search for dummy file in writer directory #3589
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jack-moseley
changed the title
vikrambohra
suggested changes
Oct 21, 2022
| String baseDatasetString = state.getProp(ConfigurationKeys.DATA_PUBLISHER_DATASET_DIR); | ||
| Path searchPath = new Path(baseDatasetString); | ||
| if (state.contains(TimeBasedWriterPartitioner.WRITER_PARTITION_PREFIX)) { | ||
| searchPath = new Path(searchPath, state.getProp(TimeBasedWriterPartitioner.WRITER_PARTITION_PREFIX)); |
There was a problem hiding this comment.
will the WRITER_PARTITION_PREFIX always be hourly?
There was a problem hiding this comment.
For streaming yes this property is set like writer.partition.prefix=hourly
Codecov Report
@@ Coverage Diff @@
## master #3589 +/- ##
============================================
+ Coverage 43.22% 46.89% +3.66%
- Complexity 4866 10675 +5809
============================================
Files 1058 2119 +1061
Lines 42697 83027 +40330
Branches 4727 9245 +4518
============================================
+ Hits 18457 38934 +20477
- Misses 22441 40532 +18091
- Partials 1799 3561 +1762
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
LGTM. Thanks for the quick fix. |
Will-Lo
approved these changes
Oct 21, 2022
phet
pushed a commit
to phet/gobblin
that referenced
this pull request
Dec 3, 2022
phet
added a commit
to phet/gobblin
that referenced
this pull request
Dec 5, 2022
* upstream/master: move dataset handler code before cleaning up staging data (apache#3594) [GOBBLIN-1730] Include flow execution id when try to cancel/submit job using SimpleKafkaSpecProducer (apache#3588) [GOBBLIN-1734] make DestinationDatasetHandler work on streaming sources (apache#3592) give option to cancel helix workflow through Delete API (apache#3580) [GOBBLIN-1728] Fix YarnService incorrect container allocation behavior (apache#3586) Support multiple node types in shared flowgraph, fix logs (apache#3590) Search for dummy file in writer directory (apache#3589) Use root cause for checking if exception is transient (apache#3585) [GOBBLIN-1724] Support a shared flowgraph layout in GaaS (apache#3583) [GOBBLIN-1731] Enable HiveMetadataWriter to override table schema lit… (apache#3587) [GOBBLIN-1726] Avro 1.9 upgrade of Gobblin OSS (apache#3581) [GOBBLIN-1725] Fix bugs in gaas warm standby mode (apache#3582) [GOBBLIN-1718] Define DagActionStoreMonitor to listen for kill/resume… (apache#3572) Add log line for committing/retrieving watermarks in streaming (apache#3578) [GOBBLIN-1707] Enhance `IcebergDataset` to detect when files already at dest then proceed with only delta (apache#3575) Ignore AlreadyExistsException in hive writer (apache#3579) Fail GMIP container for known transient exceptions to avoid data loss (apache#3576) GOBBLIN-1715: Support vectorized row batch pooling (apache#3574) [GOBBLIN-1696] Implement file based flowgraph that detects changes to the underlying… (apache#3548) GOBBLIN-1719 Replace moveToTrash with moveToAppropriateTrash for hadoop trash (apache#3573) [GOBBLIN-1703] avoid double quota increase for adhoc flows (apache#3550)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA
Description
Before, when sending change_property events, the GMCE publisher searches in the top level dataset directory for an example file. This could find files with other schemas/paths if there are different directories in the dataset directory. This PR changes it to search under the
TimeBasedWriterPartitioner.WRITER_PARTITION_PREFIXfolder if that config exists to avoid this.Also, there was no reason why there was a list/for loop here before, because
ConfigurationKeys.DATA_PUBLISHER_DATASET_DIRis a single path, so the for loop has been removed.Tests
Updated unit test
Commits