[GOBBLIN-1814] Add MRJobLauncher configurability for any failing mapper to be fatal to the MR job
#3675
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* upstream/master: (124 commits) [GOBBLIN-1699] Log progress of reducer task for visibility with slow compaction jobs apache#3552 fix helix job wait completion bug when job goes to STOPPING state (apache#3556) [GOBBLIN-1695] Fix: Failure to add spec executors doesn't block deployment (apache#3551) [GOBBLIN-1701] Replace jcenter with either maven central or gradle plugin portal (apache#3554) [GOBBLIN-1700] Remove unused coveralls-gradle-plugin dependency add MysqlUserQuotaManager (apache#3545) [GOBBLIN-1689] Decouple compiler from scheduler in warm standby mode (apache#3544) Add GMCE topic explicitly to hive commit event (apache#3547) [GOBBLIN-1678] Refactor git flowgraph component to be extensible (apache#3536) [GOBBLIN-1690] Added logging to ORC writer Allow all iceberg exceptions to be fault tolerant (apache#3541) Guard against exists fs call as well (apache#3538) Add error handling for timeaware finder to handle scenarios where fil… (apache#3537) [GOBBLIN-1675] Add pagination for GaaS on server side (apache#3533) [GOBBLIN-1672] Refactor metrics from DagManager into its own class, add metrics per … (apache#3532) [GOBBLIN-1677] Fix timezone property to read from key correctly (apache#3535) [Gobblin-931] Fix typo in gobblin CLI usage (apache#3530) [GOBBLIN-1671] : Fix gobblin.sh script to add external jars as colon separated to HADOOP_CLASSPATH (apache#3531) [GOBBLIN-1656] Return a http status 503 on GaaS when quota is exceeded for user or flowgroup (apache#3516) [GOBBLIN-1669] Clean up TimeAwareRecursiveCopyableDataset to support seconds in time… (apache#3528) [GOBBLIN-1670] Remove rat tasks and unneeded checkstyles blocking build pipeline (apache#3529) [GOBBLIN-1668] Add audit counts for iceberg registration (apache#3527) [GOBBLIN-1667] Create new predicate - ExistingPartitionSkipPredicate (apache#3526) Calculate requested container count based on adding allocated count and outstanding ContainerRequests in Yarn (apache#3524) make the requestedContainerCountMap correctly update the container count (apache#3523) Fix running counts for retried flows (apache#3520) Allow table to flush after write failure (apache#3522) [GOBBLIN-1652]Add more log in the KafkaJobStatusMonitor in case it fails to process one GobblinTrackingEvent (apache#3513) Make Yarn container and helix instance allocation group by tag (apache#3519) [GOBBLIN-1657] Update completion watermark on change_property in IcebergMetadataWriter (apache#3517) [GOBBLIN-1654] Add capacity floor to avoid aggressively requesting resource and small files. (apache#3515) [GOBBLIN-1653] Shorten job name length if it exceeds 255 characters (apache#3514) [GOBBLIN-1650] Implement flowGroup quotas for the DagManager (apache#3511) [GOBBLIN-1648] Complete use of JDBC `DataSource` 'read-only' validation query by incorporating where previously omitted (apache#3509) Add config to set close timeout in HiveRegister (apache#3512) add an API in AbstractBaseKafkaConsumerClient to list selected topics (apache#3501) [GOBBLIN-1649] Revert gobblin-1633 (apache#3510) [GOBBLIN-1639] Prevent metrics reporting if configured, clean up workunit count metric (apache#3500) [GOBBLIN-1647] Add hive commit GTE to HiveMetadataWriter (apache#3508) [GOBBLIN-1633] Fix compaction actions on job failure not retried if compaction succeeds (apache#3494) [GOBBLIN-1646] Revert yarn container / helix tag group changes (apache#3507) [GOBBLIN-1641] Add meter for sla exceeded flows (apache#3502) GOBBLIN-1644 (apache#3506) [GOBBLIN-1645]Change the prefix of dagManager heartbeat to make it consistent with other metrics (apache#3505) Fix bug when shrinking the container in Yarn service (apache#3504) [GOBBLIN-1637] Add writer, operation, and partition info to failed metadata writer events (apache#3498) [GOBBLIN-1638] Fix unbalanced running count metrics due to Azkaban failures (apache#3499) [GOBBLIN-1634] Add retries on flow sla kills (apache#3495) [GOBBLIN-1620]Make yarn container allocation group by helix tag (apache#3487) [GOBBLIN-1636] Close DatasetCleaner after clean task (apache#3497) [GOBBLIN-1635] Avoid loading env configuration when using config store to improve the performance (apache#3496) use user supplied props to create FileSystem in DatasetCleanerTask (apache#3483) [GOBBLIN-1619] WriterUtils.mkdirsWithRecursivePermission contains race condition and puts unnecessary load on filesystem (apache#3477) use data node aliases to figure out data node names before using DMAS (apache#3493) [GOBBLIN-1630] Remove flow level metrics for adhoc flows (apache#3491) [GOBBLIN-1631]Emit heartbeat for dagManagerThread (apache#3492) [GOBBLIN-1624] Refactor quota management, fix various bugs in accounting of running … (apache#3481) [GOBBLIN-1613] Add metadata writers field to GMCE schema (apache#3490) Update README.md [GOBBLIN-1629] Make GobblinMCEWriter be able to catch error when calculating hive specs (apache#3489) Add/fix some fields of MetadataWriterFailureEvent (apache#3485) [GOBBLIN-1627] provide option to convert datanodes names (apache#3484) Add coverage for edge cases when table paths do not exist, check parents (apache#3482) [GOBBLIN-1616] Add close connection logic in salseforceSource (apache#3486) [GOBBLIN-1621] Make HelixRetriggeringJobCallable emit job skip event when job is dropped due to previous job is running (apache#3478) [GOBBLIN-1623] Fix NPE when try to close RestApiConnector (apache#3480) Clear bad mysql packages from cache in CI/CD machines (apache#3479) [GOBBLIN-1617] pass configurations to some HadoopUtils APIs (apache#3475) [GOBBLIN-1616] Make RestApiConnector be able to close the connection finally (apache#3474) add config to set log level for any class (apache#3473) Fix bug where partitioned tables would always return the wrong equality in paths (apache#3472) [GOBBLIN-1602] Change hive table location and partition check to validate using FS r… (apache#3459) Don't flush on change_property operation (apache#3467) Fix case where error GTE is incorrectly sent from MCE writer (apache#3466) partial rollback of PR 3464 (apache#3465) [GOBBLIN-1604] Throw exception if there are no allocated requests due to lack of res… (apache#3461) [GOBBLIN-1603] Throws error if configured when encountering an IO exception while co… (apache#3460) [GOBBLIN-1606] change DEFAULT_GOBBLIN_COPY_CHECK_FILESIZE value (apache#3464) Upgraded dropwizard metrics library version from 3.2.3 -> 4.1.2 and added a new wrapper class on dropwizard Timer.Context class to handle the code compatibility as the newer version of this class implements AutoClosable instead of Closable. (apache#3463) [GOBBLIN-1605] Fix mysql ubuntu download 404 not found for Github Actions CI/CD (apache#3462) [GOBBLIN-1601] implement ChangePermissionCommitStep (apache#3457) [GOBBLIN-1598]Fix metrics already exist issue in dag manager (apache#3454) [GOBBLIN-1597] Add error handling in dagmanager to continue if dag fails to process,… (apache#3452) GOBBLIN-1579 Fail job on hive existing target table location mismatch (apache#3433) [GOBBLIN-1596] Ignore already exists exception if the table has already been created… (apache#3451) [GOBBLIn-1595]Fix the dead lock during hive registration (apache#3450) Add guard in DagManager for improperly formed SLA (apache#3449) [GOBBLIN-1588] Send failure events for write failures when watermark is advanced in MCE writer (apache#3441) [GOBBLIN-1593] Fix bugs in dag manager about metric reporting and job status monitor (apache#3448) Fix bug in `JobSpecSerializer` of inadequately preventing access errors (within `MysqlJobCatalog`) (apache#3447) [GOBBLIN-1583] Add System level job start SLA (apache#3437) [GOBBLIN-1592] Make hive copy be able to apply filter on directory (apache#3446) [GOBBLIN-1585]GaaS (DagManager) keep retrying a failed job beyond max attempt number (apache#3439) [GOBBLIN-1590] Add low/high watermark information in event emitted by Gobblin cluster (apache#3443) [HotFix]Try to fix the mysql dependency issue in Github action (apache#3445) Lazily initialize FileContext and do not store a handle of it so it can be GC'ed when required (apache#3444) [GOBBLIN-1584] Add replace record logic for Mysql writer (apache#3438) Bump up code cov version (apache#3440) [GOBBLIN-1581] Iterate over Sql ResultSet in Only the Forward Direction (apache#3435) [GOBBLIN-1575] use reference count in helix manager, so that connect/disconnect are called once and at the right time (apache#3427) ...
* upstream/master: [GOBBLIN-1673][GOBBLIN-1683] Skeleton code for handling messages between task runner / application master for Dynamic work unit allocation (apache#3539)
* upstream/master: [GOBBLIN-1710] Codecov should be optional in CI and not fail Github Actions (apache#3562) Define basics for collecting Iceberg metadata for the current snapshot (apache#3559) [GOBBLIN-1698] Fast fail during work unit generation based on config. (apache#3542)
* upstream/master: [GOBBLIN-1716] refactor HighLevelConsumer to make consumer initializa… (apache#3570) Correct semantics of `IcebergDatasetTest` and streamline both impl and test code (apache#3571) [GOBBLIN-1713] Add missing sql source validation (apache#3567)
* upstream/master: move dataset handler code before cleaning up staging data (apache#3594) [GOBBLIN-1730] Include flow execution id when try to cancel/submit job using SimpleKafkaSpecProducer (apache#3588) [GOBBLIN-1734] make DestinationDatasetHandler work on streaming sources (apache#3592) give option to cancel helix workflow through Delete API (apache#3580) [GOBBLIN-1728] Fix YarnService incorrect container allocation behavior (apache#3586) Support multiple node types in shared flowgraph, fix logs (apache#3590) Search for dummy file in writer directory (apache#3589) Use root cause for checking if exception is transient (apache#3585) [GOBBLIN-1724] Support a shared flowgraph layout in GaaS (apache#3583) [GOBBLIN-1731] Enable HiveMetadataWriter to override table schema lit… (apache#3587) [GOBBLIN-1726] Avro 1.9 upgrade of Gobblin OSS (apache#3581) [GOBBLIN-1725] Fix bugs in gaas warm standby mode (apache#3582) [GOBBLIN-1718] Define DagActionStoreMonitor to listen for kill/resume… (apache#3572) Add log line for committing/retrieving watermarks in streaming (apache#3578) [GOBBLIN-1707] Enhance `IcebergDataset` to detect when files already at dest then proceed with only delta (apache#3575) Ignore AlreadyExistsException in hive writer (apache#3579) Fail GMIP container for known transient exceptions to avoid data loss (apache#3576) GOBBLIN-1715: Support vectorized row batch pooling (apache#3574) [GOBBLIN-1696] Implement file based flowgraph that detects changes to the underlying… (apache#3548) GOBBLIN-1719 Replace moveToTrash with moveToAppropriateTrash for hadoop trash (apache#3573) [GOBBLIN-1703] avoid double quota increase for adhoc flows (apache#3550)
* upstream/master: [GOBBLIN-1747] add job.name and job.id to kafka and compaction workunits (apache#3607) [GOBBLIN-1746] Add fs.uri to FsDatasetDescriptor to support copy between volumes in GaaS (apache#3605) [GOBBLIN-1743] Ensure GobblinTaskRunner works without Yarn use (apache#3602) [GOBBLIN-1745] Fix bug in SimpleKafkaSpecProducer (apache#3604) [GOBBLIN-1739]Define Datanodes and Dataset Descriptor for Iceberg (apache#3596) do not close DestinationDatasetHandlerService prematurely (apache#3601) [GOBBLIN-1720]Add ancestors owner permissions preservations for iceberg distcp (apache#3577) [HOTFIX] Fix checkstyleMain (apache#3600) [GOBBLIN-1736] Add metrics for change stream monitor and mysql quota manager (apache#3593) [GOBBLIN-1737] Fix bug when using mysql user quota manager (apache#3595) Correct a log line and GTE with currect number of total task count (apache#3591)
* upstream/master: [GOBBLIN-1741] Create manifest based dataset finder (apache#3598)
* upstream/master: upgrade hadoop version (apache#3621) [GOBBLIN-1744] Improve handling of null value edge cases when querying Helix (apache#3603) Handle d2 markup case when not only announcing leader (apache#3622) Update Gobblin OSS Slack channel link to a never-expire link (apache#3620) [GOBBLIN-1750] Add schemas for observability events in GaaS (apache#3610) [GOBBLIN-1757]Refactor manifest, add reader/writer and iterator for efficient reading (apache#3618) Disable flaky HiveMaterializerTest on CI/CD (apache#3619) [GOBBLIN-1754] Fixes for mysql store change monitors (apache#3615) [GOBBLIN-1756] Fix the issue that we skipping flows for multihop jobs (apache#3617) [GOBBLIN-1752] Fix race condition where FSTemplateCatalog would update at the same t… (apache#3612) [GOBBLIN-1753] Migrate DB connection pool from o.a.commons.dbcp/dbcp2 to HikariCP (apache#3613) [GOBBLIN-1748] Add logs to debug multi-hop flows creation, progression, and cleanup (apache#3608)
* upstream/master: [GOBBLIN-1767] Update references to deprecated Mysql connector/j driver to new name (apache#3626)
* upstream/master: [GOBBLIN-1759] Add error reporting when attempting to resolve flow configs (apache#3614) [GOBBLIN-1612] Add description about downloading gradle wrapper (apache#3471) [GOBBLIN-1764] Emit observability event (apache#3623)
* upstream/master: [GOBBLIN-1771] Clean up logs for dataset commit and file cleanup (apache#3631) [GOBBLIN-1770] Allow null values for fields in GaaSObservabilityEvent.Issue fields which are optional [GOBBLIN-1769] Change a noisy log that indicates that the queue capacity is almost full to debug (apache#3629) [GOBBLIN-1766] Define metric to measure lag from producing to consume… (apache#3625) [GOBBLIN-1765] Add support to sync metadata for dir in manifest based copy (apache#3624) [GOBBLIN-1768] Fix constructor in KafkaJobStatusMonitorFactory so that it can be injected (apache#3628) Specifically name each Hikari connection pool created, for traceability (apache#3627)
* upstream/master: [GOBBLIN-1774] Util for detecting non optional uniontypes Hive tables (apache#3632) [GOBBLIN-1773] Fix bugs in quota manager (apache#3636) [GOBBLIN-1782] Fix Merge State for Flow Pending Resume statuses (apache#3639) [GOBBLIN-1755] Support extended ACLs and sticky bit for file based distcp (apache#3616) [GOBBLIN-1780] Refactor/rename YarnServiceIT to YarnServiceTest (apache#3637) [GOBBLIN-1778] Add house keeping thread in DagManager to periodically sync in memory state with mysql table (apache#3635) Register gauge metrics for change monitors (apache#3634)
* upstream/master: [GOBBLIN-1804] Reject flow config updates that would fail compilation by returning service error (apache#3664)
* upstream/master:
[GOBBLIN-1804] Merge similar logic between `FlowConfig{,V2}ResourceLocalHandler.update` into single base class impl. (apache#3665)
* upstream/master: [GOBBLIN-1807] Replaces conjars.org with conjars.wensel.net (apache#3668) [GOBBLIN-1802]Register iceberg table metadata update with destination side catalog (apache#3663) Add matching of non-transient exceptions that will avoid failing the container in GMIP (apache#3662)
…pper to be fatal to the MR job
Will-Lo
approved these changes
Apr 12, 2023
Codecov Report
@@ Coverage Diff @@
## master #3675 +/- ##
============================================
+ Coverage 46.84% 46.86% +0.01%
- Complexity 10751 10757 +6
============================================
Files 2139 2140 +1
Lines 84032 84056 +24
Branches 9335 9339 +4
============================================
+ Hits 39366 39391 +25
+ Misses 41082 41080 -2
- Partials 3584 3585 +1
... and 7 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
phet
added a commit
to phet/gobblin
that referenced
this pull request
Aug 15, 2023
* upstream/master: [GOBBLIN-1832] Emit warning instead of failing job for retention of Hive Table Views (apache#3695) [GOBBLIN-1831] Use flowexecutionid in kafka monitor and jobnames (apache#3694) [GOBBLIN-1824]Improving the Efficiency of Work Planning in Manifest-Based DistCp Jobs (apache#3686) [GOBBLIN-1829] Fixes bug where the wrong workunit event was being tracked for keepin… (apache#3691) [GOBBLIN-1828] Implement Timeout for Creating Writer Functionality (apache#3690) [GOBBLIN-1827] Add check that if nested field is optional and has a non-null default… (apache#3689) [GOBBLIN-1826] Change isAssignableFrom() to isSuperTypeOf() per Guava 20 javadocs to… (apache#3688) [GOBBLIN-1822]Logging Abnormal Helix Task States (apache#3685) [GOBBLIN-1819] Log helix workflow information and timeout information during submission wait / polling (apache#3681) [GOBBLIN-1821] Let flow execution ID propagate to the Job ID if it exists (apache#3684) [GOBBLIN-1810] Support general iceberg catalog (support configurable behavior for metadata retention policy) (apache#3680) Add null default value to observability events that are additionally added (apache#3682) [GOBBLIN-1816] Add job properties and GaaS instance ID to observability event (apache#3676) [GOBBLIN-1785] add MR_JARS_BASE_DIR and logic to delete old mr jar dirs (apache#3642) initiliaze yarn clients in yarn app launcher so that a child class can override the yarn client creation logic (apache#3679) [GOBBLIN-1811]Fix Iceberg Registration Serialization (apache#3673) [GOBBLIN-1817] change some deprecated code and fix minor codestyle (apache#3678) [GOBBLIN-1812] Mockito should only be test compile (apache#3674) [GOBBLIN-1813] Helix workflows submission timeouts are configurable (apache#3677) [GOBBLIN-1810] Support general iceberg catalog in icebergMetadataWriter (apache#3672) Refactor yarn app launchers to support extending these classes (apache#3671) [GOBBLIN-1808] Bump Guava version from 15.0 to 20.0 (apache#3669) [GOBBLIN-1806] Submit dataset summary event post commit and integrate them into GaaSObservabilityEvent (apache#3667) [GOBBLIN-1814] Add `MRJobLauncher` configurability for any failing mapper to be fatal to the MR job (apache#3675) Add new lookback version finder for use with iceberg retention (apache#3670)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA
Description
Currently, the value of the MR config
mapreduce.map.failures.maxpercentis hardcoded to 100%, which means even though mappers may fail, the job would still continue. a customer seeks the ability to setmaxpercentto 0, so if any mapper fails, the overall job too would fail.Tests
none, as configuration is passed-through directly to MR, with interpretation there
Commits