Compaction validate, unschedule and repair #481
Conversation
|
@n3nash : Please review and let @vinothchandar know if the PR looks good. |
| import org.apache.spark.api.java.JavaSparkContext; | ||
|
|
||
| /** | ||
| * Client to perform admin operations related to compaction |
There was a problem hiding this comment.
is there a plan to use this class to schedule compaction internally as well?
There was a problem hiding this comment.
I was thinking of keeping only potentially unsafe (non MVCC ) compaction operation in this CompactionAdminClient (manually invoked with no compaction/write is happening).
|
|
||
| private final Config cfg; | ||
|
|
||
| public HoodieCompactorAdmin(Config cfg) { |
There was a problem hiding this comment.
rename to "HoodieCompactionDriver" ?
There was a problem hiding this comment.
or HoodieCompactionTool. Something that calls out this is an utility to run and not an internal class
There was a problem hiding this comment.
is this the class one should consider to be scheduled via a Workflow manager ? I am wondering if we need a separate class or we can just run a series of hoodie-cli commands instead passed into stdin?
There was a problem hiding this comment.
I will rename to HoodieCompactionAdminTool. This class only contain methods to unschedule and repair compactions - which requires manual triggering and guarantee that no concurrent compaction/ingestion is happening. The general compaction triggering methods - compaction schedule and compaction run are in the original HoodieCompactor class.
I kept the pattern is similar to other Hoodie-CLI commands (e:g - hdfsparquetimport) which involves running Spark-Launcher process.
There are 2 options to run - One to invoke from Hoodie-CLI and another to run directly as spark-submit. Agree that this class must not be scheduled as part of Workflow-Manager. But, for whatever reasons, if someone want to submit this as spark-submit job, its possible with this approach.
|
|
||
| Validating a compaction plan : Check if all the files necessary for compactions are present and are valid | ||
|
|
||
| ``` | ||
| hoodie:trips->compaction validate --compactionInstant <instantId> | ||
| hoodie:stock_ticks_mor->compaction validate --instant 20181005222611 |
There was a problem hiding this comment.
Should we add an example of what an invalid plan looks like and what error the user can expect ?
docs/admin_guide.md
Outdated
| ``` | ||
| hoodie:trips->compaction unscheduleFileId --fileId <FileUUID> | ||
| .... | ||
| No File renames needed to unschedule compaction. Compaction successfully unscheduled |
There was a problem hiding this comment.
Is this text correct ? Here if we unschedule a fileId, may be we should say 👍
No File renames needed to unschedule file id from compaction. FileId successfully unscheduled from compaction plan.. ?
There was a problem hiding this comment.
It was coming from a reused method. Customized the message.
| ``` | ||
|
|
||
| ##### Repair Compaction |
| ObjectInputStream in = new ObjectInputStream(fsDataInputStream); | ||
| T result = (T) in.readObject(); | ||
| log.info("Result : " + result); | ||
| in.close(); |
| import java.io.Serializable; | ||
|
|
||
| /** | ||
| * Holds Operation result |
There was a problem hiding this comment.
Added more context around the usage.
|
Minor comments, didn't go through each command in very detail but in general looks fine to me. |
bvaradar
force-pushed
the
async_compac_cli_part_2
branch
from
b7b88e0 to
de74153
Compare
|
@vinothchandar @n3nash : Addressed review comments. |
bvaradar
force-pushed
the
async_compac_cli_part_2
branch
from
de74153 to
ee11726
Compare
CLI commands to validate, unschedule and repair pending compactions.