Implement observe, regroup and compute as distinct execution phases
#809
Comments
zanete
added
the
draft
zanete
changed the title
zanete
changed the title
observe, group and compute as distinct execution phases
jmcook1186
changed the title
observe, group and compute as distinct execution phasesobserve, regroup and compute as distinct execution phases
jmcook1186
removed
the
draft
|
@jawache please review the AC to ensure you're in alignment |
|
@jmcook1186 re: if-run should include some validation logic that determines which phases to run. Specifically, if input data is already provided int he manifest file provided to if-run then the observe phase should be skipped. I think it's fine (and expected) to run the observe step even if it has inputs, the intention would be to gather observations again and run with new observations. |
|
@jmcook1186 first AC with observe, the inputs should be populated not outputs |
|
"AND if the same command is run with if-run --observe -m manifest.yml --suppress-output then no data is displayed to the console and no data is saved to file." - I'm unsure what the rationale is for surpess-output? |
|
@jmcook1186 all good apart from my comments above, very thorough! Only additional is that we should be able to run say regroup and compute in one step. So if-run -regroup -compute should be able to run both steps in one go. In fact perhaps the right flag might be -skip-observe (-skip-regoup etc...) i think the most common use case here is just to skip the observe step and run the other two. |
|
Oh and one other point, regroup should be able to run on a manifest file with previously grouped inputs. It should first flatten all inputs into a 1d array and regroup from that baseline |
|
Thanks @jawache - agree on all points, updating ticket now. The rationale for |
|
@narekhovhannisyan please deep-dive to confirm solution is clear and let me know once you have, I will proceed to split it into 5 subtickets:
|
|
Waiting to get all the manisfests in the same style to fully test the phased execution #812 |
Why: Sub of #762
What: Refactor IF into three distinct execution phases:
observere-groupcomputeEach phase can be executed individually using a command (
if-compute,if-observe,if-regroup). When run as individual commands, they accept yaml files or yaml data on stdin to operate over. They can save their outputs to yaml files or dump to console via stdout.The three phases can be run end-to-end using a single command,
if-run.However, each phase can be run individually, by passing the individual phase flags on the command line, e.g.
Note that
observeandregroupMODIFY INPUTS only.computedoes not modify inputs, it CREATES OUTPUTS only.Changes to manifests
The execution pipeline should also be separated into three phases inside the manifest so that what used to be a single contiguous pipeline now looks something like:
A manifest that has not yet passed through the
observephase will havenullinputs:--observe
if-run --observecovers the import/data creation phase of the IF execution. It should take a manifest with no input data and return a manifest with a populatedinputsarray. This could happen by executing an importer plugin or mock observations.The observe phase needs to receive a manifest file with the importer(s) configured. It will usually receive a manifest with the
inputsdata missing or empty.As -
-observeis runnable in isolation, it should apture the state of the manifest after the plugins int heobservepipeline have completed and save it to the yaml filepath specified to the--output/-ocommand. It can only output yaml data. If no output command is provided, it should display the output data to the console. However, we should also support a--suppress-outputcommand that prevents the data being displayed to the console.The importers will be initialized in precisely the same way as they are today (no separation into phases required in the
initializeblock) but the plugins to execute in theobservephase are those defined in thetree: children: child: pipeline: observe:block in the manifest.The following command should enrich a manifest with no inputs and save a copy with input data:
if-run --observe -m manifest.yml -o manifest-with-data.yml--regroup
if-run --regroupshould apply the logic from the existinggroup-byplugin but apply it as an IF core feature rather than a plugin. Thegroupbyplugin should be deleted as part of this task. Theregroupfeature will get its arguments from thetree: children: child: pipeline:regroup:config in the manifest. This block should include an array of keys to regroup by, for example:regroupwill see thecloud/instance-typevalue in the manifest and regroup the input values using thecloud/instance-typevalues found in theinputsarray, equivalent to this in today'sgroup-byplugin:regroupneeds to operate on yaml data whereinputdata exists. This can come from a file where the input data was pre-populated byif-run --observesome time in the past, or can come from a freshif-run --observe, i.e. all the following should work:if-run --regroup -m manifest-with-input-data.yml -o regrouped-manifest.ymlif-run --observe -m manifest-with-no-data.yml | if-run --regroup -o regrouped-manifest.ymlif-run --observe -m manifest-with-no-data.yml -o manifest-with-input-data.yml && if-run -m manifest-with-input-data.yml --regroup -o regrouped-manifest.yml--compute
if-run --computeshould execute all the plugins in thecomputesection of thetree: children:child: pipeline: computeblock in a manifest. For example, given the following information:--computewill execute thesum,multiply,divideandinterpolateplugins over theinputdata in sequence according to the config provided in theinitializeblock.--computeneeds to accept yaml data that has a populatedinputsarray. It will create anoutputsarray.Both of the following commands should worK
if-run --compute -m regrouped-manifest.yml -o output.ymlif-run --regroup ungrouped-manifest | if-run --computeif-run
Simply using
if-runwith none of the execution phase flags should run an entire execution pipeline that includes theobserve,regroupandcomputephases.if-runshould be able to optionally save the final output to file or display in the console using the--output/ocommand like today. If so, it is the final output from--computethat gets outputted. However, it should also be possible to save the intermediate representations (i.e. the outputs from--observeand--regreoupby specifying this on the command line.if-run -m manifest.yml -o output.yml --save-intermediates outputswill save the outputs from
--observeand--regroupto theoutputsfolder with the filenamesmanifest-observe.ymlandmanifest-regroup.yml. The output folder name is provided to--save-intermediatesand the filenames are simply the given manifest name with the-observeor-regroupsuffix appended.You can also do
if-run -o output.yml --display-intermediatesto display the results from
if-observeandif-regroupin the console but not save them to file.if-runshould include some validation logic that determines which phases to run. If there is noregroupsection or theregroupsection of the manifest is empty, then theregroupphase should be skipped.computeshould always be run.manifest validation
Each execution phase now has its own specific requirements for the content of a manifest, so they should do their own independent validation steps. The current validation of the
contextis common to all three phases, butobserverequires there to be noinputswhereasregroupandcomputerequireinputsto be present and correctly formatted.Scope of work:
if-run--observeruns from the command line #893if-run--regroupruns from the command line #894if-run--computeruns from the command line. #895if-runwith none or a combinations of phase arguments #896Acceptance criteria
Scenario 1:
if-run --observecan be run on the command lineif-run --observecan be run on the command line--observeis a flag that modifies the behaviour ofif-runso that it only executes theobservephase. It expects to operate on a manifest without input data and can either return its result to the console or save to file (this is just the normalif-runbehaviour). The result will always be yaml data. The command enriches theinputsarray but never creates outputs. The command executes plugins initialized in theinitializesection of the manifest and listed in thetree: pipeline: observesection of each node in the tree. It uses the values passed to the--manifestand--outputsubcommands.GIVEN
if-run --observeexistsWHEN I run the following command:
if-run --observe -m manifest.yml -o manifest-with-data.ymlAND manifest.yml contains the following:
THEN
if-run --observeshould createmanifest-with-inputs.ymlcontaining the following:AND if the command is updated to
if-run --observe -m manifest.yml(i.e. no-ocommand) then the same output data is displayed in the consoleAND if the same command is run with
if-run --observe -m manifest.yml --suppress-outputthen no data is displayed to the console and no data is saved to file.Scenario 2:
if-run --regroupis available on the command line.if-run --regroupis available on the command line. It accepts a manifest file with input data or yaml data arriving via stdin. It applies thegroupbylogic grouping by the keys provided intree: children: child: pipeline: regroup. It should output data to yaml file or as yaml data via stdout. It should use data passed via the--manifest,--outputand--suppress-outputsubcommands.GIVEN the feature is implemented
WHEN I run the following command:
if-run --regroup -m manifest.yml -o regrouped-manifest.ymlAND manifest.yml contains the following yaml data:
THEN
if-run --regroupsaves a file calledregrouped-manifest.ymlthat contains the following:AND when the command is updated to
if-run --regroup -m manifest.yml(i.e.-osubcommand is omitted) the same data is displayed in the console and no file is savedAND when the command is updated to
if-run --regroup -m manifest.yml --suppress-outputno data is displayed to the console and no file is savedAND if the command is updated to
if-run --observe -m manifest.yml | if-run --regroupthenif-regroupoperates over the yaml data piped in from stdin and displays the result to the console.Scenario 3:
if-run --regroupcan operate on a manifest with inputs that are already grouped.if-run --regroupcan operate on a manifest with inputs that are already grouped.If a manifest already has grouped
inputsthen--regroupshould recast them into a single, flat 1D array and then re-execute the grouping according to the config provided intree: children: child: pipeline: regroup- the original grouping should be completely destroyed and replaced with the grouping defined int he manifest.GIVEN
regroupcan act on pre-grouped manifestsWHEN I run
if-run -m manifest.yml --regroupAND
manifest.ymlcontains the following (notice the values are already grouped bycloud/regionand we're configuring it to group by bothcloud/regionandcloud/instance-type)THEN I see the following output:
Scenario 4:
if-run --computeis available on the command line.if-run --computeis available on the command line. It accepts a manifest file with input data OR yaml data arriving via stdin. It executes the plugins defined intree: children: child: pipeline: compute. It should output data to yaml file or as yaml data printed to the console. It should use values passed to the--manifest,--outputand--suppress-outputsubcommands.if-run --compute should throw an exception loudly if a) there is no input data, or b) there are duplicate timestamps in input arrays (this indicatesregroup` is needed but hasn't been applied).GIVEN the feature is available
WHEN I run the command
if-run --compute -m manifest.yml -o output.ymlAND manifest.yml contains the following yaml data:
THEN
if-run --computeshould generate a new file,outputs.ymlthat contains the following data:AND when the command is updated to
if-run --compute -m manifest.yml(i.e.-osubcommand is omitted) the same data is displayed in the console and no file is savedAND when the command is updated to
if-run --compute -m manifest.yml --suppress-outputno data is displayed to the console and no file is savedAND if the command is updated to
if-run --observe -m manifest.yml | if-run --computethenif-run --computeoperates over the yaml data piped in from stdin and displays the result to the console.AND if there are duplicate timestamps in an
inputsarray or theinputsarray is missing/empty, thenif-computeshould error out.Scenario 5:
if-runwith no phase arguments executes the full end-to-end pipelineif-runwith no phase arguments executes the full end-to-end pipelineif-runshould execute the full set of execution phases. It should be able to accept a manifest with noinputdata if there isobserveconfig available to execute, manifests with noregroupconfig as long as there are no duplicate timestamps in any inputs arrays, or manifests with noobserveconfig if there is input data already available. I should be able to save intermediate representations (i.e. the outputs fromobserveandregroup) to file or display them in the console.if-runshould also expose--manifest,--output,--suppress-outputs,save-intermediatesandshow-intermediatescommands.GIVEN
if-runis available and refactored according to the issue descriptionWHEN I run the following command:
if-run -m manifestAND
manifest.ymlcontains the following:THEN I see the following output in the console:
AND if I update the command to
if-run -m manifest.yml -o output.ymlthen the same data is saved to fileAND if I update the command to
if-run -m manifest.yml --save-intermediates ./outputsthen I get two additional files saved to theoutputsfolder:manifest-observe.ymlandmanifest-regroup.ymland the final output is printed to the console.AND if I update the command to
if-run -m manifest.yml --save-intermediates ./outputs -o ./outputs/result.ymlthen I get three files saved to theoutputsfolder:manifest-observe.ymlandmanifest-regroup.ymland the final outputresult,yml.Scenario 6:
if-runshould accept combinations of phase argumentsif-runshould accept combinations of phase arguments that allows a subset of [--observe,--regroup,--compute] to execute without having to pipe betweenif-runinvocationsWe should be able to execute, for example
if-run --regroup --computeto causeif-runto skip theobservestep but execute theregroupandcomputesteps without hvaing to doif-run --regroup -m manifest.yml | if-run compute -s. By passing multiple commands we can select which phases are executed without having to pipe outputs between separateif-runinvocations.GIVEN multiple commands are supported
WHEN I run the following command
THEN the regroup and compute phases are executed on whatever inputs are already available in the manifest.
AND if I update the command to
THEN the
observeandcomputephases should run and theregroupstage is skipped.AND if I update the command to
THEN the
observeandregroupphases should run and thecomputestage is skipped.The text was updated successfully, but these errors were encountered: