Feature Request: Run a single (or a set of) task(s) and any dependent tasks

[benjaminrose]

I want to run only a single task, and all the tasks it depends on.

Similar features in Pippin

I know you can run starting (-s) at, stopping (-f) at, or skipping (-i) a step, but I don't need all the tasks from a step, but rather just one or two tasks.

Example

Below is a (truncated) example output from pippin.sh -c where you can see that Pippin already knows all the tasks and their dependencies. The updated code would just need to track what task should be run/skipped, similar to the code that keeps track of skipped tasks.

ROMAN_DATA_LCFIT_ROMAN_SN_CC Listing tasks:
	SNANASimulation ROMAN_PLAsTiCC task (1 jobs, deps [])
	SNANASimulation ROMAN_SN_Ia task (1 jobs, deps [])
	SNANASimulation ROMAN_SN_CC task (1 jobs, deps [])
	SNANASimulation ROMAN_BIASCOR task (1 jobs, deps [])
	SNANASimulation LOWZ_DATA task (1 jobs, deps [])
	SNANASimulation LOWZ_BIASCOR task (1 jobs, deps [])
	SNANALightCurveFit ROMAN_DATA_LCFIT_ROMAN_SN_Ia task (12 jobs, deps ['ROMAN_SN_Ia'])
	SNANALightCurveFit ROMAN_DATA_LCFIT_ROMAN_SN_CC task (12 jobs, deps ['ROMAN_SN_CC'])
	SNANALightCurveFit ROMAN_BIAS_LCFIT_ROMAN_BIASCOR task (36 jobs, deps ['ROMAN_BIASCOR'])
	SNANALightCurveFit LOWZ_DATA_LCFIT_LOWZ_DATA task (10 jobs, deps ['LOWZ_DATA'])
	SNANALightCurveFit LOWZ_BIAS_LCFIT_LOWZ_BIASCOR task (10 jobs, deps ['LOWZ_BIASCOR'])

I would like to run something like pippin.sh --task ROMAN_PLAsTiCC or --task ROMAN_DATA_LCFIT_ROMAN_SN_Ia (this will run both ROMAN_DATA_LCFIT_ROMAN_SN_Ia and ROMAN_SN_Ia).

Reason

Really what I am building two Pippin-pipelines in one file since there are a few overlapping simulations. I want one pipeline to make a full set of simulations of about a dozen transient types, and another pipeline that performs a Roman photometrically classified cosmology analysis. These are mostly unique tasks, but overlap in the SNIa and CCSN data sims.

[djbrout]

Perhaps you should just separate out into two separate pippin files and have one of them point to external directories for the sims that are common to both pipelines.

…

On Wed, Aug 10, 2022 at 12:18 PM Benjamin Rose ***@***.***> wrote: I want to run only a single task, and all the tasks it depends on. Similar features in Pippin I know you can run starting (-s) at, stopping (-f) at, or skipping (-i) a step, but I don't need all the tasks from a step, but rather just one or two tasks. Example Below is a (truncated) example output from pippin.sh -c where you can see that Pippin already knows all the tasks and their dependencies. The updated code would just need to track what task should be run/skipped, similar to the code that keeps track of skipped tasks. ROMAN_DATA_LCFIT_ROMAN_SN_CC Listing tasks: SNANASimulation ROMAN_PLAsTiCC task (1 jobs, deps []) SNANASimulation ROMAN_SN_Ia task (1 jobs, deps []) SNANASimulation ROMAN_SN_CC task (1 jobs, deps []) SNANASimulation ROMAN_BIASCOR task (1 jobs, deps []) SNANASimulation LOWZ_DATA task (1 jobs, deps []) SNANASimulation LOWZ_BIASCOR task (1 jobs, deps []) SNANALightCurveFit ROMAN_DATA_LCFIT_ROMAN_SN_Ia task (12 jobs, deps ['ROMAN_SN_Ia']) SNANALightCurveFit ROMAN_DATA_LCFIT_ROMAN_SN_CC task (12 jobs, deps ['ROMAN_SN_CC']) SNANALightCurveFit ROMAN_BIAS_LCFIT_ROMAN_BIASCOR task (36 jobs, deps ['ROMAN_BIASCOR']) SNANALightCurveFit LOWZ_DATA_LCFIT_LOWZ_DATA task (10 jobs, deps ['LOWZ_DATA']) SNANALightCurveFit LOWZ_BIAS_LCFIT_LOWZ_BIASCOR task (10 jobs, deps ['LOWZ_BIASCOR']) I would like to run something like pippin.sh --task ROMAN_PLAsTiCC or --task ROMAN_DATA_LCFIT_ROMAN_SN_Ia (this will run both ROMAN_DATA_LCFIT_ROMAN_SN_Ia and ROMAN_SN_Ia). Reason Really what I am building two Pippin-pipelines in one file since there are a few overlapping simulations. I want one pipeline to make a full set of simulations of about a dozen transient types, and another pipeline that performs a Roman photometrically classified cosmology analysis. These are mostly unique tasks, but overlap in the SNIa and CCSN data sims. — Reply to this email directly, view it on GitHub <#98>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJHI2JDI4TLA2OHNH7N7JDVYPI4VANCNFSM56FEHYYA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[benjaminrose]

That sort of works, but then would require three pippin files: one for each task and one for building the external directories. My goal is to make it easy to get someone access to an SNANA simulation of the Roman reference survey. They can then take that fiducial survey but tweak the one thing of interest to them (say kcor or fitops) and then run Pippin on only the relevant science (say cosmology or transient detection).

I can take the first stab at trying to implement this feature.

[djbrout]

ok, well if you tweak one thing in pippin its designed to only run the dependencies. so you dont need any flags (no -s -f)... just tweak what you want to change and it will only run the things that depend on that by default.

…

On Wed, Aug 10, 2022 at 1:20 PM Benjamin Rose ***@***.***> wrote: That sort of works, but then would require three pippin files: one for each task and one for building the external directories. My goal is to make it easy to get someone access to an SNANA simulation of the Roman reference survey. They can then take that fiducial survey but tweak the one thing of interest to them (say kcor or fitops) and then run Pippin on only the relevant science (say cosmology or transient detection). I can take the first stab at trying to implement this feature. — Reply to this email directly, view it on GitHub <#98 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJHI2I75EOLETQXJRGBJUDVYPQG7ANCNFSM56FEHYYA> . You are receiving this because you commented.Message ID: ***@***.***>