Coping with slow workflow analysis

[nschneid]

$ ducttape mwe.tape -p pBestConfig
ducttape 0.3
by Jonathan Clark
Loading workflow version history...
WARNING: 1 corrupt or incomplete workflow versions found. Ignoring them. (This could be due to upgrading from an older version of ducttape that doesn't support versioning)
WARNING: Most recent corruption: /home/nschneid/mwe/./.versions/362/manifest.txt (No such file or directory)
Have 391 previous workflow versions
Finding hyperpaths contained in plan...
Found 49 vertices implied by realization plan pBestConfig
Union of all planned vertices has size 49

...and then it takes several minutes before displaying:

Checking for completed tasks from versions 1 through 391...

...then it sits for awhile longer until it finds the two realizations that have been invalidated. Then a couple more minutes to check the packages and the work plan (17 task realizations).

In total, on the order of 10 minutes.

It shouldn't be a memory issue because I set the JVM flag to allow 2g and it's only using 1–1.5g.

I have been using fairly restrictive plans to try to limit the possibilities it has to search. But this has not entirely avoided slowness. This plan is only asking for 49 vertices, which should not be too onerous.

Could all the old versions be the cause? Why should ducttape need to examine all the older versions to run a workflow—why doesn't the previous version suffice? If I want to start experiments from scratch, can I safely clear (or rename) the .versions directory as a workaround, or will that cause problems?

[dowobeha]

I don't know much about this part - you may need to talk with Jon.

On Thu, Aug 15, 2013 at 10:54 PM, nschneid [email protected] wrote:

$ ducttape mwe.tape -p pBestConfig
ducttape 0.3
by Jonathan Clark
Loading workflow version history...
WARNING: 1 corrupt or incomplete workflow versions found. Ignoring them. (This could be due to upgrading from an older version of ducttape that doesn't support versioning)
WARNING: Most recent corruption: /home/nschneid/mwe/./.versions/362/manifest.txt (No such file or directory)
Have 391 previous workflow versions
Finding hyperpaths contained in plan...
Found 49 vertices implied by realization plan pBestConfig
Union of all planned vertices has size 49

...and then it takes several minutes before displaying:

Checking for completed tasks from versions 1 through 391...

...then it sits for awhile longer until it finds the two realizations that
have been invalidated. Then a couple more minutes to check the packages and
the work plan (17 task realizations).

In total, on the order of 10 minutes.

It shouldn't be a memory issue because I set the JVM flag to allow 2g and
it's only using 1–1.5g.

I have been using fairly restrictive plans to try to limit the
possibilities it has to search. But this has not entirely avoided slowness.
This plan is only asking for 49 vertices, which should not be too onerous.

Could all the old versions be the cause? Why should ducttape need to
examine all the older versions to run a workflow—why doesn't the previous
version suffice? If I want to start experiments from scratch, can I safely
clear (or rename) the .versions directory as a workaround, or will that
cause problems?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/155
.

When a place gets crowded enough to require ID's, social collapse is not
far away. It is time to go elsewhere. The best thing about space travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"