completed flag check may erroneously stop on regex matches

[nsheff]

I'm trying to submit 6 jobs with looper. I've never submitted any before, it's a brand new project. I noticed one of them says:

Found existing status: completed. Skipping sample.

This is bizarre because it's a brand new project! It has never been submitted before.

I realize this sample shares a prefix with another sample: one is named pairs_swap_maintain_coords, which the pipeline runs, and then the next sample is named pair_swap -- which the pipeline incorrectly says is already completed.

I'm guessing there's a regex that's looking for {sample_name}*_completed.flag -- if that's the case, it would actually register the first one as completed for the second one, and then never submit that job.

[nsheff]

This may actually be a bug with pipestat, rather than looper. I'm guessing this coincides with a switch to using pipestat for status checks.

[donaldcampbelljr]

I can't reproduce this using a modified hello_looper example for both the basic and the pipestat approaches (which look for their flags in slightly different ways).

[donaldcampbelljr]

However, looking at the basic, non-pipestat example, I can see where the function fetch_sample_flags might have issues if you had a flag from a different sample in the results folder, because of this logic:

looper/looper/utils.py

Lines 93 to 98 in 1468956

	folder_contents = [os.path.join(sfolder, f) for f in os.listdir(sfolder)]
	return [
	x
	for x in folder_contents
	if os.path.splitext(x)[1] == ".flag" and os.path.basename(x).startswith(pl_name)
	]

Appears it is only concerned with .flag and the pipeline_name. The sample name doesn't matter.

[nsheff]

was this fixed by the pipestat update referenced above?

[donaldcampbelljr]

I don't believe so. The pipestat code above was broken for filebackend and is not used for getting sample statuses.

[donaldcampbelljr]

Should be solved with the above commit.