Parsl Components for maturity and usage assessment

[benclifford]

This issue is an attempt to begin identifying distinct "components" or pieces of functionality within parsl that might be individually assessed for maturity (both by gathering a consensus opinion on component maturity, and through changes in the parsl usage tracking system).

Plugin modules (such as executor and providers) should probably each count as their own component.
In addition, functionality not separated out in that way might sometimes be counted as a component (for example "memoization/checkpointing").

Please add comments to this issue to reflect your opinions.

general functionality

Component	Technical sponsor	Maturity comments/rating
memo/checkpoint	@benclifford	not used by any @benclifford project
executor frontend	@WardLT
garbage collection	@benclifford
monitoring	@benclifford	mostly functional - unquantified poor performance
monitoring.viz	@benclifford	very rough, and not the only way to access monitoring data
retries	@benclifford
retries.retry handlers	@benclifford	this was a prototype @benclifford hears occasional use of and would like feedback on
non-default logging	@benclifford	not a complicated feature but maybe interesting to understand what people want logged
non-trivial scaling	@benclifford	anything beyond "keep 1 block alive at all times"
dependencies	core	eg did a user make any tasks that depended on another task? (c.f. funcX hypothesis that many users want to launch a large bag of unordered tasks)
grid/cloud execution		eg. no shared filesystems, htex writing log directories in arbitrary wrong places, network connectivity. people keep trying to use this and we keep listing it as something parsl can do. @benclifford has even seen people try to do this inside funcX which was supposed to be the cloud executor layer that doesn't use parsl! This could use a bunch of engineering but no one is particularly motivated to do it. So it's mostly an abandoned feature that saps support time without a goal for improvement. however in certain scaling situations even on hosts with a shared fs, it's desirable to minimise/avoid the shared fs so there is some potential use here.
tutorials		these are versioned separately and can be comedically out of date (eg. in Jan 2024, they were pinned to a version of parsl over 2 years old

operating systems

Linux is the core operating system and doesn't get its own technical sponsor

Component	Technical sponsor	Maturity comments/rating
OS X support		This is the principal non-Linux platform that parsl is occasionally used on, and so occasionally breaks due to unintended linuxisms
Windows support		See issue #1878 for some general discussion.

app types

Component	Technical sponsor	Maturity comments/rating
@python_app	core
@python_app timeout
@bash_app	core
@join_app	@benclifford	experimental - @benclifford interested in feedback

executors

Component	Technical sponsor	Maturity comments/rating
htex	-	commonly used - some options might be individually identified
htex scaledown	-	uncommon
htex core pinning	@WardLT
htex accelerator allocation	@WardLT
htex start methods	@WardLT
extreme scale		deprecated. untested in CI, defacto abandoned and could not pass test suite last time @benclifford tried - PR #2622 removes almost everything except the raw code
flux		untested in CI
low latency		deprecated. untested, clearly broken, @benclifford removed in #2580
work queue	@benclifford / DESC	regular use at nersc with desc, regular testing by cctools CI
task vine		cctools people propose this as the successor to work queue
threads	@benclifford / core	unclear how much users use this but it isn't very complicated
swift_t		untested in CI

channels

Component	Technical sponsor	Maturity comments/rating
local	core	regular use with all htex/work queue users
ssh
ssh_il
oauth_ssh		@yadudoc says this should be deprecated

providers

Component	Technical sponsor	Maturity comments/rating
ad_hoc	@yadudoc
aws		cloud non-shared-fs was abandonware
azure		cloud non-shared-fs was abandonware
cobalt	@yadudoc	Regularly used at ALCF
condor	@yadudoc	Waiting for confirmation from users
googlecloud		cloud non-shared-fs was abandonware
grid_engine	@yadudoc
kubernetes		cloud non-shared-fs was abandonware
local	core	regularly used in CI
lsf	@yadudoc
pbspro	@ryanchard
slurm	@benclifford / desc	regularly used at nersc with desc
torque	@yadudoc

launchers

Component	Technical sponsor	Maturity comments/rating
simple	core
wrapped	@WardLT
SingleNode	core	regularly used by @yadudoc
GnuParallel
MpiExecLauncher
MpiRunLauncher
SrunLauncher	@benclifford / DESC	regularly used at nersc
AprunLauncher		@yadudoc regularly used at ALCF
JsrunLauncher

file staging

as cloud (non-shared-fs) style of use for parsl has been neglected, mostly file staging is neglected too.

Component	Technical sponsor	Maturity comments/rating
use of File()
ftp		not tested in CI
http
rsync		not tested in CI
globus		not tested in CI, likely broken due to globus changes that have not been implemented here