add asynchronous execution on run_submission

[ixsluo]

Description & Motivation

I have this situation:
I will cycling submit batches of VASP jobs. Each job may consume different time. I want to handle each VASP result right after it finished, and submit the next VASP job according to that previous result.
I need to submit each job one-by-one, but not waiting for the previous job until it is done. If the run_submission can be changed to async, then there may be a way to achieve the above needs.

1. the submission will wait a long time for the results
2. add control on the check interval, because there is no need to wait that long in short job

Pitch

1. add Submission.async_run_submission async method
2. add check_interval argument

Additional context

Known bugs:
In LazyLocalContext, the async func runs well, but fail in slurm/pbs system.
Results will be deleted before retrieved if setting clean=True in slurm/pbs system. But setting clean=False will raise error if the same job is submitted again (e.g. running unittest in local).

[njzjz]

English is encouraged as some people don't speak Chinese.

[ixsluo]

English is encouraged as only some people don't speak Chinese.

The issue is modified, and add a more precise description
#348 (comment)

[njzjz]

I will keep this issue open as there are some known bugs.