Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add PyHPS CLI #3091

Open
wants to merge 35 commits into
base: main
Choose a base branch
from
Open

feat: add PyHPS CLI #3091

wants to merge 35 commits into from

Conversation

germa89
Copy link
Collaborator

@germa89 germa89 commented May 14, 2024

As the title.

The idea is you can have:

$ pymapdl submit main_file.py 
$ pymapdl submit --name="my project" main_file.py --requirements_file=requirements.txt

The help shows:

(venv) pymapdl $ pymapdl submit --help
Usage: pymapdl submit [OPTIONS] MAIN_FILE

  Submit jobs to an HPC cluster using PyHPS package.

Options:
  --name TEXT                 Name of the PyHPS project to be created.
  --url TEXT                  URL where the HPS cluster is deployed. For
                              example: "https://myserver:3000/hps"
  --user TEXT                 Username to login into the HPC cluster.
  --password TEXT             Password used to login into the HPC cluster.
  --python TEXT               Set python version to be used to create the
                              virtual environment and run the python file. By
                              default it uses python3 (default in cluster).
  --output_files TEXT         Set the output files to be monitored.
  --shell_file TEXT           If desired, you can provide a shell script to
                              execute instead of the python file. You can call
                              your python file from it if you wish. By
                              default, it is not used.
  --requirements_file TEXT    If provided, the created virtual environment is
                              installed with the libraries specified in this
                              file. If not, the activated virtual environment
                              is cloned through a temporary 'pip list' file.
                              If you are using editable package, it is
                              recommended you attach your own requirement file
                              using ``pip freeze``
  --extra_files TEXT          To upload extra files which can be called from
                              your main python file (or from the shell file).
  --config_file TEXT          To load job configuration from a file.
  --num_cores TEXT            Set the amount of CPU cores reserved for the
                              job. By default it is 1 CPU.
  --memory TEXT               Set the amount of memory RAM in MB reserved for
                              the job. By default it is 100 MB.
  --disk_space TEXT           Set the amount of hard drive space in MB
                              reserved for the job. By default it is 100 MB.
  --exclusive TEXT            Set the job to run in a machine exclusively,
                              without sharing it with other jobs. By default
                              it is False
  --max_execution_time TEXT   Set the maximum execution time for the job. By
                              default it is zero (unlimited).
  --wait TEXT                 Set the terminal to wait for job completion
                              before return the control to the user.
  --save_config_file BOOLEAN  Writes the configuration to the config file,
                              after successfully submit the job. It overwrites
                              the configuration file. The configuration file
                              path is given using ``config_file`` argument.
  --debug BOOLEAN             Set PyMAPDL to prints the debug logging to the
                              console output.
  --help                      Show this message and exit.

@germa89 germa89 requested a review from koubaa May 14, 2024 15:41
@germa89 germa89 self-assigned this May 14, 2024
@ansys-reviewer-bot
Copy link
Contributor

Thanks for opening a Pull Request. If you want to perform a review write a comment saying:

@ansys-reviewer-bot review

@codecov-commenter
Copy link

codecov-commenter commented May 14, 2024

Codecov Report

Attention: Patch coverage is 0.88832% with 781 lines in your changes missing coverage. Please review.

Project coverage is 80.68%. Comparing base (6de31cc) to head (a78aa4c).
Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3091      +/-   ##
==========================================
- Coverage   87.12%   80.68%   -6.44%     
==========================================
  Files          55       60       +5     
  Lines        9768    10556     +788     
==========================================
+ Hits         8510     8517       +7     
- Misses       1258     2039     +781     

)

if config_file is None:
config_file = os.path.join(os.getcwd(), "hps_config.json")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@germa89 should the config be in appdirs instead?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MMhhh... if we use a file in the root directory, we could point to different files.... and hence multiple configurations.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But maybe you are right... maybe it is better to use appdirs.... buttt... it might be a bit hidden for the user...

Copy link
Collaborator Author

@germa89 germa89 May 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And we have many options to be keep in the file:

{
    "url": "https://10.231.106.91:3000/hps",
    "user": "repuser",
    "password": "repuser",
    "python": "3.9",
    "name": "my job",
    "num_cores": 1,
    "memory": 100,
    "disk_space": 100,
    "exclusive": false,
    "max_execution_time": 1000
}

keeping so many important options hidden in a appdir path might bring some complications.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@germa89 use keyring for user/password. I don't think its secure enough to put that in plaintext like this. For everything else I would agree

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree.

requirements_file = create_tmp_file("requirements.txt", content)
logger.debug(f"Requirements file in: {requirements_file}")

if not shell_file:
Copy link
Contributor

@koubaa koubaa May 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shell file is needed because pyhps doesn't support python jobs with a venv. I think we should see if that feature can be added to pyhps itself. Maybe using pip-run

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree we need to support venv within pyhps. Using pip-run seems interesting. For flexibility I would keep an optional shell file though.

requirements_file: str = None,
extra_files: Optional[Union[str, list]] = None,
config_file: str = None,
num_cores: int = None,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

option to download the result files?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not aware if we can do that from the PyHPS library... i will check it out.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, It can be done.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

germa89 added 2 commits June 18, 2024 17:19
Making ``wait`` ``true`` if `output_to_json`
@germa89
Copy link
Collaborator Author

germa89 commented Jun 18, 2024

Update 18-Jun-2024

Notes

This approach needs carefull plannification which should be done in the following days.

Features

There are some feature requests which I'm going to write here, instead of an issue.

To be implemented before end of sprint

  • Unit testing -> it will depend on where the code ends up.

Postponed until current approach is supported officially.

  • Hability to download files.
  • Use appdirs to for default configuration.

@github-actions github-actions bot added enhancement Improve any current implemented feature new feature Request or proposal for a new feature labels Jun 24, 2024
pyansys-ci-bot and others added 3 commits June 24, 2024 15:47
* First approach to login

* Adding HPS dependencies

* Adding changelog entry: 3205.miscellaneous.md

* feat: Coupling login code to current implementation.
Allowing login using token which is now the preferred method.

* coupling cli

* fix: command name in CLI

* Adding changelog entry: 3205.added.md

* fix: wrong argument that avoid having the input file as argument

* chore: checking config file in submit function.

* feat: avoid venv creation of ``requirements_file`` is False.

* feat: login CLI finished

* feat: making sure we don't get import errors when PyHPS is not installed.

* feat: Adding docstrings

* chore: renaming 'access' to 'get_token_access'.

* fix: failing piping on the CLI.

---------

Co-authored-by: pyansys-ci-bot <[email protected]>
@github-actions github-actions bot added dependencies maintenance General maintenance of the repo (libraries, cicd, etc) labels Jun 26, 2024
@germa89 germa89 added the DO NOT MERGE Not ready to be merged yet label Jul 17, 2024
@github-actions github-actions bot added the documentation Documentation related (improving, adding, etc) label Jul 17, 2024
@germa89 germa89 mentioned this pull request Oct 7, 2024
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies DO NOT MERGE Not ready to be merged yet documentation Documentation related (improving, adding, etc) enhancement Improve any current implemented feature maintenance General maintenance of the repo (libraries, cicd, etc) new feature Request or proposal for a new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants