Implement `pre-commit` & various tools #245

CasperWA · 2021-09-29T11:22:06Z

Description:

Closes #243.
~~Partly addresses #223, but does not fully close it.~~ Now fully closes #223.

Implement pre-commit.
Pre-commit is a tool that one installs in the git repository and runs prior to committing, checking various parts of the changed files.
For more information go to pre-commit.com.

This slowly introduces pre-commit hooks.
First several hooks from the pre-commit-hooks repository maintained by pre-commit themselves are added:

check-symlinks
check-xml (running for all xml, rdf, and ttl files)
check-yaml
destroyed-symlinks
end-of-file-fixer
requirements-txt-fixer
trailing-whitespace

To understand what each of these hooks do, go the README for the pre-commit-hooks repo.

Then bandit and safety will be introduced.
bandit will be introduced both as a pre-commit hooks, as well as running in the CI.
safety will only run in the CI.

Type of change:

Bug fix.
New feature.
Documentation update.

Checklist:

This checklist can be used as a help for the reviewer.

Is the code easy to read and understand?
Are comments for humans to read, not computers to disregard?
Does a new feature has an accompanying new test (in the CI or unit testing schemes)?
Has the documentation been updated as necessary?
Does this close the issue?
Is the change limited to the issue?
Are errors handled for all outcomes?
Does the new feature provide new restrictions on dependencies, and if so is this documented?

Comments:

To do:

CasperWA · 2021-09-29T12:07:28Z

Issues solved with bandit (checkout the documentation for more information):

B101
B405
B310
B404
B314
B110

CasperWA · 2021-09-29T14:12:52Z

Special pre-commit hooks have been added to update the documentation nicely using the invoke tasks.

CasperWA · 2021-09-29T14:18:03Z

This PR should also slightly address #241.
However, the "EMMO documentation" job should still be revisited, as well as the now out-commented lines for the "demos".

This adds a new installable "extra": `dev`. `dev` installs all `docs` dependencies (the ones from `requirements_docs.txt`) as well as all development-specific dependencies (the ones from `requirements_dev.txt`).

These hooks include: - Check symlinks - Check XML (run only for .xml, .rdf, and .ttl files). - Check YAML - Destroyed symlinks - End-of-file fixer - Requirements*.txt fixer - Trailing whitespace To see what all the hooks do, go to https://github.com/pre-commit/pre-commit-hooks

Use `defusedxml` instead of `xml` Replace `assert` statements with `if` -> `raise`. All notes related to `subprocess` have been suppressed. It might be good to truly ensure no harmful commands can be run using any of the `subprocess` commands at some point. The found and treated codes include: - B101 - B405 - B310 - B404 - B314 - B110 For more information see the `bandit` documentation at: https://bandit.readthedocs.io

If any Python file in the `emmopy` or `ontopy` folders are changed, the `invoke` task `create-api-reference-docs` is run, updating the API Reference section of the documentation. If the README.md file is changed, the `invoke` task `create-docs-index` is run, updating the landing page for the documentation (´index.md`).

The pre-commit hook couldn't fail previously. Now a `--pre-commit` option has been added to the `invoke` task that runs `git status` and checks whether any changes are present and if they have been staged (added to the git index). If changes are present and they have *not* been staged, the hook will fail, listing all the non-staged changes. Otherwise it succeeds. Ran the invoke task and updated the API Reference documentation.

Use `vMAJOR` version tags for appliccable actions. Add a job to run `safety`. This may also include running `pylint` in the future, due to `pylint`'s aversion to running in a virtual environment via `pre-commit` due to importability. Comment out the "demo" steps of the `tests` job so that it only runs `pytest` now - which is essentially equivalent to before.

Update `pre-commit` CI job accordingly, avoiding to install `pytest`.

Add a section under "Developers" in the documentation to document how to setup a developer environment for EMMOntoPy and installing it for development use.

Black is a stylizer, and as such is re-organizes, but does not change code. Gone through all changes, put multi-line strings into parentheses where applicable. Implemented very minor fixes (turned a stand-alone string into a comment).

Update `pyproject.toml` config file with recommended setup from `black`: https://black.readthedocs.io/en/stable/guides/using_black_with_other_tools.html#pylint

Disable `no-member` in ontodoc and ontograph. Generally disable `duplicate-code`. Identified, corrected, and disabled `cyclic-import`s.

Include `tools` explicitly, since these Python files do not include the `.py` file extension.

Issues with renaming `format` -> `fmt` and renaming `EMMObased` -> `emmo_based`.

Disable `unspecified-encoding` for pylint.

Since it's now done within a `with`-statement, the `run_pandoc()` function must also be called within that `with`-statement block.

No updates detected after running pre-commit with updated version.

docs/developers/setup.md

Co-authored-by: Francesca L. Bleken <[email protected]>

Update `venv` section, remove `conda` section, and add `virtual env considerations` tip. Furthermore, use `$` in `console` code blocks to enhance coloring. An alternative can be to use `shell` and leave out `$`.

francescalb

Lots of very good improvements, this will enhance the code development greatly I think.

CasperWA force-pushed the cwa/close-243-223-pre-commit-linting-formatting branch 2 times, most recently from 3c45922 to af952a7 Compare October 14, 2021 11:09

CasperWA force-pushed the cwa/close-243-223-pre-commit-linting-formatting branch 2 times, most recently from 1021a98 to 7279f4f Compare October 26, 2021 12:34

CasperWA marked this pull request as ready for review October 26, 2021 13:41

CasperWA force-pushed the cwa/close-243-223-pre-commit-linting-formatting branch from 970f53f to c715e76 Compare October 28, 2021 11:35

CasperWA mentioned this pull request Nov 1, 2021

Added Standard methods to Ontology #246

Merged

11 tasks

CasperWA force-pushed the cwa/close-243-223-pre-commit-linting-formatting branch 2 times, most recently from 31a1e52 to 5b67bf6 Compare November 2, 2021 12:22

CasperWA and others added 17 commits November 2, 2021 14:43

Add pre-commit config hooks and dependency

5b6f7dd

This adds a new installable "extra": `dev`. `dev` installs all `docs` dependencies (the ones from `requirements_docs.txt`) as well as all development-specific dependencies (the ones from `requirements_dev.txt`).

Update flake8 CI job to pre-commit CI job

e25ba51

Add pytest to dev requirements

9003b70

Update `pre-commit` CI job accordingly, avoiding to install `pytest`.

Update CI job names for packages

18c1265

Add documentation for developer setup/install

72ff6e4

Add a section under "Developers" in the documentation to document how to setup a developer environment for EMMOntoPy and installing it for development use.

Move section after installation one tier up in doc

94f220e

Optimize diff check in API ref. doc task

e552546

Add black pre-commit hook and run for all files

49f1c1a

Black is a stylizer, and as such is re-organizes, but does not change code. Gone through all changes, put multi-line strings into parentheses where applicable. Implemented very minor fixes (turned a stand-alone string into a comment).

Add pylint hook and CI

fd21566

Update `pyproject.toml` config file with recommended setup from `black`: https://black.readthedocs.io/en/stable/guides/using_black_with_other_tools.html#pylint

Starting to comply with pylint (slow progress)

c178ed1

Further satisfy pylint

8061434

Further pylint stuff

1b38351

CasperWA and others added 11 commits November 2, 2021 14:43

Further pylint updates

75d08e5

Further pylint stuff

8b9def0

Move exceptions to ontopy.utils

a8c3013

Complete pylint edits

52143f5

Disable `no-member` in ontodoc and ontograph. Generally disable `duplicate-code`. Identified, corrected, and disabled `cyclic-import`s.

Fully satisfy pylint (outside ontopy/emmopy)

68301de

Run pylint in CI for all python files in repo

f018b64

Include `tools` explicitly, since these Python files do not include the `.py` file extension.

Fix pytest issues

23ff380

Issues with renaming `format` -> `fmt` and renaming `EMMObased` -> `emmo_based`.

Remove all specified encoding for open()

9865d62

Disable `unspecified-encoding` for pylint.

Adapt to changed use of NamedTemporaryFile

1e40260

Since it's now done within a `with`-statement, the `run_pandoc()` function must also be called within that `with`-statement block.

Update pre-commit hook black

d37af38

No updates detected after running pre-commit with updated version.

Run pre-commit and update code base accordingly

384d844

CasperWA force-pushed the cwa/close-243-223-pre-commit-linting-formatting branch from 5604a00 to 384d844 Compare November 2, 2021 13:45

francescalb reviewed Nov 4, 2021

View reviewed changes

docs/developers/setup.md Outdated Show resolved Hide resolved

francescalb reviewed Nov 4, 2021

View reviewed changes

docs/developers/setup.md Outdated Show resolved Hide resolved

francescalb reviewed Nov 5, 2021

View reviewed changes

docs/developers/setup.md Show resolved Hide resolved

francescalb reviewed Nov 5, 2021

View reviewed changes

docs/developers/setup.md Outdated Show resolved Hide resolved

CasperWA and others added 2 commits November 8, 2021 09:54

Add venv to dev-setup instructions

b2ee8de

Co-authored-by: Francesca L. Bleken <[email protected]>

Update development env setup documentation

ebd36fa

Update `venv` section, remove `conda` section, and add `virtual env considerations` tip. Furthermore, use `$` in `console` code blocks to enhance coloring. An alternative can be to use `shell` and leave out `$`.

CasperWA requested a review from francescalb November 8, 2021 11:03

francescalb approved these changes Nov 8, 2021

View reviewed changes

CasperWA merged commit d5ae907 into master Nov 8, 2021

CasperWA deleted the cwa/close-243-223-pre-commit-linting-formatting branch November 8, 2021 13:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement `pre-commit` & various tools #245

Implement `pre-commit` & various tools #245

CasperWA commented Sep 29, 2021 •

edited by francescalb

Loading

CasperWA commented Sep 29, 2021

CasperWA commented Sep 29, 2021

CasperWA commented Sep 29, 2021 •

edited

Loading

francescalb left a comment

Implement pre-commit & various tools #245

Implement pre-commit & various tools #245

Conversation

CasperWA commented Sep 29, 2021 • edited by francescalb Loading

Description:

Type of change:

Checklist:

Comments:

CasperWA commented Sep 29, 2021

CasperWA commented Sep 29, 2021

CasperWA commented Sep 29, 2021 • edited Loading

francescalb left a comment

Choose a reason for hiding this comment

Implement `pre-commit` & various tools #245

Implement `pre-commit` & various tools #245

CasperWA commented Sep 29, 2021 •

edited by francescalb

Loading

CasperWA commented Sep 29, 2021 •

edited

Loading