Sync with internal repository

Thank you for all your contributions.
Contributions:
    Yuki Igarashi: 14 commits, 431 lines
    Ryo Miyajima: 5 commits, 296 lines

git-pysen-release-hash: 1baa6934e91ffee1763013c53738aa749a3ceb29

Co-authored-by: Yuki Igarashi <[email protected]>
Co-authored-by: Ryo Miyajima <[email protected]>

README.md

@@ -2,57 +2,40 @@
 
 ![](https://github.com/pfnet/pysen/blob/main/assets/imgs/pysen.gif?raw=true)
 
-## What is pysen?
-
-pysen aims to provide a unified platform to configure and run day-to-day development tools.
-We envision the following scenarios in the future:
-
-- You open any project and `pysen run lint`, `pysen run format` will check and format the entire codebase
-- Standardized coding styles are setup with a few lines in a single `pyproject.toml` file
-
-pysen centralizes the code and knowledge related to development tools that teams have accumulated, most notably for python linters.
-You can make tasks that can be executed from both `setup.py` and our command-line tool.
-We currently provide tasks that manage setting files for the following tools:
-
-- linters
-  - flake8
-  - isort
-  - mypy
-  - black
-- utilities
-  - (planned) protoc
-
-## What isn't pysen?
-
-* pysen is not a linting tool per se. Rather, `pysen run lint` orchestrates multiple python linting tools by automatically setting up their configurations from a more abstract setting for pysen.
-* pysen does not manage your depedencies and packages. We recommend using package managers such as [pipenv](https://github.com/pypa/pipenv) or [poetry](https://python-poetry.org/) to lock your dependecy versions, **including the versions for the linting tools that pysen coordinates** (i.e., isort, mypy, flake8, black). The supported versions for these tools can be found in the `extra_requires/lint` section in pysen's [setup.py](https://github.com/pfnet/pysen/blob/main/setup.py). You should **not** rely on `pip install pysen[lint]` to control the versions of your linting tools.
-* pysen is not limited to linting purposes or python. See the [plugin section](README.md#create-a-plugin-to-customize-pysen) for details.
-
 ## Install
 
 ### PyPI
 
+#### If you have no preference of linter versions (recommended for newbies)
+
 ```sh
 pip install "pysen[lint]"
 ```
 
+#### Install pysen with your choice of linter versions
+
+```sh
+pip install pysen
+pip install black==21.10b0 flake8==4.0.1 isort==5.10.1 mypy==0.910
+```
+
 
 ### Other installation examples
 
 ```sh
 # pipenv
-pipenv install --dev "pysen[lint]==0.9.1"
+pipenv install --dev "pysen[lint]==0.10.1"
 # poetry
-poetry add -D pysen==0.9.1 -E lint
+poetry add -D pysen==0.10.1 -E lint
 ```
 
 
 ## Quickstart: Set up linters using pysen
 
-Put the following pysen configuration to `pyproject.toml` of your python package:
+Put the following pysen configuration to either `pysen.toml` or `pyproject.toml` of your python package:
 ```toml
 [tool.pysen]
-version = "0.9"
+version = "0.10"
 
 [tool.pysen.lint]
 enable_black = true
@@ -75,7 +58,7 @@ $ pysen run format  # corrects errors with compatible commands (black, isort)
 That's it!
 pysen, or more accurately pysen tasks that support the specified linters, generate setting files for black, isort, mypy, and flake8
 and run them with the appropriate configuration.
-For more details about the configuration items that you can write in `pyproject.toml`, please refer to `pysen/pyproject_model.py`.
+For more details about the configuration items that you can write in a config file, please refer to `pysen/pyproject_model.py`.
 
 You can also add custom setup commands to your Python package by adding the following lines to its `setup.py`:
 ```py
@@ -92,6 +75,56 @@ For more details, please refer to the following two examples:
 - Example configuration from Python: `examples/advanced_example/config.py`
 - Example plugin for pysen: `examples/plugin_example/plugin.py`
 
+## Frequently Asked Questions
+
+Q. How do I use `mypy >= 0.800`?  
+A. See [Install pysen with your choice of linter versions](#install-pysen-with-your-choice-of-linter-versions)
+
+Q. mypy reports the error `Source file found twice under different module names`.  
+A. Add `tool.pysen.lint.mypy_targets` section(s) so file names are unique in each section.
+
+Q. How do I change specific settings for linter X?  
+A. We prioritize convention over configuration. However you can always create your own plugin. See: [Create a plugin to customize pysen](#create-a-plugin-to-customize-pysen)
+
+Q. pysen seems to ignore some files.  
+A. pysen only checks files that are tracked in git. Try `git add`ing the file under question.
+You can also disable this behavior by setting the environment variable `PYSEN_IGNORE_GIT=1`.
+
+Q. How do I run only [flake8|black|isort|mypy]?  
+A. Try the `--enable` and `--disable` options, for example, `pysen --enable flake --enable black run lint`.
+
+Q. Files without filename extensions are not checked.  
+A. Explicitly add those files under the include section in `tool.pysen.lint.source`.
+
+Q. How do I add additional settings to my `pyproject.toml`, e.g., [pydantic-mypy](https://pydantic-docs.helpmanual.io/mypy_plugin/#configuring-the-plugin)?  
+A. Add `settings_dir="."` under the `[tool.pysen-cli]` section.
+
+## What is pysen?
+
+pysen aims to provide a unified platform to configure and run day-to-day development tools.
+We envision the following scenarios in the future:
+
+- You open any project and `pysen run lint`, `pysen run format` will check and format the entire codebase
+- Standardized coding styles are setup with a few lines in a single config file
+
+pysen centralizes the code and knowledge related to development tools that teams have accumulated, most notably for python linters.
+You can make tasks that can be executed from both `setup.py` and our command-line tool.
+We currently provide tasks that manage setting files for the following tools:
+
+- linters
+  - flake8
+  - isort
+  - mypy
+  - black
+- utilities
+  - (planned) protoc
+
+## What isn't pysen?
+
+* pysen is not a linting tool per se. Rather, `pysen run lint` orchestrates multiple python linting tools by automatically setting up their configurations from a more abstract setting for pysen.
+* pysen does not manage your depedencies and packages. We recommend using package managers such as [pipenv](https://github.com/pypa/pipenv) or [poetry](https://python-poetry.org/) to lock your dependecy versions, **including the versions for the linting tools that pysen coordinates** (i.e., isort, mypy, flake8, black). The supported versions for these tools can be found in the `extra_requires/lint` section in pysen's [setup.py](https://github.com/pfnet/pysen/blob/main/setup.py). You should **not** rely on `pip install pysen[lint]` to control the versions of your linting tools.
+* pysen is not limited to linting purposes or python. See the [plugin section](README.md#create-a-plugin-to-customize-pysen) for details.
+
 ## How it works: Settings file directory
 
 Under the hood, whenever you run pysen, it generates the setting files as ephemeral temporary files to be used by linters.
@@ -103,7 +136,7 @@ $ pysen generate [out_dir]
 ```
 
 You can specify the settings directory that pysen uses when you `pysen run`.
-To do so add the following section to your `pyproject.toml`:
+To do so add the following section to your config:
 
 ```toml
 [tool.pysen-cli]
@@ -137,6 +170,9 @@ The result will look like the following:
 
 ![pysen-vim](https://github.com/pfnet/pysen/blob/main/assets/imgs/pysen_vim.gif?raw=true)
 
+A third party plugin is also available.
+- [pysen.vim](https://github.com/bonprosoft/pysen.vim)
+
 ### Emacs
 
 Refer to the [Compilation mode](https://www.gnu.org/software/emacs/manual/html_node/emacs/Compilation-Mode.html).
@@ -158,21 +194,21 @@ Note that this may report duplicate errors if you have configured linters like `
 
 We provide two methods to write configuration for pysen.
 
-One is the `[tool.pysen.lint]` section in `pyproject.toml`.
+One is the `[tool.pysen.lint]` section in the config.
 It is the most simple way to configure pysen, but the settings we provide are limited.
 
 The other method is to write a python script that configures pysen directly.
 If you want to customize configuration files that pysen generates, command-line arguments that pysen takes, or whatever actions pysen performs, we recommend you use this method.
 For more examples, please refer to `pysen/examples`.
 
-### pyproject.toml configuration model
+### Configuration model
 
 Please refer to `pysen/pyproject_model.py` for the latest model.
 
 Here is an example of a basic configuration:
 ```toml
 [tool.pysen]
-version = "0.9"
+version = "0.10"
 
 [tool.pysen.lint]
 enable_black = true
@@ -202,6 +238,10 @@ mypy_path = ["stubs"]
   ignore_errors = true
 ```
 
+pysen looks for a configuration file in the following order:
+1. `pysen.toml` with a `tool.pysen` section
+2. `pyproject.toml` with a `tool.pysen` section
+
 ### Create a plugin to customize pysen
 
 We provide a plugin interface for customizing our tool support, setting files management, setup commands and so on.

pyproject.toml

@@ -2,14 +2,14 @@
 settings_dir = "."
 
 [tool.pysen]
-version = "0.9"
+version = "0.10"
 
 [tool.pysen.lint]
 enable_black = true
 enable_flake8 = true
 enable_isort = true
 enable_mypy = true
-mypy_preset = "strict"
+mypy_preset = "very_strict"
 py_version = "py37"
 isort_known_first_party = ["fakes", "pysen"]
 [[tool.pysen.lint.mypy_targets]]

pysen/__main__.py

@@ -0,0 +1,4 @@
+from .cli import cli
+
+if __name__ == "__main__":
+    cli()

pysen/_version.py

@@ -1 +1 @@
-__version__ = "0.9.1"
+__version__ = "0.10.1"

pysen/cli.py

@@ -223,7 +223,7 @@ def _setup_manifest_parser() -> argparse.ArgumentParser:
     parser.add_argument(
         "--config",
         type=str,
-        help="Path for pyproject.toml",
+        help="Path to a config file",
         default=None,
     )
     parser.add_argument(

pysen/command.py

@@ -28,7 +28,9 @@ def run_files(self, reporter: Reporter, files: Sequence[pathlib.Path]) -> int:
         raise RunTargetFileNotSupported(self.name)
 
 
-def check_command_installed(*validation_command: str) -> None:
+def check_command_installed(
+    *validation_command: str,
+) -> None:
     err = CommandNotFoundError(
         f"The command `{' '.join(validation_command)}` failed."
         " Make sure it is installed."

pysen/ext/black_wrapper.py

@@ -54,15 +54,11 @@ def _parse_file_path(file_path: str) -> pathlib.Path:
 @functools.lru_cache(1)
 def _check_black_version() -> None:
     version = get_version("black")
-    compatible_versions = [
-        VersionRepresentation(19, 10),
-        VersionRepresentation(20, 8),
-    ]
+    minimum_supported = VersionRepresentation(19, 10)
 
-    if all(not v.is_compatible(version) for v in compatible_versions):
+    if version < minimum_supported:
         raise IncompatibleVersionError(
-            "pysen only supports black versions: "
-            f"{{{', '.join(v.version for v in compatible_versions)}}}. "
+            f"pysen only supports black >= {minimum_supported}, "
             f"version {version} is not supported."
         )
 
@@ -74,7 +70,7 @@ def run(
     sources: Iterable[pathlib.Path],
     inplace_edit: bool,
 ) -> int:
-    check_command_installed("black", "--version")
+    check_command_installed(*process_utils.add_python_executable("black", "--version"))
     _check_black_version()
 
     targets = [str(d) for d in sources]
@@ -87,7 +83,9 @@ def run(
         + targets
     )
     with change_dir(base_dir):
-        ret, stdout, _ = process_utils.run(cmd, reporter)
+        ret, stdout, _ = process_utils.run(
+            process_utils.add_python_executable(*cmd), reporter
+        )
 
     diagnostics = parse_error_diffs(stdout, _parse_file_path, logger=reporter.logger)
     reporter.report_diagnostics(list(diagnostics))

pysen/ext/flake8_wrapper.py

@@ -10,6 +10,7 @@
 from pysen.error_lines import parse_error_lines
 from pysen.exceptions import IncompatibleVersionError
 from pysen.path import change_dir
+from pysen.py_version import VersionRepresentation
 from pysen.reporter import Reporter
 from pysen.setting import SettingBase, to_dash_case
 
@@ -91,9 +92,10 @@ def export(self) -> Tuple[Sequence[str], Dict[str, Any]]:
 @functools.lru_cache(1)
 def _check_flake8_version() -> None:
     version = get_version("flake8")
-    if version.major != 3 or version.minor < 7:
+    minimum_supported = VersionRepresentation(3, 7)
+    if version < minimum_supported:
         raise IncompatibleVersionError(
-            "pysen only supports flake8 version >=3.7, <4. "
+            f"pysen only supports flake8 >= {minimum_supported}, "
             f"version {version} is not supported."
         )
 
@@ -104,15 +106,17 @@ def run(
     setting_path: pathlib.Path,
     sources: Iterable[pathlib.Path],
 ) -> int:
-    check_command_installed("flake8", "--version")
+    check_command_installed(*process_utils.add_python_executable("flake8", "--version"))
     _check_flake8_version()
     targets = [str(d) for d in sources]
     if len(targets) == 0:
         return 0
 
     cmd = ["flake8", "--config", str(setting_path)] + targets
     with change_dir(base_dir):
-        ret, stdout, _ = process_utils.run(cmd, reporter)
+        ret, stdout, _ = process_utils.run(
+            process_utils.add_python_executable(*cmd), reporter
+        )
 
     diagnostics = parse_error_lines(stdout, logger=reporter.logger)
     reporter.report_diagnostics(list(diagnostics))

pysen/ext/isort_wrapper.py

@@ -121,7 +121,7 @@ def run(
     sources: Iterable[pathlib.Path],
     inplace_edit: bool,
 ) -> int:
-    check_command_installed("isort", "--version")
+    check_command_installed(*process_utils.add_python_executable("isort", "--version"))
     version = _get_isort_version()
 
     targets = [str(d) for d in sources]
@@ -136,7 +136,9 @@ def run(
     cmd += targets
 
     with change_dir(base_dir):
-        ret, stdout, _ = process_utils.run(cmd, reporter)
+        ret, stdout, _ = process_utils.run(
+            process_utils.add_python_executable(*cmd), reporter
+        )
 
     diagnostics = parse_error_diffs(stdout, _parse_file_path, logger=reporter.logger)
     reporter.report_diagnostics(list(diagnostics))