-
Notifications
You must be signed in to change notification settings - Fork 53
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
cEP-0031: Improve Generic Bear Quality
Closes #183
- Loading branch information
Showing
2 changed files
with
255 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,254 @@ | ||
| # Improve Generic Bear Quality | ||
|
|
||
| | Metadata | | | ||
| | -------- | ------------------------------------------- | | ||
| | cEP | 31 | | ||
| | Version | 0.1 | | ||
| | Title | Improve Generic Bear Quality | | ||
| | Authors | Bhushan Khanale <mailto:[email protected]> | | ||
| | Status | Proposed | | ||
| | Type | Feature | | ||
|
|
||
| ## Abstract | ||
|
|
||
| This cEP describes the improvement in the generic bears currently available | ||
| along with the implementation of the new bears as a part of the | ||
| [GSoC 2019 Project][project]. | ||
|
|
||
| ## Introduction | ||
|
|
||
| coala has few generic bears which have the potential to perform better | ||
| fixing some of the issues in them. Some of these issues are as follows: | ||
|
|
||
| - [coala/coala-bears#2092][2092]: Pygmented AnnotationBear | ||
| Pygments is a Python library for syntax highlighting of source code. | ||
| Therefore it contains complete syntax definitions of any relevant | ||
| programming language and a custom parser framework for tokenizing source | ||
| code of all those languages. This gives the opportunity of reducing the | ||
| current complex implementation of the bear into a much simpler one. | ||
|
|
||
| - [coala/coala-bears#644][644]: Ignore doc comments/doc strings | ||
| Comments or docstrings do not have any set standards while the | ||
| IndentationBear does not ignore them in a file. IndentationBear checks for | ||
| correct indentation in the program statements since that is crucial for | ||
| proper code style. Docstrings/comments, on the other hand, do not have any | ||
| such compulsion. So ideally docstrings should be ignored by the bear. coala | ||
| already uses documentation extraction algorithms in DocumentationStyleBear. | ||
| Using the existing algorithms and constructing ignore ranges out of the | ||
| extracted comments inside the IndentationBear, this issue can be solved. | ||
|
|
||
| - [coala/coala-bears#502][502]: MarkdownBear is not very helpful | ||
| On running MarkdownBear all we get is the result message 'The text does not | ||
| comply to the set style.'. This is not very helpful for most users. We will | ||
| be using a plugin to solve this issue. | ||
|
|
||
| - [coala/coala-bears#1897][1897]: Show PEP8 error description | ||
| On running PEP8Bear the default message is 'The code does not comply to | ||
| PEP8.' which is not very helpful for the user. This issue will be solved | ||
| by implementing upstream changes to autopep8. | ||
|
|
||
| - [hhatto/autopep8#227][227]: Create diff for a specific pep8 issue | ||
| User should have a choice of selecting the issue he wants to generate a fix | ||
| for. Although the issue is a part of the autopep8, we will be solving | ||
| this as a part of upstream changes to autopep8. | ||
|
|
||
| Along with fixing these issues, new bears including `OutdatedDependencyBear`, | ||
| `FileExistsBear`, `RegexLintBear`, `FileModeBear` and `RequirementsCheckBear` | ||
| will be implemented. | ||
|
|
||
| ## Implementation of new bears | ||
|
|
||
| ### 1. OutdatedDependencyBear | ||
|
|
||
| Issue: [coala/coala-bears#2445][2445] | ||
|
|
||
| New functions to the existing package manager would be the best way of | ||
| implementing this. | ||
|
|
||
| - For pip requirements, we could use the PyPI JSON API which provides the | ||
| version info by specifying the `package_name`. | ||
|
|
||
| ```py | ||
| class PipRequirement(PackageRequirement): | ||
| def get_version_info(self): | ||
| # This would get the package info in json format | ||
| url = 'https://pypi.python.org/pypi/%s/json' % (self.package,) | ||
|
|
||
| # Get the required data | ||
| data = json.load() | ||
| versions = data['releases'].keys().sort(key = StrictVersion) | ||
| return versions | ||
| ``` | ||
|
|
||
| - For npm dependencies, we can use the npm cli command `npm outdated` which | ||
| produces an output as below: | ||
|
|
||
| ``` | ||
| $ npm outdated | ||
| Package Current Wanted Latest Location | ||
| glob 5.0.15 5.0.15 6.0.1 test-outdated-output | ||
| nothingness 0.0.3 git git test-outdated-output | ||
| npm 3.5.1 3.5.2 3.5.1 test-outdated-output | ||
| local-dev 0.0.3 linked linked test-outdated-output | ||
| once 1.3.2 1.3.3 1.3.3 test-outdated-output | ||
| ``` | ||
|
|
||
| The output can be directly parsed and can be used to get the list of all | ||
| outdated dependencies. | ||
|
|
||
| ### 2. FileExistBear | ||
|
|
||
| Issue: [coala/coala-bears#527][527] | ||
|
|
||
| The bear will be responsible to check if the files exist in the project | ||
| directory. The name of the files can be passed as a regex to the bear, to | ||
| which the bear will check their existence. | ||
|
|
||
| ```py | ||
| class FileExistBear(LocalBear): | ||
| def run(self, required_files: tuple,): | ||
| for file in required_files: | ||
| if not os.path.is_file(): | ||
| message = 'File {} not found.'.format(file) | ||
| yield Result.from_values(origin=self, message=message,) | ||
| ``` | ||
|
|
||
| ### 3. RegexLintBear | ||
|
|
||
| Issue: [coala/coala-bears#1532][1532] | ||
|
|
||
| The task of the bear is to check the regex in strings. This can be done using | ||
| the AnnotationBear to detect all the regex strings and then check for the valid | ||
| regex through the prepared algorithm for each type. | ||
|
|
||
| For Python, the bear can be written as follows: | ||
|
|
||
| ```py | ||
| def is_regex(code, language): | ||
| """ | ||
| Returns true if the string is a regex. | ||
| """ | ||
|
|
||
| class RegexLintBear(LocalBear): | ||
| def run(self, file, filename, language,): | ||
| strings = run_local_bear(AnnotationBear) | ||
| for string in strings: | ||
| if is_regex(string, language): | ||
| try: | ||
| re.compile(string) | ||
| except re.error as Exception: | ||
| message = 'This is not a valid regex.' | ||
| yield Result.from_values(origin=self, message=message) | ||
| ``` | ||
|
|
||
| ### 4. FileModeBear | ||
|
|
||
| Issue: [coala/coala-bears#2370][2370] | ||
|
|
||
| The bear will check the permissions on the files provided by the user | ||
| and to ensure that the file permissions are the one that is expected. | ||
|
|
||
| The bear will be used as follows: | ||
|
|
||
| ``` | ||
| [all.mode] | ||
| bears = FileModeBear | ||
| filemode = rw | ||
| [all.shell.mode] | ||
| bears = FileModeBear | ||
| filemode = rwx | ||
| ``` | ||
|
|
||
| The bear will first find the permissions of the files specified by the user and | ||
| then if the permissions are not suitable, the bear will try to change those | ||
| permissions into the expected ones. If the bear doesn't have enough permissions | ||
| to do so then the bear will let the user know about this. | ||
|
|
||
| ```py | ||
| class FileModeBear(LocalBear): | ||
| def run(self, file, filename, filemode,): | ||
| stats = os.stat(filename) | ||
| stats_read = bool(stats.st_mode & stat.S_IRUSR) | ||
| stats_write = bool(stats.st_mode & stat.S_IWUSR) | ||
|
|
||
| mode = {'r': stats_read, 'w': stats_write} | ||
| ch_mode = {'r': stat.S_IRUSR, 'w': stat.S_IWUSR} | ||
|
|
||
| # Similarly, by bitwise operations on the st_mode, the relevant | ||
| # information can be found out. | ||
|
|
||
| change_mode = 0 | ||
| for char in filemode: | ||
| if not mode[char]: | ||
| change_mode += ch_mode[char] | ||
| if not change_mode: | ||
| os.chmod(change_mode) | ||
| ``` | ||
|
|
||
| ### 5. RequirementsCheckBear | ||
|
|
||
| Issue: [coala/coala-bears#1113][1113] | ||
|
|
||
| The bear will be focused on Python only since they are most prone for | ||
| conflicting requirements. | ||
|
|
||
| The implementation is based on the recursive check for the requirements of each | ||
| package in requirement.txt file of the project. The requirements are checked | ||
| through PyPI's JSON API. | ||
|
|
||
| ```py | ||
| class RequirementsCheckBear(LocalBear): | ||
| def collect_requirements(): | ||
| """ | ||
| This will get the requirements through the requirements.txt file and | ||
| through requirements folder and pip files. | ||
| """ | ||
| packages = get_packages() | ||
|
|
||
| for package in packages: | ||
| requirements = get_requirements(package) | ||
| packages.append(r for r in requirements) | ||
|
|
||
| return packages | ||
|
|
||
| def get_requirements(package_name): | ||
| """ | ||
| This will get the requirements of a package through the PyPI's JSON API. | ||
| """ | ||
| url = 'https://pypi.python.org/pypi/%s/json' % (package_name,) | ||
|
|
||
| data = requests.get(url).json() | ||
|
|
||
| return data['info']['requires_dist'] | ||
|
|
||
| def run(self, file, filaname,): | ||
| packages = get_requirements() | ||
| conflicts = detect_conflicts() | ||
| if conflicts: | ||
| for conflict in conflicts: | ||
| message = '{} is conflicting package.'.format(conflict) | ||
| yield Result.from_values(origin=self, message=message,) | ||
| ``` | ||
|
|
||
| [project]: https://summerofcode.withgoogle.com/projects/#4866569388163072 | ||
|
|
||
| [227]: https://github.com/hhatto/autopep8/issues/227 | ||
|
|
||
| [502]: https://github.com/coala/coala-bears/issues/502 | ||
|
|
||
| [527]: https://github.com/coala/coala-bears/issues/527 | ||
|
|
||
| [551]: https://github.com/coala/projects/issues/551 | ||
|
|
||
| [644]: https://github.com/coala/coala-bears/issues/644 | ||
|
|
||
| [1113]: https://github.com/coala/coala-bears/issues/1113 | ||
|
|
||
| [1532]: https://github.com/coala/coala-bears/issues/1532 | ||
|
|
||
| [1897]: https://github.com/coala/coala-bears/issues/1897 | ||
|
|
||
| [2092]: https://github.com/coala/coala-bears/issues/2092 | ||
|
|
||
| [2445]: https://github.com/coala/coala-bears/issues/2445 |