[FEA] Consolidate python code-style checking and formatting to just use ruff

[wence-]

Is your feature request related to a problem? Please describe.

We have already migrated to using ruff to replace flake8 and pyflakes. A previous barrier to using ruff's isort lint was lack of support for custom sections, that is now supported. Similarly, ruff format (which produces effectively black-compatible formatting) is now in "stable" beta and very usable.

We should consider migrating the separate isort and black configs to use ruff check --fix and ruff format respectively.

The advantage here is that ruff is orders of magnitude faster than both isort and black. For those of us that use format-on-save this is a significant quality of life improvement (formatting a large python file when editing cudf can easily take a few seconds with black).

We would also reduce our tool configuration options.

We might also at the same time consider increasing the default line length from the current (somewhat miserly) 79 characters.

[mroeschke]

+1 pandas recently transitioned black (but not isort) to ruff format and didn't have issues.

[vyasr]

@bdice and I were just talking about wanting to do this.

The main concern on the isort front is that ruff does not support Cython yet, whereas isort does. We'd have to be willing to give that up, or alternatively migrate to ruff over isort only for Cython files (which seems a bit convoluted).

ruff has supported sectioning for quite a while in its isort functionality. I'm fairly certain it was available even at the time that I first enabled ruff. We just need to remember that we also use different sectioning for the different subprojects, so we would need to do something like this:

# pyproject.toml
[tool.ruff]
# standard ruff configuration

# python/cudf/pyproject.toml
[tool.ruff]
extend = "../../pyproject.toml"
sections = ...

I think that should work but have not tested it

[bdice]

We should definitely remove pydocstyle in favor of ruff. Its README says that the project is formally deprecated, and proposes ruff (with rules for "D") as an alternative. pydocstyle is one of the longer-running checks today.

Ruff offers full parity with pydocstyle along with advanced features, better support for the latest Python versions, and ongoing development to ensure a top-notch linting experience. We highly recommend pydocstyle users to switch over to ruff for a seamless transition.

[wence-]

The main concern on the isort front is that ruff does not support Cython yet, whereas isort does. We'd have to be willing to give that up, or alternatively migrate to ruff over isort only for Cython files (which seems a bit convoluted).

TBH, and this is perhaps a minority view, at the moment the cython file formatting is not as uniform any (there's no auto-formatting). So I am perhaps less concerned about dropping isort just for those.

I think that should work but have not tested it

it does (I tried)

[bdice]

Correct, there isn't much available for Cython formatting last I had looked. I do like that we at least have isort for Cython, though. I would just keep that hook in place and transition everything else (besides isort) to ruff.

[vyasr]

Personally I don't think isort does a good enough job on Cython files to be worth keeping around. Consider copying.pyx. There are 6 different groupings of cudf, with a libcpp cimport in the middle. AFAICT any cimport confuses isort and makes it restart grouping. If you manually edit this file and group all the imports together and group all the cimports separately (and move the extraneous libcpp cimport up to the top) then isort will properly sort within the groups, but you have to know to do that. Is the partial sort we get better than nothing? I'm sure that's debatable, but IMHO it gives a false sense of security that our imports are sorted better than they really are. I'd rather just use ruff for everything and wait until ruff adds Cython support or cython-lint starts sorting imports.

[bdice]

Interesting, I didn’t realize it was like that. I have not assessed this in a while. I trust your judgement on that, and am fine with removing isort.