Generate pyproject dependencies using dfg

[vyasr]

Description

This PR updates cudf to generate all build, run and test dependencies in pyproject.toml using rapids-dependency-file-generator. In the process, a few additional relevant changes were made:

• dependencies.yaml was refactored to more precisely capture the requirements of the different packages in the repo as well as more accurately separate the build/run/test dependencies. This should facilitate generation of meta.yaml dependencies with rapids-dependency-file-generator if we should choose to do so.
• cudf_kafka's setup.py had as much logic as possible moved to pyproject.toml without having to change the way that extension modules are built. This package is somewhat unique within RAPIDS in having a C++ component that we have not converted to scikit-build because we have no plans for wheels there at this time. As a result, it was not previously converted to a pyproject.toml-driven build (classic Cython builds are not compatible with pyproject.toml-driven builds).
• A tiny unrelated fix to the Java build that removed an outdated CMake flag.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[wence-]

Some very small questions, that said, I think this looks fine as is.

[msadang]

LGTM

[robertmaynard]

Approving CMake comment

[vyasr]

/merge