Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ci] warnings from mypy #3867

Open
jameslamb opened this issue Jan 27, 2021 · 18 comments
Open

[ci] warnings from mypy #3867

jameslamb opened this issue Jan 27, 2021 · 18 comments

Comments

@jameslamb
Copy link
Collaborator

jameslamb commented Jan 27, 2021

Summary

mypy shows some issues in LightGBM's Python package.

mypy \
    --exclude='python-package/compile/|python-package/build' \
    --ignore-missing-imports \
    python-package/
18 errors in 4 files (click me)
python-package/lightgbm/compat.py:12: error: Name 'Series' already defined (possibly by an import)
python-package/lightgbm/compat.py:17: error: Name 'DataFrame' already defined (possibly by an import)
python-package/lightgbm/compat.py:49: error: Name 'DataTable' already defined on line 42
python-package/setup.py:325: error: Argument "verbose" to "copy_file" has incompatible type "int"; expected "bool"
python-package/lightgbm/sklearn.py:797: error: Value of type "Optional[str]" is not indexable
python-package/lightgbm/sklearn.py:797: error: Item "None" of "Optional[str]" has no attribute "find"
python-package/lightgbm/sklearn.py:798: error: Value of type "Optional[str]" is not indexable
python-package/lightgbm/sklearn.py:798: error: Item "None" of "Optional[str]" has no attribute "find"
python-package/lightgbm/sklearn.py:870: error: Value of type "Optional[str]" is not indexable
python-package/lightgbm/sklearn.py:870: error: Item "None" of "Optional[str]" has no attribute "find"
python-package/lightgbm/sklearn.py:871: error: Value of type "Optional[str]" is not indexable
python-package/lightgbm/sklearn.py:871: error: Item "None" of "Optional[str]" has no attribute "find"
python-package/lightgbm/sklearn.py:993: error: Value of type "Optional[str]" is not indexable
python-package/lightgbm/sklearn.py:993: error: Item "None" of "Optional[str]" has no attribute "find"
python-package/lightgbm/sklearn.py:994: error: Value of type "Optional[str]" is not indexable
python-package/lightgbm/sklearn.py:994: error: Item "None" of "Optional[str]" has no attribute "find"
python-package/lightgbm/dask.py:70: error: Incompatible return value type (got "Optional[int]", expected "int")
python-package/lightgbm/dask.py:93: error: Need type annotation for 'lightgbm_ports' (hint: "lightgbm_ports: Set[<type>] = ...")
Found 18 errors in 4 files (checked 10 source files)

Motivation

mypy is a static analysis tool for type checking Python code. It can catch issues not caught by other linters and unit tests, like:

  • a value can be None but other code acts on it as if it is not None
  • code calls a method that does not exist for a type (e.g. num.lower() when num is an int)
  • unused variables
  • possible type mismatches, e.g. fun(other_fun()) being problematic if fun() expects a data frame and other_fun() returns a list

LightGBM's Python package recently started on the path to adding type hints (#3756), so this project can start to get more value out of mypy.

Description

If you're interested in contributing, you do not need to fix all of the mypy issues at once. Pull requests that address any of the issues found by mypy are welcome!

To test your fixes, install the latest version of mypy, then run it over the code.

pip install --upgrade mypy
mypy \
    --exclude='python-package/compile/|python-package/build' \
    --ignore-missing-imports \
    python-package/
@jameslamb
Copy link
Collaborator Author

Up to 30 errors as of d6ebd06. This will happen...adding more type hints gives mypy more stuff to check, so we should expect some volatility in that number as #3756 is slowly addressed.

Also, some of the issues below will be fixed when #3916 is merged.

30 errors in 3 files (check 10 source files)
python-package/lightgbm/compat.py:14: error: Name 'pd_Series' already defined (possibly by an import)
python-package/lightgbm/compat.py:19: error: Name 'pd_DataFrame' already defined (possibly by an import)
python-package/lightgbm/compat.py:52: error: Name 'dt_DataTable' already defined on line 45
python-package/lightgbm/compat.py:127: error: Name 'dask_Array' already defined (possibly by an import)
python-package/lightgbm/compat.py:132: error: Name 'dask_DataFrame' already defined (possibly by an import)
python-package/lightgbm/compat.py:137: error: Name 'dask_Series' already defined (possibly by an import)
python-package/lightgbm/callback.py:28: error: First argument to namedtuple() should be 'CallbackEnv', not 'LightGBMCallbackEnv'
python-package/lightgbm/dask.py:303: error: Incompatible return value type (got "Dict[str, Any]", expected "LGBMModel")
python-package/lightgbm/dask.py:471: error: "_DaskLGBMModel" has no attribute "_other_params"
python-package/lightgbm/dask.py:489: error: "_DaskLGBMModel" has no attribute "get_params"
python-package/lightgbm/dask.py:503: error: "_DaskLGBMModel" has no attribute "set_params"
python-package/lightgbm/dask.py:509: error: "_DaskLGBMModel" has no attribute "get_params"
python-package/lightgbm/dask.py:518: error: Item "_DaskLGBMModel" of "Union[_DaskLGBMModel, LGBMModel]" has no attribute "get_params"
python-package/lightgbm/dask.py:580: error: Item "None" of "Optional[str]" has no attribute "partition"
python-package/lightgbm/dask.py:595: error: Signature of "fit" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:595: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:603: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMClassifier")
python-package/lightgbm/dask.py:629: error: Signature of "predict" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:629: error: Signature of "predict" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:647: error: Signature of "predict_proba" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:731: error: Item "None" of "Optional[str]" has no attribute "partition"
python-package/lightgbm/dask.py:746: error: Signature of "fit" incompatible with supertype "LGBMRegressor"
python-package/lightgbm/dask.py:746: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:754: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMRegressor")
python-package/lightgbm/dask.py:780: error: Signature of "predict" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:863: error: Item "None" of "Optional[str]" has no attribute "partition"
python-package/lightgbm/dask.py:878: error: Signature of "fit" incompatible with supertype "LGBMRanker"
python-package/lightgbm/dask.py:878: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:891: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMRanker")
python-package/lightgbm/dask.py:921: error: Signature of "predict" incompatible with supertype "LGBMModel"
Found 30 errors in 3 files (checked 10 source files

@AXRegister
Copy link

I was looking for some more information on how to test my fixes. Is there something more extensive to ensure I'm not breaking anything, other than running mypy? I'm quite new to python, and I just wanted to ensure I'm making correct changes. Any advice on this one?

@jameslamb
Copy link
Collaborator Author

Hi @AXRegister , thanks very much for your interest in contributing!

To test locally, you can try installing the package locally and running the unit tests. Like this:

# install the Python package
git submodule init
git submodule update --recursive
cd python-package
python setup.py install

# test the Python package
cd ..
pytest tests/python_package_test

If those work ok, you can have some confidence that your changes won't break existing functionality. When you open a pull request, a much more extensive set of checks (checking different combinations of Python version, compiler, and operating system) will run automatically and might suggest additional changes.

If you run into any problems, please either ask a question here or open a pull request and ask questions there.

@jameslamb
Copy link
Collaborator Author

Up to 64 errors as of b5502d1.

Found 64 errors in 3 files (checked 10 source files)
python-package/setup.py:43: error: "str" not callable
python-package/setup.py:60: error: Argument 2 to "create_tree" has incompatible type "str"; expected "List[str]"
python-package/setup.py:76: error: Argument "verbose" to "copy_file" has incompatible type "int"; expected "bool"
python-package/setup.py:79: error: Argument "verbose" to "copy_file" has incompatible type "int"; expected "bool"
python-package/setup.py:82: error: Argument "verbose" to "copy_file" has incompatible type "int"; expected "bool"
python-package/setup.py:85: error: Argument "verbose" to "copy_file" has incompatible type "int"; expected "bool"
python-package/setup.py:91: error: Argument "verbose" to "copy_file" has incompatible type "int"; expected "bool"
python-package/setup.py:260: error: Argument "integrated_opencl" to "copy_files" has incompatible type "int"; expected "bool"
python-package/setup.py:260: error: Argument "use_gpu" to "copy_files" has incompatible type "int"; expected "bool"
python-package/setup.py:261: error: Argument "use_mingw" to "compile_cpp" has incompatible type "int"; expected "bool"
python-package/setup.py:261: error: Argument "use_gpu" to "compile_cpp" has incompatible type "int"; expected "bool"
python-package/setup.py:261: error: Argument "use_cuda" to "compile_cpp" has incompatible type "int"; expected "bool"
python-package/setup.py:261: error: Argument "use_mpi" to "compile_cpp" has incompatible type "int"; expected "bool"
python-package/setup.py:262: error: Argument "use_hdfs" to "compile_cpp" has incompatible type "int"; expected "bool"
python-package/setup.py:265: error: Argument "nomp" to "compile_cpp" has incompatible type "int"; expected "bool"
python-package/setup.py:265: error: Argument "bit32" to "compile_cpp" has incompatible type "int"; expected "bool"
python-package/setup.py:265: error: Argument "integrated_opencl" to "compile_cpp" has incompatible type "int"; expected "bool"
python-package/lightgbm/plotting.py:235: error: Value of type "Union[int, str, None]" is not indexable
python-package/lightgbm/plotting.py:235: error: Unsupported left operand type for - ("str")
python-package/lightgbm/plotting.py:235: note: Both left and right operands are unions
python-package/lightgbm/plotting.py:236: error: Unsupported operand types for / ("str" and "int")
python-package/lightgbm/plotting.py:236: note: Left operand is of type "Union[Any, str]"
python-package/lightgbm/plotting.py:236: error: Value of type "Union[int, str, None]" is not indexable
python-package/lightgbm/plotting.py:248: error: Value of type "Union[int, str, None]" is not indexable
python-package/lightgbm/plotting.py:248: error: Unsupported left operand type for - ("str")
python-package/lightgbm/plotting.py:248: note: Both left and right operands are unions
python-package/lightgbm/plotting.py:249: error: Value of type "Union[int, str, None]" is not indexable
python-package/lightgbm/plotting.py:350: error: Incompatible types in assignment (expression has type "Iterator[Any]", variable has type "Optional[List[str]]")
python-package/lightgbm/plotting.py:354: error: Incompatible types in assignment (expression has type "Iterator[str]", variable has type "Optional[List[str]]")
python-package/lightgbm/plotting.py:356: error: No overload variant of "next" matches argument type "Optional[List[str]]"
python-package/lightgbm/plotting.py:356: note: Possible overload variant:
python-package/lightgbm/plotting.py:356: note:     def [_T] next(__i, Iterator[_T]) -> _T
python-package/lightgbm/plotting.py:356: note:     <1 more non-matching overload not shown>
python-package/lightgbm/plotting.py:371: error: Item "None" of "Optional[List[str]]" has no attribute "__iter__" (not iterable)
python-package/lightgbm/dask.py:326: error: Invalid index type "Optional[str]" for "defaultdict[str, Set[int]]"; expected type "str"
python-package/lightgbm/dask.py:335: error: Need type annotation for "host_to_port"
python-package/lightgbm/dask.py:568: error: Argument 1 to "append" of "list" has incompatible type "Tuple[List[Union[ndarray, Any]], List[Union[ndarray, Any]]]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:571: error: Value of type "_DatasetNames" is not indexable
python-package/lightgbm/dask.py:572: error: Value of type "_DatasetNames" is not indexable
python-package/lightgbm/dask.py:590: error: Argument 1 to "append" of "list" has incompatible type "List[Union[ndarray, Any]]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:592: error: "_DatasetNames" has no attribute "append"
python-package/lightgbm/dask.py:608: error: Argument 1 to "append" of "list" has incompatible type "List[Union[ndarray, Any]]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:610: error: "_DatasetNames" has no attribute "append"
python-package/lightgbm/dask.py:626: error: Argument 1 to "append" of "list" has incompatible type "List[Union[ndarray, Any]]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:628: error: "_DatasetNames" has no attribute "append"
python-package/lightgbm/dask.py:647: error: Incompatible return value type (got "Dict[str, Union[ndarray, Any]]", expected "LGBMModel")
python-package/lightgbm/dask.py:662: error: "Dict[str, Union[ndarray, Any]]" has no attribute "result"
python-package/lightgbm/dask.py:707: error: Argument "worker_addresses" to "_machines_to_worker_map" has incompatible type "KeysView[Any]"; expected "List[str]"
python-package/lightgbm/dask.py:921: error: "_DaskLGBMModel" has no attribute "_other_params"
python-package/lightgbm/dask.py:952: error: "_DaskLGBMModel" has no attribute "get_params"
python-package/lightgbm/dask.py:975: error: "_DaskLGBMModel" has no attribute "set_params"
python-package/lightgbm/dask.py:981: error: "_DaskLGBMModel" has no attribute "get_params"
python-package/lightgbm/dask.py:990: error: Item "_DaskLGBMModel" of "Union[_DaskLGBMModel, LGBMModel]" has no attribute "get_params"
python-package/lightgbm/dask.py:1052: error: Item "None" of "Optional[str]" has no attribute "partition"
python-package/lightgbm/dask.py:1066: error: Signature of "fit" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:1066: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1085: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMClassifier")
python-package/lightgbm/dask.py:1128: error: Signature of "predict" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:1128: error: Signature of "predict" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1146: error: Signature of "predict_proba" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:1230: error: Item "None" of "Optional[str]" has no attribute "partition"
python-package/lightgbm/dask.py:1243: error: Signature of "fit" incompatible with supertype "LGBMRegressor"
python-package/lightgbm/dask.py:1243: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1261: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMRegressor")
python-package/lightgbm/dask.py:1306: error: Signature of "predict" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1389: error: Item "None" of "Optional[str]" has no attribute "partition"
python-package/lightgbm/dask.py:1403: error: Signature of "fit" incompatible with supertype "LGBMRanker"
python-package/lightgbm/dask.py:1403: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1424: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMRanker")
python-package/lightgbm/dask.py:1468: error: Signature of "predict" incompatible with supertype "LGBMModel"
Found 64 errors in 3 files (checked 10 source files)

@jameslamb
Copy link
Collaborator Author

71 errors as of a2b60e8

Found 71 errors in 5 files (checked 10 source files)
python-package/setup.py:43: error: "Path" not callable
python-package/lightgbm/engine.py:231: error: Incompatible types in assignment (expression has type "Set[<nothing>]", variable has type "Optional[List[Callable[..., Any]]]")
python-package/lightgbm/engine.py:235: error: Incompatible types in assignment (expression has type "Set[Callable[..., Any]]", variable has type "Optional[List[Callable[..., Any]]]")
python-package/lightgbm/engine.py:247: error: Item "List[Callable[..., Any]]" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:247: error: Item "None" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:249: error: Item "List[Callable[..., Any]]" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:249: error: Item "None" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:252: error: Item "List[Callable[..., Any]]" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:252: error: Item "None" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:257: error: Item "List[Callable[..., Any]]" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:257: error: Item "None" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:262: error: Item "List[Callable[..., Any]]" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:262: error: Item "None" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:264: error: Item "None" of "Optional[List[Callable[..., Any]]]" has no attribute "__iter__" (not iterable)
python-package/lightgbm/engine.py:265: error: Unsupported left operand type for - ("List[Callable[..., Any]]")
python-package/lightgbm/engine.py:265: error: Unsupported left operand type for - ("None")
python-package/lightgbm/engine.py:265: note: Left operand is of type "Optional[List[Callable[..., Any]]]"
python-package/lightgbm/engine.py:266: error: Incompatible types in assignment (expression has type "List[Union[Callable[..., Any], Any]]", variable has type "Set[Union[Callable[..., Any], Any]]")
python-package/lightgbm/engine.py:310: error: Incompatible types in assignment (expression has type "float", variable has type "List[Any]")
python-package/lightgbm/sklearn.py:509: error: Need type annotation for "_other_params" (hint: "_other_params: Dict[<type>, <type>] = ...")
python-package/lightgbm/plotting.py:243: error: Value of type "Union[int, str, None]" is not indexable
python-package/lightgbm/plotting.py:243: error: Unsupported left operand type for - ("str")
python-package/lightgbm/plotting.py:243: note: Both left and right operands are unions
python-package/lightgbm/plotting.py:244: error: Unsupported operand types for / ("str" and "int")
python-package/lightgbm/plotting.py:244: note: Left operand is of type "Union[Any, str]"
python-package/lightgbm/plotting.py:244: error: Value of type "Union[int, str, None]" is not indexable
python-package/lightgbm/plotting.py:256: error: Value of type "Union[int, str, None]" is not indexable
python-package/lightgbm/plotting.py:256: error: Unsupported left operand type for - ("str")
python-package/lightgbm/plotting.py:256: note: Both left and right operands are unions
python-package/lightgbm/plotting.py:257: error: Value of type "Union[int, str, None]" is not indexable
python-package/lightgbm/plotting.py:359: error: Incompatible types in assignment (expression has type "Iterator[Any]", variable has type "Optional[List[str]]")
python-package/lightgbm/plotting.py:363: error: Incompatible types in assignment (expression has type "Iterator[str]", variable has type "Optional[List[str]]")
python-package/lightgbm/plotting.py:365: error: No overload variant of "next" matches argument type "Optional[List[str]]"
python-package/lightgbm/plotting.py:365: note: Possible overload variant:
python-package/lightgbm/plotting.py:365: note:     def [_T] next(__i, Iterator[_T]) -> _T
python-package/lightgbm/plotting.py:365: note:     <1 more non-matching overload not shown>
python-package/lightgbm/plotting.py:382: error: Item "None" of "Optional[List[str]]" has no attribute "__iter__" (not iterable)
python-package/lightgbm/dask.py:99: error: Invalid index type "Optional[str]" for "Dict[str, _HostWorkers]"; expected type "str"
python-package/lightgbm/dask.py:383: error: Invalid index type "Optional[str]" for "defaultdict[str, Set[int]]"; expected type "str"
python-package/lightgbm/dask.py:587: error: Argument 1 to "append" of "list" has incompatible type "Tuple[List[Any], List[Any]]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:590: error: Value of type "_DatasetNames" is not indexable
python-package/lightgbm/dask.py:591: error: Value of type "_DatasetNames" is not indexable
python-package/lightgbm/dask.py:609: error: Argument 1 to "append" of "list" has incompatible type "List[Any]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:611: error: "_DatasetNames" has no attribute "append"
python-package/lightgbm/dask.py:627: error: Argument 1 to "append" of "list" has incompatible type "List[Any]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:629: error: "_DatasetNames" has no attribute "append"
python-package/lightgbm/dask.py:645: error: Argument 1 to "append" of "list" has incompatible type "List[Any]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:647: error: "_DatasetNames" has no attribute "append"
python-package/lightgbm/dask.py:666: error: Incompatible return value type (got "Dict[str, Any]", expected "LGBMModel")
python-package/lightgbm/dask.py:681: error: "Dict[str, Any]" has no attribute "result"
python-package/lightgbm/dask.py:726: error: Argument "worker_addresses" to "_machines_to_worker_map" has incompatible type "KeysView[Any]"; expected "List[str]"
python-package/lightgbm/dask.py:933: error: Need type annotation for "out"
python-package/lightgbm/dask.py:981: error: Incompatible types in assignment (expression has type "Tuple[Any, Any]", variable has type "Tuple[Any]")
python-package/lightgbm/dask.py:1012: error: "_DaskLGBMModel" has no attribute "_other_params"
python-package/lightgbm/dask.py:1043: error: "_DaskLGBMModel" has no attribute "get_params"
python-package/lightgbm/dask.py:1066: error: "_DaskLGBMModel" has no attribute "set_params"
python-package/lightgbm/dask.py:1072: error: "_DaskLGBMModel" has no attribute "get_params"
python-package/lightgbm/dask.py:1081: error: Item "_DaskLGBMModel" of "Union[_DaskLGBMModel, LGBMModel]" has no attribute "get_params"
python-package/lightgbm/dask.py:1111: error: Incompatible default for argument "silent" (default has type "str", argument has type "bool")
python-package/lightgbm/dask.py:1143: error: Item "None" of "Optional[str]" has no attribute "partition"
python-package/lightgbm/dask.py:1157: error: Signature of "fit" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:1157: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1176: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMClassifier")
python-package/lightgbm/dask.py:1219: error: Signature of "predict" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:1219: error: Signature of "predict" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1238: error: Signature of "predict_proba" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:1291: error: Incompatible default for argument "silent" (default has type "str", argument has type "bool")
python-package/lightgbm/dask.py:1323: error: Item "None" of "Optional[str]" has no attribute "partition"
python-package/lightgbm/dask.py:1336: error: Signature of "fit" incompatible with supertype "LGBMRegressor"
python-package/lightgbm/dask.py:1336: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1354: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMRegressor")
python-package/lightgbm/dask.py:1399: error: Signature of "predict" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1451: error: Incompatible default for argument "silent" (default has type "str", argument has type "bool")
python-package/lightgbm/dask.py:1483: error: Item "None" of "Optional[str]" has no attribute "partition"
python-package/lightgbm/dask.py:1497: error: Signature of "fit" incompatible with supertype "LGBMRanker"
python-package/lightgbm/dask.py:1497: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1518: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMRanker")
python-package/lightgbm/dask.py:1562: error: Signature of "predict" incompatible with supertype "LGBMModel"
Found 71 errors in 5 files (checked 10 source files)

@jameslamb
Copy link
Collaborator Author

This issue has been open with the label good first issue for about 10 months, and hasn't attracted much attention from outside contributors.

I opened some PRs for this tonight, and plan to finish it over the next few days. I'd like to enable mypy in the lint job in CI soon, to help catch bugs caused by future refactorings.

mypy --ignore-missing-imports python-package/ || true

@jameslamb
Copy link
Collaborator Author

59 errors as of f57ef6f

Found 59 errors in 3 files (checked 10 source files)
python-package/lightgbm/engine.py:226: error: Incompatible types in assignment (expression has type "Set[<nothing>]", variable has type "Optional[List[Callable[..., Any]]]")
python-package/lightgbm/engine.py:230: error: Incompatible types in assignment (expression has type "Set[Callable[..., Any]]", variable has type "Optional[List[Callable[..., Any]]]")
python-package/lightgbm/engine.py:242: error: Item "List[Callable[..., Any]]" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:242: error: Item "None" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:244: error: Item "List[Callable[..., Any]]" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:244: error: Item "None" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:247: error: Item "List[Callable[..., Any]]" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:247: error: Item "None" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:252: error: Item "List[Callable[..., Any]]" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:252: error: Item "None" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:254: error: Item "None" of "Optional[List[Callable[..., Any]]]" has no attribute "__iter__" (not iterable)
python-package/lightgbm/engine.py:255: error: Unsupported left operand type for - ("List[Callable[..., Any]]")
python-package/lightgbm/engine.py:255: error: Unsupported left operand type for - ("None")
python-package/lightgbm/engine.py:255: note: Left operand is of type "Optional[List[Callable[..., Any]]]"
python-package/lightgbm/engine.py:256: error: Incompatible types in assignment (expression has type "List[Union[Callable[..., Any], Any]]", variable has type "Set[Union[Callable[..., Any], Any]]")
python-package/lightgbm/engine.py:300: error: Incompatible types in assignment (expression has type "float", variable has type "List[Any]")
python-package/lightgbm/sklearn.py:521: error: Need type annotation for "_other_params" (hint: "_other_params: Dict[<type>, <type>] = ...")
python-package/lightgbm/sklearn.py:627: error: Incompatible types in assignment (expression has type "None", variable has type "_ObjectiveFunctionWrapper")
python-package/lightgbm/sklearn.py:896: error: "None" has no attribute "current_iteration"
python-package/lightgbm/sklearn.py:907: error: "None" has no attribute "current_iteration"
python-package/lightgbm/sklearn.py:961: error: Item "None" of "Optional[str]" has no attribute "replace"
python-package/lightgbm/sklearn.py:1028: error: Item "None" of "Optional[str]" has no attribute "replace"
python-package/lightgbm/sklearn.py:1127: error: Item "None" of "Optional[str]" has no attribute "replace"
python-package/lightgbm/dask.py:99: error: Invalid index type "Optional[str]" for "Dict[str, _HostWorkers]"; expected type "str"
python-package/lightgbm/dask.py:383: error: Invalid index type "Optional[str]" for "defaultdict[str, Set[int]]"; expected type "str"
python-package/lightgbm/dask.py:587: error: Argument 1 to "append" of "list" has incompatible type "Tuple[List[Any], List[Any]]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:590: error: Value of type "_DatasetNames" is not indexable
python-package/lightgbm/dask.py:591: error: Value of type "_DatasetNames" is not indexable
python-package/lightgbm/dask.py:609: error: Argument 1 to "append" of "list" has incompatible type "List[Any]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:611: error: "_DatasetNames" has no attribute "append"
python-package/lightgbm/dask.py:627: error: Argument 1 to "append" of "list" has incompatible type "List[Any]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:629: error: "_DatasetNames" has no attribute "append"
python-package/lightgbm/dask.py:645: error: Argument 1 to "append" of "list" has incompatible type "List[Any]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:647: error: "_DatasetNames" has no attribute "append"
python-package/lightgbm/dask.py:666: error: Incompatible return value type (got "Dict[str, Any]", expected "LGBMModel")
python-package/lightgbm/dask.py:681: error: "Dict[str, Any]" has no attribute "result"
python-package/lightgbm/dask.py:726: error: Argument "worker_addresses" to "_machines_to_worker_map" has incompatible type "KeysView[Any]"; expected "List[str]"
python-package/lightgbm/dask.py:984: error: Incompatible types in assignment (expression has type "Tuple[Any, Any]", variable has type "Tuple[Any]")
python-package/lightgbm/dask.py:1015: error: "_DaskLGBMModel" has no attribute "_other_params"
python-package/lightgbm/dask.py:1046: error: "_DaskLGBMModel" has no attribute "get_params"
python-package/lightgbm/dask.py:1069: error: "_DaskLGBMModel" has no attribute "set_params"
python-package/lightgbm/dask.py:1075: error: "_DaskLGBMModel" has no attribute "get_params"
python-package/lightgbm/dask.py:1084: error: Item "_DaskLGBMModel" of "Union[_DaskLGBMModel, LGBMModel]" has no attribute "get_params"
python-package/lightgbm/dask.py:1144: error: Item "None" of "Optional[str]" has no attribute "partition"
python-package/lightgbm/dask.py:1158: error: Signature of "fit" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:1158: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1177: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMClassifier")
python-package/lightgbm/dask.py:1225: error: Signature of "predict" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:1225: error: Signature of "predict" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1244: error: Signature of "predict_proba" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:1327: error: Item "None" of "Optional[str]" has no attribute "partition"
python-package/lightgbm/dask.py:1340: error: Signature of "fit" incompatible with supertype "LGBMRegressor"
python-package/lightgbm/dask.py:1340: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1358: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMRegressor")
python-package/lightgbm/dask.py:1408: error: Signature of "predict" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1490: error: Item "None" of "Optional[str]" has no attribute "partition"
python-package/lightgbm/dask.py:1504: error: Signature of "fit" incompatible with supertype "LGBMRanker"
python-package/lightgbm/dask.py:1504: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1525: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMRanker")
python-package/lightgbm/dask.py:1574: error: Signature of "predict" incompatible with supertype "LGBMModel"

@jameslamb
Copy link
Collaborator Author

53 errors as of e048a6b

Found 53 errors in 3 files (checked 10 source files)
python-package/lightgbm/engine.py:226: error: Incompatible types in assignment (expression has type "Set[<nothing>]", variable has type "Optional[List[Callable[..., Any]]]")
python-package/lightgbm/engine.py:230: error: Incompatible types in assignment (expression has type "Set[Callable[..., Any]]", variable has type "Optional[List[Callable[..., Any]]]")
python-package/lightgbm/engine.py:242: error: Item "List[Callable[..., Any]]" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:242: error: Item "None" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:244: error: Item "List[Callable[..., Any]]" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:244: error: Item "None" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:247: error: Item "List[Callable[..., Any]]" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:247: error: Item "None" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:252: error: Item "List[Callable[..., Any]]" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:252: error: Item "None" of "Optional[List[Callable[..., Any]]]" has no attribute "add"
python-package/lightgbm/engine.py:254: error: Item "None" of "Optional[List[Callable[..., Any]]]" has no attribute "__iter__" (not iterable)
python-package/lightgbm/engine.py:255: error: Unsupported left operand type for - ("List[Callable[..., Any]]")
python-package/lightgbm/engine.py:255: error: Unsupported left operand type for - ("None")
python-package/lightgbm/engine.py:255: note: Left operand is of type "Optional[List[Callable[..., Any]]]"
python-package/lightgbm/engine.py:256: error: Incompatible types in assignment (expression has type "List[Union[Callable[..., Any], Any]]", variable has type "Set[Union[Callable[..., Any], Any]]")
python-package/lightgbm/engine.py:300: error: Incompatible types in assignment (expression has type "float", variable has type "List[Any]")
python-package/lightgbm/sklearn.py:627: error: Incompatible types in assignment (expression has type "None", variable has type "_ObjectiveFunctionWrapper")
python-package/lightgbm/dask.py:99: error: Invalid index type "Optional[str]" for "Dict[str, _HostWorkers]"; expected type "str"
python-package/lightgbm/dask.py:383: error: Invalid index type "Optional[str]" for "defaultdict[str, Set[int]]"; expected type "str"
python-package/lightgbm/dask.py:587: error: Argument 1 to "append" of "list" has incompatible type "Tuple[List[Any], List[Any]]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:590: error: Value of type "_DatasetNames" is not indexable
python-package/lightgbm/dask.py:591: error: Value of type "_DatasetNames" is not indexable
python-package/lightgbm/dask.py:609: error: Argument 1 to "append" of "list" has incompatible type "List[Any]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:611: error: "_DatasetNames" has no attribute "append"
python-package/lightgbm/dask.py:627: error: Argument 1 to "append" of "list" has incompatible type "List[Any]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:629: error: "_DatasetNames" has no attribute "append"
python-package/lightgbm/dask.py:645: error: Argument 1 to "append" of "list" has incompatible type "List[Any]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:647: error: "_DatasetNames" has no attribute "append"
python-package/lightgbm/dask.py:666: error: Incompatible return value type (got "Dict[str, Any]", expected "LGBMModel")
python-package/lightgbm/dask.py:681: error: "Dict[str, Any]" has no attribute "result"
python-package/lightgbm/dask.py:726: error: Argument "worker_addresses" to "_machines_to_worker_map" has incompatible type "KeysView[Any]"; expected "List[str]"
python-package/lightgbm/dask.py:984: error: Incompatible types in assignment (expression has type "Tuple[Any, Any]", variable has type "Tuple[Any]")
python-package/lightgbm/dask.py:1015: error: "_DaskLGBMModel" has no attribute "_other_params"
python-package/lightgbm/dask.py:1046: error: "_DaskLGBMModel" has no attribute "get_params"
python-package/lightgbm/dask.py:1069: error: "_DaskLGBMModel" has no attribute "set_params"
python-package/lightgbm/dask.py:1075: error: "_DaskLGBMModel" has no attribute "get_params"
python-package/lightgbm/dask.py:1084: error: Item "_DaskLGBMModel" of "Union[_DaskLGBMModel, LGBMModel]" has no attribute "get_params"
python-package/lightgbm/dask.py:1144: error: Item "None" of "Optional[str]" has no attribute "partition"
python-package/lightgbm/dask.py:1158: error: Signature of "fit" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:1158: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1177: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMClassifier")
python-package/lightgbm/dask.py:1225: error: Signature of "predict" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:1225: error: Signature of "predict" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1244: error: Signature of "predict_proba" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:1327: error: Item "None" of "Optional[str]" has no attribute "partition"
python-package/lightgbm/dask.py:1340: error: Signature of "fit" incompatible with supertype "LGBMRegressor"
python-package/lightgbm/dask.py:1340: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1358: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMRegressor")
python-package/lightgbm/dask.py:1408: error: Signature of "predict" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1490: error: Item "None" of "Optional[str]" has no attribute "partition"
python-package/lightgbm/dask.py:1504: error: Signature of "fit" incompatible with supertype "LGBMRanker"
python-package/lightgbm/dask.py:1504: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1525: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMRanker")
python-package/lightgbm/dask.py:1574: error: Signature of "predict" incompatible with supertype "LGBMModel"

@jameslamb
Copy link
Collaborator Author

35 errors as of 946817a

Found 35 errors in 2 files (checked 10 source files)
python-package/lightgbm/sklearn.py:627: error: Incompatible types in assignment (expression has type "None", variable has type "_ObjectiveFunctionWrapper")
python-package/lightgbm/dask.py:99: error: Invalid index type "Optional[str]" for "Dict[str, _HostWorkers]"; expected type "str"
python-package/lightgbm/dask.py:383: error: Invalid index type "Optional[str]" for "defaultdict[str, Set[int]]"; expected type "str"
python-package/lightgbm/dask.py:587: error: Argument 1 to "append" of "list" has incompatible type "Tuple[List[Any], List[Any]]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:590: error: Value of type "_DatasetNames" is not indexable
python-package/lightgbm/dask.py:591: error: Value of type "_DatasetNames" is not indexable
python-package/lightgbm/dask.py:609: error: Argument 1 to "append" of "list" has incompatible type "List[Any]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:611: error: "_DatasetNames" has no attribute "append"
python-package/lightgbm/dask.py:627: error: Argument 1 to "append" of "list" has incompatible type "List[Any]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:629: error: "_DatasetNames" has no attribute "append"
python-package/lightgbm/dask.py:645: error: Argument 1 to "append" of "list" has incompatible type "List[Any]"; expected "_DatasetNames"
python-package/lightgbm/dask.py:647: error: "_DatasetNames" has no attribute "append"
python-package/lightgbm/dask.py:666: error: Incompatible return value type (got "Dict[str, Any]", expected "LGBMModel")
python-package/lightgbm/dask.py:681: error: "Dict[str, Any]" has no attribute "result"
python-package/lightgbm/dask.py:726: error: Argument "worker_addresses" to "_machines_to_worker_map" has incompatible type "KeysView[Any]"; expected "List[str]"
python-package/lightgbm/dask.py:984: error: Incompatible types in assignment (expression has type "Tuple[Any, Any]", variable has type "Tuple[Any]")
python-package/lightgbm/dask.py:1015: error: "_DaskLGBMModel" has no attribute "_other_params"
python-package/lightgbm/dask.py:1046: error: "_DaskLGBMModel" has no attribute "get_params"
python-package/lightgbm/dask.py:1069: error: "_DaskLGBMModel" has no attribute "set_params"
python-package/lightgbm/dask.py:1075: error: "_DaskLGBMModel" has no attribute "get_params"
python-package/lightgbm/dask.py:1084: error: Item "_DaskLGBMModel" of "Union[_DaskLGBMModel, LGBMModel]" has no attribute "get_params"
python-package/lightgbm/dask.py:1158: error: Signature of "fit" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:1158: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1177: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMClassifier")
python-package/lightgbm/dask.py:1225: error: Signature of "predict" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:1225: error: Signature of "predict" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1244: error: Signature of "predict_proba" incompatible with supertype "LGBMClassifier"
python-package/lightgbm/dask.py:1340: error: Signature of "fit" incompatible with supertype "LGBMRegressor"
python-package/lightgbm/dask.py:1340: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1358: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMRegressor")
python-package/lightgbm/dask.py:1408: error: Signature of "predict" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1504: error: Signature of "fit" incompatible with supertype "LGBMRanker"
python-package/lightgbm/dask.py:1504: error: Signature of "fit" incompatible with supertype "LGBMModel"
python-package/lightgbm/dask.py:1525: error: Incompatible return value type (got "_DaskLGBMModel", expected "DaskLGBMRanker")
python-package/lightgbm/dask.py:1574: error: Signature of "predict" incompatible with supertype "LGBMModel"

@jameslamb
Copy link
Collaborator Author

jameslamb commented Oct 6, 2023

8 errors as of 8f577de

Found 8 errors in 2 files (checked 9 source files)
python-package/lightgbm/basic.py:739: error: Incompatible return value type (got "tuple[Any, list[str], list[str] | list[int], list[list[Any]]]", expected "tuple[Any, list[str], list[str], list[list[Any]]]")  [return-value]
python-package/lightgbm/basic.py:2809: error: Array constructor argument 1 of type "map[int]" is not convertible to the array element type "Iterable[c_char_p]"  [misc]
python-package/lightgbm/basic.py:2823: error: Array constructor argument 1 of type "map[int]" is not convertible to the array element type "Iterable[c_char_p]"  [misc]
python-package/lightgbm/basic.py:4484: error: Array constructor argument 1 of type "map[int]" is not convertible to the array element type "Iterable[c_char_p]"  [misc]
python-package/lightgbm/basic.py:4498: error: Array constructor argument 1 of type "map[int]" is not convertible to the array element type "Iterable[c_char_p]"  [misc]
python-package/lightgbm/basic.py:4708: error: Array constructor argument 1 of type "map[int]" is not convertible to the array element type "Iterable[c_char_p]"  [misc]
python-package/lightgbm/basic.py:4724: error: Array constructor argument 1 of type "map[int]" is not convertible to the array element type "Iterable[c_char_p]"  [misc]
python-package/lightgbm/engine.py:294: error: Incompatible types in assignment (expression has type "list[tuple[str, str, float, bool]] | list[tuple[str, str, float, bool, float]]", variable has type "list[tuple[str, str, float, bool]]")  [assignment]
Found 8 errors in 2 files (checked 9 source files)

That's especially good given that the type hints are so much more complete in the project than they were the last time I posted an update here back in February: #3867 (comment)

Getting close!

@jameslamb
Copy link
Collaborator Author

jameslamb commented Feb 24, 2024

I keep looking this up every time I run into it, so wanted to put it in writing here.

For almost all cases, the .predict() methods in the Python package return a numpy array or scipy.sparse.spmatrix. That makes it safe to do array-like operations on the result from .predict() unconditionally (e.g., accessing .shape, np.concat()-ing them).

There's one case where that isn't true. For the following specific combination, .predict() returns a Python list of scipy.sparse.spmatrix matrices:

  • getting SHAP feature contributions (.predict(..., pred_contrib=True))
  • input is a scipy.sparse.spmatrix (e.g., CSC or CSR matrix)
  • model is using a multiclass classification objective

That's been true since support for SHAP contributions on sparse matrices was added in #3000.

Here's the code path where that happens:

def __create_sparse_native(

# reformat output into a csr or csc matrix or list of csr or csc matrices
cs_output_matrices = []
offset = 0
for cs_indptr in out_indptr_arrays:
matrix_indptr_len = cs_indptr[cs_indptr.shape[0] - 1]
cs_indices = out_indices[offset + cs_indptr[0]:offset + matrix_indptr_len]
cs_data = out_data[offset + cs_indptr[0]:offset + matrix_indptr_len]
offset += matrix_indptr_len
# same shape as input csr or csc matrix except extra column for expected value
cs_shape = [cs.shape[0], cs.shape[1] + 1]
# note: make sure we copy data as it will be deallocated next
if is_csr:
cs_output_matrices.append(scipy.sparse.csr_matrix((cs_data, cs_indices, cs_indptr), cs_shape))
else:
cs_output_matrices.append(scipy.sparse.csc_matrix((cs_data, cs_indices, cs_indptr), cs_shape))

matrices = self.__create_sparse_native(

That one possibility means that everything handling the result of a .predict() call has to account for the possibility of receiving a list.

For example, this added some complexity to the Dask interface:

elif isinstance(data, dask_Array):
# for multi-class classification with sparse matrices, pred_contrib predictions
# are returned as a list of sparse matrices (one per class)
num_classes = model._n_classes
if (
num_classes > 2
and pred_contrib
and isinstance(data._meta, ss.spmatrix)
):
predict_function = partial(
_predict_part,
model=model,
raw_score=False,
pred_proba=pred_proba,
pred_leaf=False,
pred_contrib=True,
**kwargs
)
delayed_chunks = data.to_delayed()
bag = dask_bag_from_delayed(delayed_chunks[:, 0])
@delayed
def _extract(items: List[Any], i: int) -> Any:
return items[i]
preds = bag.map_partitions(predict_function)
# pred_contrib output will have one column per feature,
# plus one more for the base value
num_cols = model.n_features_ + 1
nrows_per_chunk = data.chunks[0]
out: List[List[dask_Array]] = [[] for _ in range(num_classes)]
# need to tell Dask the expected type and shape of individual preds
pred_meta = data._meta
for j, partition in enumerate(preds.to_delayed()):
for i in range(num_classes):
part = dask_array_from_delayed(
value=_extract(partition, i),
shape=(nrows_per_chunk[j], num_cols),
meta=pred_meta
)
out[i].append(part)
# by default, dask.array.concatenate() concatenates sparse arrays into a COO matrix
# the code below is used instead to ensure that the sparse type is preserved during concatentation
if isinstance(pred_meta, ss.csr_matrix):
concat_fn = partial(ss.vstack, format='csr')
elif isinstance(pred_meta, ss.csc_matrix):
concat_fn = partial(ss.vstack, format='csc')
else:
concat_fn = ss.vstack
# At this point, `out` is a list of lists of delayeds (each of which points to a matrix).
# Concatenate them to return a list of Dask Arrays.
out_arrays: List[dask_Array] = []
for i in range(num_classes):
out_arrays.append(
dask_array_from_delayed(
value=delayed(concat_fn)(out[i]),
shape=(data.shape[0], num_cols),
meta=pred_meta
)
)
return out_arrays

For the purpose of this issue and #3756, this can be handled with some mix of the approaches listed at https://mypy.readthedocs.io/en/stable/type_narrowing.html.


Longer-term, I think it's worth investigating whether it's possible to return an array type for that combination. Although that'd have to be done carefully, since (for example), shap relies on LightGBM's implementation of SHAP values:

https://github.com/shap/shap/blob/f83fda473cf9f96356000101d9549bc5c2497638/shap/explainers/pytree.py#L178-L179

I couldn't quite figure out from the description on #3000 why the decision was made to return a list. @guolinke or @imatiach-msft do you remember?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants