[doc] Warning message says "you didn't set num_leaves" even if I explicitly set num_leaves=31

[bfrobin446]

Environment info

Operating System: Ubuntu 18.04.3 LTS

CPU/GPU model: Not sure (running in AWS EC2)

C++/Python/R version: GCC 7.5/Python 3.7.5

LightGBM version or commit hash: 6b6709

Error message

[LightGBM] [Warning] Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves

Steps to reproduce

Fit any model with num_leaves = 31 and max_depth >= 5

At line 307 of src/io/config.cpp, the check for whether to show the warning can't tell whether num_leaves is 31 because that's the default or because 31 was explicitly passed. The text "you didn't set num_leaves" in the message is confusing if I did explicitly set num_leaves to 31.

[AlbertoEAF]

Hello @bfrobin446 ,

What would you like to see instead?

"Accuracy may be bad as num_leaves < 2^max_depth, with num_leaves=x." ?

[bfrobin446]

Maybe something like "The default num_leaves setting of 31 may be too low for best accuracy when 2^max_depth > num_leaves. Consider increasing num_leaves or reducing max_depth."

[AlbertoEAF]

I'll leave that to @StrikerRUS, what's your input on this?

[StrikerRUS]

I agree that the current warning message is far from perfect. Contribution is more than welcome! Feel free to propose a PR.

[AlbertoEAF]

Will do, thanks :) @bfrobin446, do you want me to do it or since you were so precise with the description, do you want to get your hands dirty and do it yourself? I'm fine either way, just let me know what you prefer ;)

…

On Fri, 20 Mar 2020, 20:59 Nikita Titov, ***@***.***> wrote: I agree that the current warning message is far from perfect. Contribution is more than welcome! Feel free to propose a PR. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2898 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6K4MI2NPSXYWA5F4LHYZLRIPKLRANCNFSM4LF6LEIQ> .

[aEgoist]

Agree with @bfrobin446 . This happens when I'm refering to Chinese documentations, where said we should let
classes< num_leaves< 2^max_depth. I took it and I got this warning, by changing which to num_leaves >2^max_depth it disappeared.
So by saying

[LightGBM] [Warning] Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves,

what are you really suggesting? Should be 2^max_depth > num_leaves or 2^max_depth < num_leaves?
This is confusing. If you take

[LightGBM] [Warning] Accuracy may be bad since you didn't set num_leaves OR 2^max_depth > num_leaves

Agreements would likely be reached.

[AlbertoEAF]

I can submit a PR this or next week as bfrobin446 didn't reply.

[memeplex]

Sorry for revisiting this, but this message is shown when verbosity == -1, isn't that a bug? It's certainly not fatal.

[Wang-Yu-Qing]

Maybe something like "The default num_leaves setting of 31 may be too low for best accuracy when 2^max_depth > num_leaves. Consider increasing num_leaves or reducing max_depth."

But why in https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html, the doc tell us to set num_leaves < 2^(max_depth)?

[jameslamb]

@guolinke or @shiyu1994 could you comment on @Wang-Yu-Qing 's question?

I agree that the warning message and documentation seem to be making contradictory recommendations, and I'm not qualified to say which is the correct one.

[shiyu1994]

The warning message is actually saying: you forget to set num_leaves, but you've set max_depth. This usually means that the user wants to use a depth-wise tree structure. But because LightGBM has a default num_leaves=31, the warning message reminds the user to set num_leaves accordingly when setting max_depth.

The document is emphasizing that max_depth can act as a regularization for LightGBM's leaf-wise tree structure. Setting num_leaves == 2^max_depth grows a full complete (depth-wise) tree. But for example, setting num_leaves == 2 ^ (max_depth - 1) produces a leaf-wise tree with constrained depth, but keeps a certain degree of the flexibility of leaf-wise tree.

My proposal is to keep the document unchanged. But in config.cpp, we need to detect whether num_leaves is set by user (instead of simply checking whether it is ==31, since user can also set it to 31). And if num_leaves is not set but max_depth is set, we may produce a warning like max_depth is set to xxx but num_leaves is not set, will use the default value 31 for num_leaves.

[jameslamb]

Ok thanks for the explanation @shiyu1994 !

Given that, I think we should remove this warning entirely, and rely on documentation to help users choose good values (and, in practice, good ranges to explore during tuning).

Because it seems to me that there are legitimate reasons to want either num_leaves < 2^max_depth or num_leaves == 2^max_depth.