get_rejected_variables missing after release of v2.4.0

[neomatrix369]

Describe the bug

My old code which used to work on a version prior to v2.4.0 now does not work as it uses parameters and methods that have been removed/deprecated in this version.

One of them is access to the list of Rejected variables - this is invaluable for data-processing and other reasoning activities.

I didn't see it mentioned in the release notes or in the docs - that they are removed and if there are alternative ways to accomplish it

How would we do this in the new version? Process the report as a JSON file and then access the rejected variables from there? Any other methods?

The docs are good but would be great to see a format like this, https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html where we see a list of all parameters and then potentially examples to follow.

To Reproduce

Version being used: v2.4.0 (worked in the previous version)

Try running the below with the new version of pandas-profiling:

    profile = dataset.profile_report(title=title,
                        # if 'bayesian_blocks_bins': True fails 
                        # to provide a valid number of bins use the below config:
                        #   {'bins': 8, 'bayesian_blocks_bins': False} 
                          plot={'histogram': {'bayesian_blocks_bins': True}}, 
                          style={'full_width': True}, 
                          minify_html=True)

and then also try this:

    filename, fileext = os.path.splitext(output_file)
    rejected_variables_filename = f'{filename}-rejected-variables.txt'
    rejected_variables = profile.get_rejected_variables(threshold=0.9)
    print("Rejected variables: ", rejected_variables)

both will fail to compile/run with errors!

[sbrugman]

Hi Mani,

There are three different observations in this one issue: rejected variables are missing, documentation could be more extensive and a bug in bayesian blocks. It would be good to split them into separate issues (a feature request for documentation, a bug report for the bayesian blocks, please make sure to include a dataset). We then can dedicate this issue to the rejected variables.

You have a good point that the removal of the get_rejected_variables was not mentioned. There were a few reasons to remove it (temporarily):

• the function used to recalculate the rejections, rather than extract them from the report
• the function was restricted to only one kind of correlation (Option to chose type of correlation to get get_rejected_variables  #298 and Rejected Columns based on co-relation type #299)
• negative correlations were not considered (Feature rejection does not take into account negative correlations #284)

The v2.4.0 release fixes these issues. The same functionality can now be obtained using:
profile.get_description()['messages']. In v2.4.1 I'll add an improved mapping from the description set to rejected variables (=get_rejected_variables).

[neomatrix369]

Hi Simon, I'll try to split these but also will answer quickly to some of the points.

• bayesian bug was already reported earlier I think, the note:

                        # if 'bayesian_blocks_bins': True fails 
                        # to provide a valid number of bins use the below config:
                        #   {'bins': 8, 'bayesian_blocks_bins': False} 

comes either from your config or issues/discussions made previously, see #222, #227, #245, #293. So there might not be something to report, I just added that note for myself so I follow the right config.

[neomatrix369]

I'll request docs for the get rejected variable aspect.

And also for removal of these parameters, that worked in the previous version:

style={'full_width': True}, minify_html=True

[neomatrix369]

With regards to reinstating get rejected variables, it's better to keep the same name and internally make the above call as you suggested so +1 there:

def get_rejected_variables(threshold=0.5):
    return profile.get_description()['messages']

(a contrived example)