support another structural output(xml/json)?

[huaiweicheng]

I use pandas_profiling to check my data every day to get knowlegde of my new prodcution data.
It greaterly improve the effenciency of data quality checking. Thanks to contributers for saving my life! Life is short, use Pandas Profiling!
However, I have found that there are still 2 problems which can not be handled perfectly right now.

1. When the columns of the dataset are big enough(>200), the generated html report can not be opened using IE or Chrome, the browsers will crash.
2. The most common scenario is that I wanna commpare the difference of data between 2 days.
   For example, I have one dataframe, and one column in this dataframe called A.
   The missing rate of A is in the range of 25% to 35% on usual . If the missing rate of A of new producted data is out of this range, I wanna generate a warning.

Both of the problems require recording the statictics info using file formats such as xml/json on a daily basis. Using html, it is not convenient to get the statictics info.
However, I do not find other outputs that panas_profiling but html.
The html is great. But on the one hand, sometimes the html report is too big to open. If I could store the data and chose some important columns to present, it will generate a report only contains these columns and could be opened since the scale of html is relatively small.
Besides, the analysis in problem 2 can be accomplished using a higher layer program by comparing the xml/json generated in two days.

I think the proposal will makes pandas_profiling greater!

[sbrugman]

Some quick comments.

Regarding the first point:
#222 also pointed out that there are problems with datasets with more columns. The examples all have around 10-15 columns. I am planning to make the report adaptive to the number of columns.

Anyone can contribute by providing open datasets that are representative for larger number of columns (e.g. 40-100 and 100+).

Regarding the second point:
#198 and #173 propose a solution that could be helpful here.

Providing the output als xml/json is definitely an option. This is a great place for beginning contributors to start. Just call:

profile = df.profile_report()
profile.get_description()

[huaiweicheng]

Thanks for comments.
I see the correlated issue and find the second point seems to bother other people.
Also I run the get_description method and see the output dict.

I have a rough idea and welcome to make corrections and other inspirational ideas.
The working steps follow the workflow of dataframe -> description file(xml/json) -> html.

We can recover the raw standalone html report from a single description file. It will generate the current output html.

profile1 = read_json('sample1.json')
profile1.to_file(output_file="sample1.html")

Furthormore, we could implement a compare api.

profile2 = read_json('sample2.json')
profile2.to_file(output_file="sample2.html")
diff_profile = profile1.diff(profile2)
diff_profile.to_file(output_file="diff_between_sample1_sample2.html")

The diff_profile could be a class similar to profile.

Or diff_profile is also a "profile" class. Then we need to extend the api/methods of current profile class to support list/array format and provide more complex comparing methods in this class.

Thus, we can recover the current html report from xml/json, and also provide a more flexible tool to compare between data sets such as training datatsets and testing datasets in ml.

[marco-cardoso]

Some quick comments.

Regarding the first point:
#222 also pointed out that there are problems with datasets with more columns. The examples all have around 10-15 columns. I am planning to make the report adaptive to the number of columns.

Anyone can contribute by providing open datasets that are representative for larger number of columns (e.g. 40-100 and 100+).

Regarding the second point:
#198 and #173 propose a solution that could be helpful here.

Providing the output als xml/json is definitely an option. This is a great place for beginning contributors to start. Just call:

profile = df.profile_report()
profile.get_description()

Hello @sbrugman , I'd like to understand better how the method to generate the output as XML/JSON would be. I have some questions :

1. The idea is just a method that is exactly like to_file() but saving a JSON/XML ? Something that the user could specify the output path.
2. The resulting file would follow the structure of the get_description() returned dictionary?

Any more details about this functionality would be appreciated.

Thanks for your attention

[sbrugman]

to_file('file.json') and .to_json() are now available, closing