COMPAT: Allow multi-indexes to be written to excel #10570

flamingbear · 2015-07-14T18:12:46Z

(Even though they cannot be read back in)

jreback · 2015-07-14T18:43:00Z

pandas/io/tests/test_excel.py

+                                          ('2014','weight')])
+        df = pd.DataFrame(np.random.randn(10,3), columns=cols)
+        import warnings
+        with warnings.catch_warnings(record=True) as w:


use tm.assert_produces_warning

jorisvandenbossche · 2015-07-15T12:33:24Z

Is this warn_roundtrip_serialization keyword needed?

I agree it can be annoying to not be able to remove a warning, but a keyword just for this that maybe is not needed anymore in a next release seems also a bit unnecessary

jreback · 2015-07-15T12:39:54Z

another way to do this is to make a custom warning class (e.g. inherit from UserWarning). Then always show the warning. If the users don't like it they can always filter. I DO think this is important to always warn as you are doing something which is not reversible.

jreback · 2015-07-15T12:41:05Z

doc/source/whatsnew/v0.17.0.txt

@@ -119,6 +119,7 @@ Other API Changes
 - Enable serialization of lists and dicts to strings in ExcelWriter (:issue:`8188`)
 - Allow passing `kwargs` to the interpolation methods (:issue:`10378`).
 - Serialize metadata properties of subclasses of pandas objects (:issue:`10553`).
+- Allow multi-indexes columns to be written to Excel with a one way serialization (:issue: `10564`).


I think this needs a bit more explanation (e.g. we removed this in 0.16.2 (or was it 1?)) because of the one-way serialization.

how about ?:

+- Allow ``DataFrame`` with ``MultiIndex`` columns to be written to Excel (:issue: `10564`). The original fix implemented for (:issue:`9794`) was slightly overkill as only ``DataFrame``s without indexes turned out to be broken Excel files.

more like:

this was changed in 0.16.2 as the read-back method could not always guarantee perfect fidelity.

jorisvandenbossche · 2015-07-15T12:42:37Z

It is maybe not reversible with a simple read_excel, but it is possible to get the same as original with a little bit of processing.

BTW, the docstring of read_excel is not fully correct, as it supports creating a multi-index index with index_col (it just can't deal with the formatting of multi-indices by to_excel)

jorisvandenbossche · 2015-07-15T12:44:54Z

And, @jreback, I like the idea of having a global way to turn of this kind of UserWarnings more than special keywords all over the place (and if we want an argument for it, I would rather go with something more general, like silent or verbose)

jreback · 2015-07-15T12:54:42Z

@jorisvandenbossche

hmm, ok could have io.excel.warnings='verbose' (for an option).

Though maybe a more general io.warnings='verbose'

flamingbear · 2015-07-30T17:55:12Z

Is this still being considered?

jreback · 2015-07-30T18:07:30Z

can you rebase / squash?

@jorisvandenbossche what do you think about verbose

flamingbear · 2015-07-30T18:46:11Z

Yeah, I'll see if I can pull in the master and squash.

flamingbear · 2015-07-30T19:12:57Z

Sorry I'm bungling this at the moment.

flamingbear · 2015-07-30T19:51:48Z

Ok. Got my git figured out. Hopefully tests will still pass. Thanks.

jorisvandenbossche · 2015-07-30T21:30:29Z

pandas/core/frame.py

-                                      "not yet implemented.")
+            if not index:
+                raise NotImplementedError("Writing to Excel with MultiIndex"
+                                          " columns and no index is not yet"


can you change 'and no index' to 'and with 'index=False' ' (otherwise you could understand that the dataframe has no index)

jorisvandenbossche · 2015-07-30T21:35:40Z

In any case, I think verbose is much better as the original name.
I think I also like that you can set an option for such things, but maybe both approaches do not exclude each other for now?

jreback · 2015-08-14T19:01:29Z

pandas/core/frame.py

@@ -1305,11 +1309,17 @@ def to_excel(self, excel_writer, sheet_name='Sheet1', na_rep='',
        """
        from pandas.io.excel import ExcelWriter
        if self.columns.nlevels > 1:
-            raise NotImplementedError("Writing as Excel with a MultiIndex is "


I would move this entire thing to core.format.ExcelFormatter didn't realize it was here.

I'm not exactly sure what you're looking for. Just the 'raise NotImpl...' moved into the ExcelFormatter init? or where?

this type of checking on the index should go about here. Then all of this code will be in one place.

flamingbear · 2015-08-14T20:58:14Z

Ok. Couldn't figure out what to do with the verbose keyword, so I improvised and copied someone else's custom warning.

jreback · 2015-08-14T21:08:11Z

you need to still accept the verbose kw in DataFrame.to_excel and pass in thru as another parameter. Then reference it in the formatter; further a custom warning is fine (though emitting it should still be controlled by verbose as before)

flamingbear · 2015-08-14T21:21:45Z

Now other things are breaking and I don't really see why. I'm just about at the end of my rope here.
So now there's a verbose keyword to ExcelFormatter too?

jreback · 2015-08-14T21:26:24Z

just pass it thru, you had added it before; just put it back in to_excel (and then to ExcelFormatter).

flamingbear · 2015-08-14T22:43:59Z

I'm not sure what the tests were trying to catch before or if they're good, but I can make them pass by reordering the tests for valid input before testing for aliases. (seems kinda sketchy)

flamingbear · 2015-08-15T15:59:08Z

Don't think this was my fault: https://travis-ci.org/pydata/pandas/jobs/75683428, it passed on the branch https://travis-ci.org/flamingbear/pandas/builds/75683397

I just pushed a no-op branch up again. Is there a way I could have scheduled travis tests without pushing the branch again?

flamingbear · 2015-08-15T16:37:29Z

@jreback I think this is ready for you now. Thanks. Matt

jreback · 2015-08-15T16:57:47Z

lgtm. @jorisvandenbossche ?

jorisvandenbossche · 2015-08-16T10:29:59Z

pandas/core/format.py

@@ -1640,11 +1644,14 @@ class ExcelFormatter(object):
    inf_rep : string, default `'inf'`
        representation for np.inf values (which aren't representable in Excel)
        A `'-'` sign will be added in front of -inf.
+    verbose: boolean, default True
+             If True, warn user that the resulting output file may not be
+             re-read or parsed directly by pandas.


can you use normal indentation (4 spaces here)? no alignment with the 'boolean, ..' is needed. (se the other parameters)

yes, I can do that.

flamingbear · 2015-08-16T20:42:12Z

@jreback @jorisvandenbossche : Can you give me some help here? This happened before and I repushed and it fixed itself. I don't know what a ResourceError is, https://travis-ci.org/pydata/pandas/jobs/75848253, but it's still passing fine on my travis. I can keep pushing until it goes through again. Is that the best option? This happened before #10570 (comment)

jreback · 2015-08-16T20:49:24Z

just leave for now
I think this is caused by something else

flamingbear · 2015-08-16T20:49:57Z

alright if this one fails, I'll leave it. Thanks

jreback · 2015-08-19T01:42:47Z

pandas/core/format.py

+                                          "is not yet implemented.")
+            elif self.index and self.verbose:
+                warnings.warn("Writing to Excel with MultiIndex columns is a"
+                              " one way serializable operation. You will not"


here is where I mean

jreback · 2015-08-19T01:43:33Z

@flamingbear minor change, then lgtm. ping when green.

(Even though they cannot be read back in.) Closes #10564

flamingbear · 2015-08-19T14:37:17Z

@jreback we're green. Thanks.

jreback · 2015-08-19T15:41:49Z

@jorisvandenbossche ?

jorisvandenbossche · 2015-08-20T20:33:12Z

Looks good! Thanks!

…el-writing COMPAT: Allow multi-indexes to be written to excel

flamingbear · 2015-08-20T21:36:48Z

Thanks guys.

jreback reviewed Jul 14, 2015
View reviewed changes

jreback added IO Excel read_excel, to_excel MultiIndex Compat pandas objects compatability with Numpy or Python functions labels Jul 15, 2015

jreback added this to the 0.17.0 milestone Jul 15, 2015

jreback reviewed Jul 15, 2015
View reviewed changes

jorisvandenbossche reviewed Jul 30, 2015
View reviewed changes

jreback reviewed Aug 14, 2015
View reviewed changes

jorisvandenbossche reviewed Aug 16, 2015
View reviewed changes

jreback reviewed Aug 19, 2015
View reviewed changes

COMPAT:Allow multi-indexes to be written to excel.

0e6cd60

(Even though they cannot be read back in.) Closes #10564

jorisvandenbossche added a commit that referenced this pull request Aug 20, 2015

Merge pull request #10570 from flamingbear/10564-allow-multiindex-exc…

b63206b

…el-writing COMPAT: Allow multi-indexes to be written to excel

jorisvandenbossche merged commit b63206b into pandas-dev:master Aug 20, 2015

flamingbear deleted the 10564-allow-multiindex-excel-writing branch March 31, 2016 15:59

COMPAT: Allow multi-indexes to be written to excel #10570

COMPAT: Allow multi-indexes to be written to excel #10570

Conversation

flamingbear commented Jul 14, 2015

Choose a reason for hiding this comment

jorisvandenbossche commented Jul 15, 2015

jreback commented Jul 15, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche commented Jul 15, 2015

jorisvandenbossche commented Jul 15, 2015

jreback commented Jul 15, 2015

flamingbear commented Jul 30, 2015

jreback commented Jul 30, 2015

flamingbear commented Jul 30, 2015

flamingbear commented Jul 30, 2015

flamingbear commented Jul 30, 2015

Choose a reason for hiding this comment

jorisvandenbossche commented Jul 30, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flamingbear commented Aug 14, 2015

jreback commented Aug 14, 2015

flamingbear commented Aug 14, 2015

jreback commented Aug 14, 2015

flamingbear commented Aug 14, 2015

flamingbear commented Aug 15, 2015

flamingbear commented Aug 15, 2015

jreback commented Aug 15, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flamingbear commented Aug 16, 2015

jreback commented Aug 16, 2015

flamingbear commented Aug 16, 2015

Choose a reason for hiding this comment

jreback commented Aug 19, 2015

flamingbear commented Aug 19, 2015

jreback commented Aug 19, 2015

jorisvandenbossche commented Aug 20, 2015

flamingbear commented Aug 20, 2015