Support mrecarrays in DataFrame constructor

[jnothman]

This is intended as a fix for #3478. Sorry I do not have the time to write rigorous tests.

[jnothman]

To illustrate my point, this is what the condition where all MaskedArray stuff is handled together looks like:

        elif isinstance(data, ma.MaskedArray):
            def unmask(arr):
                mask = ma.getmaskarray(data)
                if mask.any():
                    data, fill_value = _maybe_upcast(data, copy=True)
                    data[mask] = fill_value
                    return data
                return data.copy()

            if data.dtype.names:
                data_columns, data = _rec_to_dict(data)
                data = dict((k, unmask(v)) for k, v in data.iteritems())
                if columns is None:
                    columns = data_columns
                mgr = self._init_dict(data, index, columns, dtype=dtype)
            else:
                mgr = self._init_ndarray(unmask(data), index, columns, dtype=dtype,
                                         copy=copy)

Doesn't that second half look stupendously like the if isinstance(..., ndarray) case?? With an unnecessary dict creation thrown in for fun?

I still contend this is a case where code quality should not be sacrificed for marginal efficiency.

[jreback]

You could easily make a function to combine best of both worlds here (also would put unmask outside of Frame in any event). certainly try if you would like. Still need tests though. There are lots of cases and need to avoid breaking anything

[jnothman]

The point is: you couldn't easily factor out the is/isn't rec-array case, because in unmasking already requires a non-rec array. You can easily factor out the unmasking, which is what I did.

Of course it requires testing (or alternatively, test smaller units like _unmask and _rec_to_dict and you can get away with fewer tests for the same coverage). But I don't have the time to put to that.

[jreback]

what is your use case for masked record arrays? why u creating them in the first place (rather than directly in pandas)? existing code?

[jnothman]

I am not sure the use-case is relevant, seeing as it would be natural for pandas to support them, being the equivalent numpy data structure to what pandas works with, and given that pandas already supports both numpy masking and numpy records / structured arrays. But I am considering mrecarray output from a model selection procedure in scikit-learn: to handle some cases (where the set of parameters set in each iteration is not constant) masking or equivalent is required; leaving it in a numpy structure avoids binding users to pandas and avoids the project including a pandas dependency.

[jnothman]

Okay. Here is a test that ensures input with mrecarray is identical to equivalent input with dicts.

[jreback]

closing in favor of #4836

[jnothman]

I don't get why you prefer more specialisation rather than code reuse, but I'm glad the feature is now supported.

[jreback]

your solution forced unmasking on evey ndarray which is not good

also the style was different than the rest

your tests were gr8!