Can't create DataFrame from SQLite3 cursor

[hexum]

When I pass cursor as data to DataFrame constructor an error occurs.

cursor = sqlite.execute(sql)
pd.DataFrame(cursor)


/usr/lib/python3/dist-packages/pandas/core/frame.py in __init__(self, data, index, columns, dtype, copy)
    255                                          copy=copy)
    256         elif isinstance(data, collections.Iterator):
--> 257             raise TypeError("data argument can't be an iterator")
    258         else:
    259             try:

TypeError: data argument can't be an iterator

But normal generators is accepted

def gen():
    yield (1,2,3)
    yield (4,5,6)
    yield (7,8,9)

pd.DataFrame(gen())
Out[171]: 
   0  1  2
0  1  2  3
1  4  5  6
2  7  8  9

It feels like inconsistence.

[jreback]

what is type(cursor).__mro__

[hexum]

type(cursor).__mro__
(sqlite3.Cursor, object)

[hexum]

It's not a big issue: passing cursor to list constructor results normal list wich is accepted by DataFrame.
I just can't understand why iterable is not acepted.
Type checking is a bad practice in Python, isn't it?
Why just not to check ability to iterate?

hasattr([], "__iter__")

[jreback]

type checking in python is fine. In pandas is actually quite a bit more complicated, because we need to determine, if, for example a list of-lists or list-of scalars are passed, then this is problematic

so an Iterable must have __iter__ AND __len__. A cursor doesn't have this property, while for example range(5) does. (I know that in theory rowcount works for his, but I don't think this is a guaranteed property). So a Cursor should act much like a GeneratorType (and not an Iterator).

[hexum]

I just create a generator with not defined length. And DataFrame accepts it as I expect.
I think we should turn off type check. Anyway user may construct infinity generator and pass type checking.

In [20]: def g():
   ....:     for i in range(5):
   ....:         yield [i, i ** 2, i ** 3]
   ....:         

In [21]: DataFrame(g())
Out[21]: 
   0   1   2
0  0   0   0
1  1   1   1
2  2   4   8
3  3   9  27
4  4  16  64

[jreback]

a generator is fine. you have an iterator. you can certainly make any changes you would like. but they would need to pass the test suite as is.

[hexum]

Hmmm. I'll see how to overcome it.

[ns-cweber]

@jreback Out of curiosity, why is a generator fine, but not an iterator? It looks like DataFrame's constructor checks to see if the data argument is a GeneratorType and then wraps it (data = list(data)), but if it's an iterator it raises an exception.

[jreback]

see my comment above
and iterator is not sufficient as its not required to have a len

this is an old issue - so don't really remember
have a look at the test code for Frame construction

[ns-cweber]

Generator is also not required to have a len. The solution for generators in the DataFrame constructor is to consume it into a list() and from there treat it as though a list was passed as the data arg. The same should work for iters. If you're worried about infinite iterators, then you should be equally worried about infinite generators, unless I'm misunderstanding something. I'll take a look at the tests.

[TomAugspurger]

Duplicate of #2193

I also think it'll be possible to turn the iterable into a list, just like with a generator.