ASV: occasional asv failures on xlwt

[jreback]

https://travis-ci.org/pandas-dev/pandas-ci/jobs/343616367
this is on current pandas master (this is just the CI job running it).

excel asv's

The following asvs benchmarks (if any) failed.
[ 43.02%] ··· Running io.excel.Excel.time_read_excel                  1/3 failed
                   xlwt      failed 
DONE displaying failed asvs benchmarks.

I have seen this work as well. maybe a race-condition?

[jreback]

from (pandas) bash-3.2$ more asv_bench/benchmarks/io/excel.py

def setup(...):
      ....
        self.bio_write = BytesIO()
        self.bio_write.seek(0)
        self.writer_write = ExcelWriter(self.bio_write, engine=engine)

    def time_read_excel(self, engine):
        read_excel(self.bio_read)

    def time_write_excel(self, engine):
        self.df.to_excel(self.writer_write, sheet_name='Sheet1')
        self.writer_write.save()

I think the time_write_excel should have the .writer_write setup I think.

[jorisvandenbossche]

If it is defined in the setup function (as it is now), it should be available in the benchmark function. It would be strange that this solves it.

[jreback]

its passing 1 our of n times: https://travis-ci.org/pandas-dev/pandas-ci/jobs/345996588

[jreback]

cc @mroeschke @WillAyd if you have any ideas

[WillAyd]

Perhaps we should be explicitly closing the BytesIO objects that are getting created in a teardown? Due to the intermittency of it I'm wondering if the GC is taking an pass at closing those for us, but getting tripped up with the parallelized execution that asv provides

[mroeschke]

Looks to be specifically a problem with the time_read_excel benchmark.

File "/home/travis/build/pandas-dev/pandas-ci/pandas/asv_bench/benchmarks/io/excel.py", line 29, in time_read_excel
                    read_excel(self.bio_read)
File "/home/travis/miniconda3/envs/pandas/lib/python3.6/site-packages/pandas-0.23.0.dev0+381.g8f1dfa74e-py3.6-linux-x86_64.egg/pandas/util/_decorators.py", line 172, in wrapper
                    return func(*args, **kwargs)
File "/home/travis/miniconda3/envs/pandas/lib/python3.6/site-packages/pandas-0.23.0.dev0+381.g8f1dfa74e-py3.6-linux-x86_64.egg/pandas/util/_decorators.py", line 172, in wrapper
                    return func(*args, **kwargs)
File "/home/travis/miniconda3/envs/pandas/lib/python3.6/site-packages/pandas-0.23.0.dev0+381.g8f1dfa74e-py3.6-linux-x86_64.egg/pandas/io/excel.py", line 315, in read_excel
                    io = ExcelFile(io, engine=engine)
File "/home/travis/miniconda3/envs/pandas/lib/python3.6/site-packages/pandas-0.23.0.dev0+381.g8f1dfa74e-py3.6-linux-x86_64.egg/pandas/io/excel.py", line 391, in __init__
                    self.book = xlrd.open_workbook(file_contents=data)
File "/home/travis/miniconda3/envs/pandas/lib/python3.6/site-packages/xlrd/__init__.py", line 116, in open_workbook
                    with open(filename, "rb") as f:
                TypeError: expected str, bytes or os.PathLike object, not NoneType

Digging into xlrd.open_workbook for the file_contents variable.
http://www.lexicon.net/sjmachin/xlrd.html#xlrd.open_workbook-function

file_contents
... as a string or an mmap.mmap object or some other behave-alike object. If file_contents is supplied, filename will not be used, except (possibly) in messages.

Looks like filename=None is the default as well, but for some reason its being used despite the note above?

[jreback]

its funny because it has worked at times. really odd.

[WillAyd]

While it doesn't explain why this is happening I think if we add io.seek(0) just before the below line it will "fix" the issue at hand (at least it did locally for me):

pandas/pandas/io/excel.py

Line 390 in 74dbfd0

data = io.read()

[jreback]

can u replicate this in a test? (and then fix)?

[mcrot]

Hi all,

I'm completely new to pandas development and I've just prepared a working environment following the guide Contributing to pandas, because I wanted to contribute to some other issue. So I'm not fully sure whether this is the right place in order to address the following test failure, but for me it seems to be related to #19926:

When running the tests for pandas.io.excel

pytest pandas/tests/io/test_excel.py

it comes up with three failures (because of 3 parameters) for the test method TestXlrdReader.test_read_from_http_url:

F
pandas/tests/io/test_excel.py:557 (TestXlrdReader.test_read_from_http_url[.xls])
self = <pandas.tests.io.test_excel.TestXlrdReader object at 0x7fc1d823a1d0>
ext = '.xls'

    @tm.network
    def test_read_from_http_url(self, ext):
        url = ('https://raw.github.com/pandas-dev/pandas/master/'
               'pandas/tests/io/data/test1' + ext)
>       url_table = read_excel(url)

test_excel.py:562: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../util/_decorators.py:172: in wrapper
    return func(*args, **kwargs)
../../util/_decorators.py:172: in wrapper
    return func(*args, **kwargs)
../../io/excel.py:315: in read_excel
    io = ExcelFile(io, engine=engine)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pandas.io.excel.ExcelFile object at 0x7fc1d82d50f0>
io = <http.client.HTTPResponse object at 0x7fc1d81e3320>, kwds = {}
err_msg = 'Install xlrd >= 0.9.0 for Excel support'
xlrd = <module 'xlrd' from '/home/mcrot/miniconda3/envs/pandas-dev/lib/python3.6/site-packages/xlrd/__init__.py'>
ver = (1, 1), engine = None

    def __init__(self, io, **kwds):
    
        err_msg = "Install xlrd >= 0.9.0 for Excel support"
    
        try:
            import xlrd
        except ImportError:
            raise ImportError(err_msg)
        else:
            ver = tuple(map(int, xlrd.__VERSION__.split(".")[:2]))
            if ver < (0, 9):  # pragma: no cover
                raise ImportError(err_msg +
                                  ". Current version " + xlrd.__VERSION__)
    
        # could be a str, ExcelFile, Book, etc.
        self.io = io
        # Always a string
        self._io = _stringify_path(io)
    
        engine = kwds.pop('engine', None)
    
        if engine is not None and engine != 'xlrd':
            raise ValueError("Unknown engine: {engine}".format(engine=engine))
    
        # If io is a url, want to keep the data as bytes so can't pass
        # to get_filepath_or_buffer()
        if _is_url(self._io):
            io = _urlopen(self._io)
        elif not isinstance(self.io, (ExcelFile, xlrd.Book)):
            io, _, _, _ = get_filepath_or_buffer(self._io)
    
        if engine == 'xlrd' and isinstance(io, xlrd.Book):
            self.book = io
        elif not isinstance(io, xlrd.Book) and hasattr(io, "read"):
            # N.B. xlrd.Book has a read attribute too
            if hasattr(io, 'seek'):
                # GH 19779
>               io.seek(0)
E               io.UnsupportedOperation: seek

../../io/excel.py:392: UnsupportedOperation

It seems like the HTTPResponse object returned by urllib.request.urlopen does not support seeking, although the seek() method is available.

This fixes the tests for me:

@@ -10,6 +10,7 @@ import os
 import abc
 import warnings
 import numpy as np
+from http.client import HTTPResponse
 
 from pandas.core.dtypes.common import (
     is_integer, is_float,
@@ -387,7 +388,9 @@ class ExcelFile(object):
             self.book = io
         elif not isinstance(io, xlrd.Book) and hasattr(io, "read"):
             # N.B. xlrd.Book has a read attribute too
-            if hasattr(io, 'seek'):
+            #
+            # http.client.HTTPResponse.seek() -> UnsupportedOperation exception
+            if not isinstance(io, HTTPResponse) and hasattr(io, 'seek'):
                 # GH 19779
                 io.seek(0)

Should I create a pull request here or a new issue or do I miss something in my setup such that the tests can't run?

My currently installed versions are:

INSTALLED VERSIONS
------------------
commit: 01882ba5b4c21b0caf2e6b9279fb01967aa5d650
python: 3.6.4.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-116-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: de_DE.UTF-8
LOCALE: de_DE.UTF-8

pandas: 0.23.0.dev0+657.g01882ba
pytest: 3.4.2
pip: 9.0.1
setuptools: 38.5.1
Cython: 0.27.3
numpy: 1.14.2
scipy: 1.0.0
pyarrow: 0.8.0
xarray: 0.10.2
IPython: 6.2.1
sphinx: 1.7.1
patsy: 0.5.0
dateutil: 2.7.0
pytz: 2018.3
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.4
feather: 0.4.0
matplotlib: 2.2.2
openpyxl: 2.5.0
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: 4.1.1
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.5
pymysql: 0.8.0
psycopg2: None
jinja2: 2.10
s3fs: 0.1.3
fastparquet: 0.1.4
pandas_gbq: None
pandas_datareader: None

[WillAyd]

@mcrot I would suggest that you open a new issue for that and then submit a PR referencing the issue