Change Finance Options signatures and deprecate year/month parameters #3822

gliptak · 2013-06-09T17:17:44Z

gliptak · 2013-06-11T22:03:21Z

This is the error this build gets (my local unit test was running without errors ...). Investigating:

======================================================================

ERROR: test_options (pandas.io.tests.test_yahoo.TestYahoo)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/pandas-0.11.1.dev_d26f4b6-py2.7-linux-x86_64.egg/pandas/io/tests/test_yahoo.py", line 131, in test_options

    (calls, puts) = aapl.get_near_stock_price(call=True, put=True, dt=dt)

  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/pandas-0.11.1.dev_d26f4b6-py2.7-linux-x86_64.egg/pandas/io/data.py", line 768, in get_near_stock_price

    start_index = np.where(df_c['Strike'] > price)[0][0]

IndexError: index out of bounds

======================================================================

ERROR: test_options_warnings (pandas.io.tests.test_yahoo.TestYahoo)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/pandas-0.11.1.dev_d26f4b6-py2.7-linux-x86_64.egg/pandas/io/tests/test_yahoo.py", line 155, in test_options_warnings

    (calls, puts) = aapl.get_near_stock_price(call=True, put=True, month=month, year=year)

  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/pandas-0.11.1.dev_d26f4b6-py2.7-linux-x86_64.egg/pandas/io/data.py", line 768, in get_near_stock_price

    start_index = np.where(df_c['Strike'] > price)[0][0]

IndexError: index out of bounds

----------------------------------------------------------------------

Ran 3263 tests in 85.929s

FAILED (SKIP=42, errors=2)

gliptak · 2013-06-11T23:28:34Z

Why wouldn't lxml.html be available under Python 2.6?

======================================================================
ERROR: test_options_warnings (pandas.io.tests.test_yahoo.TestYahoo)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.6_with_system_site_packages/lib/python2.6/site-packages/pandas-0.11.1.dev_23952c9-py2.6-linux-x86_64.egg/pandas/io/tests/test_yahoo.py", line 185, in test_options_warnings
    (calls, puts) = aapl.get_options_data(month=month, year=year)
  File "/home/travis/virtualenv/python2.6_with_system_site_packages/lib/python2.6/site-packages/pandas-0.11.1.dev_23952c9-py2.6-linux-x86_64.egg/pandas/io/data.py", line 526, in get_options_data
    from lxml.html import parse
ImportError: No module named lxml.html

gliptak · 2013-06-12T01:18:31Z

Could somebody offer some comments on this? I cannot reproduce the same error locally running with/without network connectivity.

jreback · 2013-06-12T11:04:18Z

pandas/io/data.py

-        s = "Failed to download:\n{0}".format(e)
-        print s
-        return None
+    lines = urllib2.urlopen(urlStr).readlines()


is there a reason you are taking out this exception? This will catch a failed network download

The reason to take out the exception

this will make it consistent with the behaviour of the other functions (which do throw this network error)

this allows in the unit test to mark the test "skip"

ok...makes sense, to keep in mind, I raised this with @hayd on #3856, should replace this call with io.common._url_open or something (which can have options to handle e.g. gzip data), and also to have consistent exceptions, say NetworkError or somthing....fine for now, this is really a FYI

@jreback I would like to make this work correctly first :)

jreback · 2013-06-13T13:24:06Z

@gliptak can you rebase to a smaller num of commits?

e.g

git rebase -i <your starting commit>
then f or s

cpcloud · 2013-06-13T14:34:38Z

git rebase -i upstream/master is nice too if you've already fetched and rebased. then u don't have to use commit hashes to remember where you started

hayd · 2013-06-13T14:41:45Z

That's a nice tip! I also recommend squashing before doing a rebase (that way you do fewer steps / one step for conflicts, and more likely to understand how they can be resolved).

jreback · 2013-06-13T14:44:53Z

@cpcloud maybe also add these tips to developer page? and It hink should have a link to developer page from somewhere in main docs......

cpcloud · 2013-06-13T14:46:15Z

there's a link...dev workflow is on my pandas to-do list: 2nd after eval!

hayd · 2013-06-13T14:47:31Z

#3156 :)

gliptak · 2013-06-13T21:38:41Z

@jreback I did squash several commits. Thanks

jtratner · 2013-06-15T11:48:15Z

@gliptak Right now the test cases for network errors aren't really doing what you want, because they catch all exceptions for bad tests, as opposed to specifically describing the behavior you want (e.g., should get_quote_yahoo bubble up a network error, or should it replace it with its own error?)

Catching exceptions with the baseclass Exception (and raising exceptions that are just Exception) is not great, because it means that users of the library have to catch bare Exceptions, as opposed to just opting into the exceptions they care about (i.e., you want to be able to see that you misspelled something as opposed to hitting a NetworkError).

Also, I suggest you consider changing the raise Exception(...) in io/data.py to raise a specific exception, i.e.:

from pandas.core.common import PandasError
class DataError(PandasError):
    pass

And that way you can catch these in your test cases, etc.

gliptak · 2013-06-15T13:36:17Z

@jtratner note that in the code I'm not raising exceptions, although not catching/converting them to 'known' exceptions either. I believe the test code works, as it only marks the failing test skip if it notes a network problem (by trying to connect to google.com) The change you are suggesting is not related to this enhancement, I would be happy to work modifying current code as a separate pull request.

the reason that I widened to catching IOError in the tests, is that I have seen other errors beside URLError showing up in my testing

as per http://stackoverflow.com/a/3465796/304690 urllib2.urlopen throws HTTPError, URLError and ValueError

jtratner · 2013-06-15T15:13:55Z

I mean exceptions like this:

    raise Exception("after %d tries, Yahoo did not "
                    "return a 200 for url %s" % (pause, url))

And in test cases, when it tests assertRaises(Exception, ... it makes it much less clear what behavior to expect.

gliptak · 2013-06-15T17:09:07Z

@jtratner Jeff, that is existing code, unmodified from current HEAD. As I said above, i would be happy to continue refactoring under a new pull request.

jtratner · 2013-06-18T06:40:49Z

pandas/io/data.py

@@ -430,8 +425,9 @@ def _parse_options_data(table):


 class Options(object):
+    import warnings


You should move this import statement outside of the class definition (preferably at the top of the file). Right now Options has an attribute warnings which is this package.

jreback · 2013-06-18T23:05:59Z

@gliptak this needs release notes / mention in v0.11.1

gliptak · 2013-06-18T23:24:17Z

@jreback updated RELEASE.rst

hayd · 2013-06-19T00:01:31Z

@gliptak You're going to need to rebase again (it happens pretty often with the RELEASE.rst as everyone is adding to it)

Tip: don't add your release note at the end, that way it's less likely to conflict :)

gliptak · 2013-06-19T01:10:03Z

@hayd Thanks Andy. Moved my note higher up.

hayd · 2013-06-19T01:17:54Z

(There's no guarantee it will merge cleanly even when not at the bottom, (I claim) it just makes it less lightly - so there still could be a conflict even when moved, only way to be sure is to do the rebase :) )

jtratner · 2013-06-19T01:19:40Z

@gliptak can you edit this after the new decorators are merged in and remove the in-test IOError checks?

gliptak · 2013-06-19T01:47:19Z

@jtratner I can work that tomorrow afternoon/evening.

cpcloud · 2013-06-21T20:46:35Z

@gliptak looks like you tried to merge from upstream. i'm going to pull down ur branch and take a look

cpcloud · 2013-06-21T20:47:32Z

i used to do this until i understood merge vs. rebase. going forward it's key that u understand the differences so that your contributions will be merged faster...

cpcloud · 2013-06-21T20:51:08Z

ok so all you need to do is the following

git fetch upstream
git rebase upstream/master
# resolve the merge conflict generated here
git add pandas/io/data.py
git rebase --continue

cpcloud · 2013-06-21T20:51:58Z

i use the first 2 commands so often that i have them aliased as

alias gfrb='git fetch upstream && git rebase upstream/master'

jreback · 2013-06-21T21:01:50Z

@cpcloud fyi.....should put how to rebase on the wiki (I think you put on developer pages as well?)

gliptak · 2013-06-22T00:00:05Z

@jreback this is what i basically did, but i could not complete the rebase ...

cpcloud · 2013-06-22T00:00:34Z

what do you mean you couldn't complete it?

cpcloud · 2013-06-22T00:00:47Z

@jreback anything elseyou want me to add to the wiki?

jreback · 2013-06-22T00:03:21Z

looks good...maybe a bit more about how to actually run git rebase -i <commit>.....e.g. what/how to squash

gliptak · 2013-06-22T00:11:04Z

@cpcloud I got:
git fatal: unrecognized input Repository lacks necessary blobs to fall back on 3-way merge.
basically ended up hand merging changes

cpcloud · 2013-06-22T00:13:06Z

hm i only get git fatals when a lock file isn't removed...then usually it is when i check and then i run it again...if u get stuck on a rebase do git rebase --abort and try again to see .e.,g if it was a lock problem

gliptak · 2013-06-22T00:15:04Z

@cpcloud yes, I tried. even after git rebase --abort, still seen the same message. But now, I hope I'm back in business.

cpcloud · 2013-06-22T00:54:46Z

hm i pulled down ur branch just after your first message and things went except for 2 merge conflicts which were easy to resolve...

cpcloud · 2013-06-22T00:56:10Z

pandas/io/data.py

@@ -897,11 +876,21 @@ def get_near_stock_price(self, above_below=2, call=True, put=False,
            else:
                return chop_put

+    def _try_parse_dates(self, year, month, expiry):
+        if (year is not None or month is not None):


don't need to change this but fyi there's no need for the parens, they are implicit

jreback · 2013-06-22T17:43:05Z

@gliptak looks ready to go?

gliptak · 2013-06-22T18:58:28Z

@jreback I think so

Change Finance Options signatures and deprecate year/month parameters

jreback · 2013-06-22T19:02:31Z

thank you

jreback · 2013-06-27T10:06:56Z

can you have a look? test_options keeps failing sometimes..

gliptak mentioned this pull request Jun 9, 2013

backwards compatible refactoring of io/data options functionality #3817

Closed

jreback reviewed Jun 12, 2013
View reviewed changes

gliptak mentioned this pull request Jun 12, 2013

RPT/BUG: yahoo retrieve failure error message is uninformative #3852

Closed

cpcloud mentioned this pull request Jun 15, 2013

TST: Make network decorator impose a skipTest catching URLErrors #3910

Closed

jtratner reviewed Jun 18, 2013
View reviewed changes

gliptak mentioned this pull request Jun 19, 2013

Skip tests on network error #3893

Merged

gliptak added 2 commits June 21, 2013 20:02

Deprecated month/year, use expiry instead

19c5812

Added tests for Options

f597732

@network now catches connectivity problems

ae388a2

cpcloud reviewed Jun 22, 2013
View reviewed changes

Remove superfluous parentheses

62e4168

jreback added a commit that referenced this pull request Jun 22, 2013

Merge pull request #3822 from gliptak/googleoptions

2b4fd7c

Change Finance Options signatures and deprecate year/month parameters

jreback merged commit 2b4fd7c into pandas-dev:master Jun 22, 2013

jreback mentioned this pull request Jun 25, 2013

TST: test_yahoo/test_options is failing (return no data) #4028

Closed

gliptak deleted the googleoptions branch June 27, 2013 23:44

This was referenced Apr 3, 2014

DEPR: create issues for the current FutureWarnings in pandas #6641

Closed

Remove number of deprecated parameters/functions/classes [fix #6641] #6813

Merged

DEPR: Clean up list of deprecations from prior versions #6581

Closed

jreback mentioned this pull request Jul 25, 2016

DEPR: deprecations log for removed issues #13777

Closed

		@@ -430,8 +425,9 @@ def _parse_options_data(table):


		class Options(object):
		import warnings

Change Finance Options signatures and deprecate year/month parameters #3822

Change Finance Options signatures and deprecate year/month parameters #3822

Conversation

gliptak commented Jun 9, 2013

gliptak commented Jun 11, 2013

gliptak commented Jun 11, 2013

gliptak commented Jun 12, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jun 13, 2013

cpcloud commented Jun 13, 2013

hayd commented Jun 13, 2013

jreback commented Jun 13, 2013

cpcloud commented Jun 13, 2013

hayd commented Jun 13, 2013

gliptak commented Jun 13, 2013

jtratner commented Jun 15, 2013

gliptak commented Jun 15, 2013

jtratner commented Jun 15, 2013

gliptak commented Jun 15, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jun 18, 2013

gliptak commented Jun 18, 2013

hayd commented Jun 19, 2013

gliptak commented Jun 19, 2013

hayd commented Jun 19, 2013

jtratner commented Jun 19, 2013

gliptak commented Jun 19, 2013

cpcloud commented Jun 21, 2013

cpcloud commented Jun 21, 2013

cpcloud commented Jun 21, 2013

cpcloud commented Jun 21, 2013

jreback commented Jun 21, 2013

gliptak commented Jun 22, 2013

cpcloud commented Jun 22, 2013

cpcloud commented Jun 22, 2013

jreback commented Jun 22, 2013

gliptak commented Jun 22, 2013

cpcloud commented Jun 22, 2013

gliptak commented Jun 22, 2013

cpcloud commented Jun 22, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jun 22, 2013

gliptak commented Jun 22, 2013

jreback commented Jun 22, 2013

jreback commented Jun 27, 2013