Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: date_range not working as it intuitively should when specifying start, end, and periods #20808

Closed
onnoeberhard opened this issue Apr 24, 2018 · 5 comments · Fixed by #20846
Labels
API Design Datetime Datetime data dtype
Milestone

Comments

@onnoeberhard
Copy link
Contributor

Code Sample, a copy-pastable example if possible

>>> start = pd.Timestamp('2008-01-02 07:51:37.999477')
>>> end = start + pd.Timedelta('2 hours')
>>> pd.date_range(start, end, periods=1000)    # Intuitively a linearly spaced time series
Traceback (most recent call last):

  File "<ipython-input-69-2304a28824c6>", line 1, in <module>
    pd.date_range(start, end, periods=1000)

  File "E:\Anaconda3\lib\site-packages\pandas\core\indexes\datetimes.py", line 2057, in date_range
    closed=closed, **kwargs)

  File "E:\Anaconda3\lib\site-packages\pandas\util\_decorators.py", line 118, in wrapper
    return func(*args, **kwargs)

  File "E:\Anaconda3\lib\site-packages\pandas\core\indexes\datetimes.py", line 324, in __new__
    ambiguous=ambiguous)

  File "E:\Anaconda3\lib\site-packages\pandas\core\indexes\datetimes.py", line 421, in _generate
    raise ValueError('Of the three parameters: start, end, and '

ValueError: Of the three parameters: start, end, and periods, exactly two must be specified

Problem description

I need a DatetimeIndex object to later use as index in a Series. DatetimeIndex should start at start, end at end and have a fixed number of elements (1000). Intuitively, this should work with pd.date_range, but it doesn't, and I haven't found a good explanation about why this is the case.
I have found a workaround on Stackoverflow (https://stackoverflow.com/questions/25796030/how-can-i-use-pandas-date-range-to-obtain-a-time-series-with-n-specified-perio) that does work:

>>> start = pd.Timestamp('2008-01-02 07:51:37.999477')
>>> end = start + pd.Timedelta('2 hours')
>>> pd.to_datetime(np.linspace(start.value, end.value, 1000))
DatetimeIndex(['2008-01-02 07:51:37.999476992',
               '2008-01-02 07:51:45.206684160',
               '2008-01-02 07:51:52.413891328',
               '2008-01-02 07:51:59.621098496',
               '2008-01-02 07:52:06.828305920',
               '2008-01-02 07:52:14.035513088',
               '2008-01-02 07:52:21.242720256',
               '2008-01-02 07:52:28.449927424',
               '2008-01-02 07:52:35.657134592',
               '2008-01-02 07:52:42.864341760',
               ...
               '2008-01-02 09:50:33.134612224',
               '2008-01-02 09:50:40.341819392',
               '2008-01-02 09:50:47.549026560',
               '2008-01-02 09:50:54.756233728',
               '2008-01-02 09:51:01.963440896',
               '2008-01-02 09:51:09.170648064',
               '2008-01-02 09:51:16.377855488',
               '2008-01-02 09:51:23.585062656',
               '2008-01-02 09:51:30.792269824',
               '2008-01-02 09:51:37.999476992'],
              dtype='datetime64[ns]', length=1000, freq=None)

Expected Output

>>> start = pd.Timestamp('2008-01-02 07:51:37.999477')
>>> end = start + pd.Timedelta('2 hours')
>>> pd.date_range(start, end, periods=1000)
DatetimeIndex(['2008-01-02 07:51:37.999476992',
               '2008-01-02 07:51:45.206684160',
               '2008-01-02 07:51:52.413891328',
               '2008-01-02 07:51:59.621098496',
               '2008-01-02 07:52:06.828305920',
               '2008-01-02 07:52:14.035513088',
               '2008-01-02 07:52:21.242720256',
               '2008-01-02 07:52:28.449927424',
               '2008-01-02 07:52:35.657134592',
               '2008-01-02 07:52:42.864341760',
               ...
               '2008-01-02 09:50:33.134612224',
               '2008-01-02 09:50:40.341819392',
               '2008-01-02 09:50:47.549026560',
               '2008-01-02 09:50:54.756233728',
               '2008-01-02 09:51:01.963440896',
               '2008-01-02 09:51:09.170648064',
               '2008-01-02 09:51:16.377855488',
               '2008-01-02 09:51:23.585062656',
               '2008-01-02 09:51:30.792269824',
               '2008-01-02 09:51:37.999476992'],
              dtype='datetime64[ns]', length=1000, freq=None)

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 44 Stepping 2, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en
LOCALE: None.None

pandas: 0.22.0
pytest: 3.3.2
pip: 9.0.1
setuptools: 38.4.0
Cython: 0.27.3
numpy: 1.14.2
scipy: 1.0.1
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.6.6
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.4
feather: None
matplotlib: 2.1.2
openpyxl: 2.4.10
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: 4.1.1
bs4: 4.6.0
html5lib: 0.9999999
sqlalchemy: 1.2.1
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

@jreback
Copy link
Contributor

jreback commented Apr 24, 2018

I suppose if freq is NOT specified then you could accept all three and give a linspace repr. What breaks if you do that?

If freq is specified then this should for sure raise. The idea of the error is simply to be helpful in telling you that you may have overspecified the args.

@chris-b1
Copy link
Contributor

Your workaround from SO is a pretty reasonable solution, not sure we should support this. I definitely would not want to change the default behavior, but suppose could with with something like freq=None or freq='interpolate'

@chris-b1 chris-b1 added Datetime Datetime data dtype API Design labels Apr 24, 2018
@onnoeberhard
Copy link
Contributor Author

onnoeberhard commented Apr 26, 2018

I agree to definetely don't change the default behaviour. As I see, the default freq is 'D', so this:

pd.date_range(start, end, periods=1000)

would not work, because it is the same as

pd.date_range(start, end, periods=1000, freq='D')

which really should not work. However, if the user explicitly sets freq=None, the linspace behaviour would be practical:

pd.date_range(start, end, periods=1000, freq=None)

I suggest a simple change in the date_range function like so:

def date_range(start=None, end=None, periods=None, freq='D', tz=None,
               normalize=False, name=None, closed=None, **kwargs):
    """
    ...
    """
    # Return a linearly spaced DatetimeIndex if `freq` is not set, but `start`, `end`, and `periods` are
    if start and end and periods and not freq:
        di = tools.to_datetime(np.linspace(start, end, periods), **kwargs)
        if name:
            di.name = name
        return di

    return DatetimeIndex(start=start, end=end, periods=periods,
                         freq=freq, tz=tz, normalize=normalize, name=name,
                         closed=closed, **kwargs)

together with appropriate changes in the docstring and test functions. This would provide the desired behaviour, while not changing anything else.

I am not yet a contributor, so I cannot implement this myself (unless I am made one 🙃)

Because the change is really just a convenience feature of the date_range function, I don't think it would be wise to implement this directly in the DatetimeIndex constructor, or in another function, like bdate_range.

@TomAugspurger
Copy link
Contributor

FWI, we could change the default freq to None, and document that it's 'D' when only two of star, end, and freq are specified. That way pd.date_range(start, end, periods=100) will work.

I am not yet a contributor, so I cannot implement this myself (unless I am made one 🙃)

Anyone can make a pull request: http://pandas-docs.github.io/pandas-docs-travis/contributing.html

@onnoeberhard
Copy link
Contributor Author

That's a good idea, I like it much better.

Anyone can make a pull request

Oh okay I didn't know that (never done this before), then I am going to try to implement it :)

@jreback jreback added this to the Next Major Release milestone Apr 30, 2018
@jreback jreback changed the title date_range not working as it intuitively should when specifying start, end, and periods ENH: date_range not working as it intuitively should when specifying start, end, and periods Apr 30, 2018
@jreback jreback modified the milestones: Next Major Release, 0.23.0 May 3, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API Design Datetime Datetime data dtype
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants