The Wayback Machine - https://web.archive.org/web/20220119145433/https://github.com/pandas-dev/pandas/issues/30353
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Resampler sometimes adds extra bin with NaN #30353

Closed
vapoto opened this issue Dec 19, 2019 · 3 comments · Fixed by #33498
Closed

Resampler sometimes adds extra bin with NaN #30353

vapoto opened this issue Dec 19, 2019 · 3 comments · Fixed by #33498
Assignees
@hasB4K
Labels
Resample
Milestone
1.1

Comments

@vapoto
Copy link

@vapoto vapoto commented Dec 19, 2019

Code Sample

import pandas as pd
idx = pd.timedelta_range(start='8H', freq='10S', periods=5040)
s = pd.Series(range(len(idx)), index=idx)
s.resample('3H').min()

Observed Output

08:00:00       0.0
11:00:00    1080.0
14:00:00    2160.0
17:00:00    3240.0
20:00:00    4320.0
23:00:00       NaN
Freq: 3H, dtype: float64

Expected Output

08:00:00       0.0
11:00:00    1080.0
14:00:00    2160.0
17:00:00    3240.0
20:00:00    4320.0
Freq: 3H, dtype: float64

Problem description

Downsampling and then applying (any) function to the Resampler object seems to add an extra bin in some cases. It is unclear when this exactly occurs, but these two conditions seem necessary (but not sufficient) to reproduce the bug:

  • the resampling period does not exactly fit in the original index
  • the original index starts with a non-zero Timedelta

This guess is based on the fact that changing '8H' to '0H' in the code above returns an expected output. Equally so does changing the resampling period to '2H'.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.5.final.0
python-bits : 64
OS : Linux
OS-release : 5.0.0-37-generic
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8

pandas : 0.25.3
numpy : 1.17.4
pytz : 2019.3
dateutil : 2.8.1
pip : 19.3.1
setuptools : 42.0.2.post20191203
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.10.3
IPython : 7.10.2
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.1.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : 1.3.2
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
@hasB4K
Copy link
Member

@hasB4K hasB4K commented Apr 12, 2020

take
hasB4K added a commit to hasB4K/pandas that referenced this issue Apr 12, 2020
@hasB4K
BUG: fix pandas-dev#30353 invalid end
5b27c55
@hasB4K hasB4K mentioned this issue Apr 12, 2020
Merged
5 tasks
@hasB4K
Copy link
Member

@hasB4K hasB4K commented Apr 12, 2020 •
edited

Hello @vapoto 🙂,

FYI, I have tracked the error in #33498, it came from a bug of pd.timedelta_ranges (a bug that used to exists for pd.date_range, but it seems it was not implemented for timeldetas).

Your example gives the proper output on my PR.

On master without the fix:

0 days 08:00:00       0.0
0 days 11:00:00    1080.0
0 days 14:00:00    2160.0
0 days 17:00:00    3240.0
0 days 20:00:00    4320.0
0 days 23:00:00       NaN
Freq: 3H, dtype: float64

On my PR:

0 days 08:00:00       0
0 days 11:00:00    1080
0 days 14:00:00    2160
0 days 17:00:00    3240
0 days 20:00:00    4320
Freq: 3H, dtype: int64

Hopefully this should solve your issue!
@vapoto
Copy link
Author

@vapoto vapoto commented Apr 13, 2020

Hi @hasB4K,

Thanks a lot for that! Seems like that would indeed solve the problem. I just added a suggestion to your PR to add tests for some of the cases that are currently working.
hasB4K added a commit to hasB4K/pandas that referenced this issue Apr 13, 2020
@hasB4K
BUG: fix pandas-dev#30353 invalid end
1ec59e6
hasB4K added a commit to hasB4K/pandas that referenced this issue Apr 14, 2020
@hasB4K
BUG: fix pandas-dev#30353 invalid end
5e269d4
hasB4K added a commit to hasB4K/pandas that referenced this issue Apr 17, 2020
@hasB4K
BUG: fix pandas-dev#30353 invalid end
7cb1b6a
hasB4K added a commit to hasB4K/pandas that referenced this issue May 8, 2020
@hasB4K
BUG: fix pandas-dev#30353 invalid end
3ec22c1
hasB4K added a commit to hasB4K/pandas that referenced this issue May 9, 2020
@hasB4K
BUG: fix pandas-dev#30353 invalid end
cab7618
@jreback jreback added this to the 1.1 milestone May 9, 2020
jreback pushed a commit that referenced this issue May 9, 2020
@hasB4K
BUG: Fix a bug in 'timedelta_range' that produced an extra point on a…
7c3f662 
… edge case (fix #30353) (#33498)
rhshadrach added a commit to rhshadrach/pandas that referenced this issue May 10, 2020
@hasB4K @rhshadrach
BUG: Fix a bug in 'timedelta_range' that produced an extra point on a…
9428a5c 
… edge case (fix pandas-dev#30353) (pandas-dev#33498)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Resample
4 participants
@jreback @vapoto @hasB4K @jbrockmendel