Handling of last batch?

[Hoeze]

Hi, thanks for this very useful library!
How does xbatcher handle the last batch? Does it throw it away or return a reduced batch?

[jhamman]

I know this is an old question but I there's a simple answer so I'll go ahead and answer it. Partial batches are currently discarded. Here's a quick demo that shows this behavior:

In [1]: import xbatcher

In [2]: import xarray as xr

In [3]: ds = xr.tutorial.open_dataset('air_temperature')

In [4]: ds['air']
Out[4]:
<xarray.DataArray 'air' (time: 2920, lat: 25, lon: 53)>
[3869000 values with dtype=float32]
Coordinates:
  * lat      (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0
  * lon      (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0
  * time     (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00
Attributes:
    long_name:     4xDaily Air temperature at sigma level 995
    units:         degK
    precision:     2
    GRIB_id:       11
    GRIB_name:     TMP
    var_desc:      Air temperature
    dataset:       NMC Reanalysis
    level_desc:    Surface
    statistic:     Individual Obs
    parent_stat:   Other
    actual_range:  [185.16 322.1 ]

In [5]: for batch in ds['air'].batch.generator({'time': 100}):
   ...:     pass
   ...:

In [6]: batch
Out[6]:
<xarray.Dataset>
Dimensions:  (sample: 1325, time: 100)
Coordinates:
  * time     (time) datetime64[ns] 2014-12-02 ... 2014-12-26T18:00:00
  * sample   (sample) MultiIndex
  - lat      (sample) float64 75.0 75.0 75.0 75.0 75.0 ... 15.0 15.0 15.0 15.0
  - lon      (sample) float64 200.0 202.5 205.0 207.5 ... 325.0 327.5 330.0
Data variables:
    air      (sample, time) float32 240.7 244.4 257.1 ... 298.7 299.4 299.7