Add TileDataset #63
Add TileDataset #63
Conversation
| iterable = self.input_dataset.__iter__(with_key=True) | ||
| else: | ||
| iterable = self.input_dataset | ||
| for example in iterable: |
| item = item + len(self) | ||
| if item < 0: | ||
| raise IndexError(_item) | ||
| if item > self.repetitions * len(self.input_dataset): |
|
Does the following code works? import lazy_dataset
ds = lazy_dataset.new([1, 2, 3])
ds = ds.shuffle(reshuffle=True)
ds = ds.tile(4).catch()
list(ds)We have to handle the combination of non-ordered (e.g. reshuffle), tile, copy(freeze) and indexing. I see two solutions:
|
| """ | ||
| if isinstance(item, str): | ||
| return self.input_dataset[item] | ||
| elif isinstance(item, numbers.Integral): |
There was a problem hiding this comment.
I am not sure, whether this code will have an effect in the performance.
How about changing the code to the follwing?
input_len = len(self.input_dataset)
if not (-self.repetitions <= item // input_len < self.repetitions):
raise IndexError(_item)
return self.input_dataset[item % input_len]
|
|
||
| """ | ||
|
|
||
| def __init__(self, input_dataset, repetitions): |
There was a problem hiding this comment.
Could you rename repetitions to reps?
I prefer to have names close to numpy. (We don't want to do the same as pytorch, where you have to learn the new names for arguments, because they differ to numpy)
| datasets = [ds.shuffle() for ds in datasets] | ||
| return self.__class__.concatenate(*datasets) | ||
| else: | ||
| return TileDataset(self, reps) |
There was a problem hiding this comment.
To minimize overhead: Could your return self, when reps is one? concatenate does this already.
TileDataset should be more efficient than concatenating the input dataset for large
repetitions(in my case in the 1000s)