From e690f6cbe30182bce20c81e061e07cf9348de376 Mon Sep 17 00:00:00 2001 From: ancestor-mithril Date: Wed, 4 Sep 2024 17:25:55 +0300 Subject: [PATCH 1/6] Extended documentation * added tutorials * moved the API Reference from the main page to a secondary one * added links to the main page --- README.md | 7 ++- docs/README.md | 31 ++++++++++++- docs/reference.md | 1 + docs/tutorials.md | 113 ++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 150 insertions(+), 2 deletions(-) create mode 100644 docs/reference.md create mode 100644 docs/tutorials.md diff --git a/README.md b/README.md index 6db7514..7ceef13 100644 --- a/README.md +++ b/README.md @@ -4,12 +4,17 @@ A Batch Size Scheduler library compatible with PyTorch DataLoaders. *** -Documentation: [API Reference](https://ancestor-mithril.github.io/bs-scheduler/). +## Documentation + +* [API Reference](https://ancestor-mithril.github.io/bs-scheduler/reference). + +*** + ## Why use a Batch Size Scheduler? diff --git a/docs/README.md b/docs/README.md index 4d93c2b..8330435 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1 +1,30 @@ -::: bs_scheduler +# bs-scheduler + +A Batch Size Scheduler library compatible with PyTorch DataLoaders. + +*** + +### Batch Size Schedulers + +1. [LambdaBS](https://ancestor-mithril.github.io/bs-scheduler/reference/#bs_scheduler.LambdaBS) - sets the batch size to the base batch size times a given lambda. +2. [MultiplicativeBS](https://ancestor-mithril.github.io/bs-scheduler/reference/#bs_scheduler.MultiplicativeBS) - sets the batch size to the current batch size times a given lambda. +3. [StepBS](https://ancestor-mithril.github.io/bs-scheduler/reference/#bs_scheduler.StepBS) - multiplies the batch size with a given factor at a given number of steps. +4. [MultiStepBS](https://ancestor-mithril.github.io/bs-scheduler/reference/#bs_scheduler.MultiStepBS) - multiplies the batch size with a given factor each time a milestone is reached. +5. [ConstantBS](https://ancestor-mithril.github.io/bs-scheduler/reference/#bs_scheduler.ConstantBS) - multiplies the batch size by a given factor once and decreases it again to its base value after a + given number of steps. +6. [LinearBS](https://ancestor-mithril.github.io/bs-scheduler/reference/#bs_scheduler.LinearBS) - increases the batch size by a linearly changing multiplicative factor for a given number of steps. +7. [ExponentialBS](https://ancestor-mithril.github.io/bs-scheduler/reference/#bs_scheduler.ExponentialBS) - increases the batch size by a given $\gamma$ each step. +8. [PolynomialBS](https://ancestor-mithril.github.io/bs-scheduler/reference/#bs_scheduler.PolynomialBS) - increases the batch size using a polynomial function in a given number of steps. +9. [CosineAnnealingBS](https://ancestor-mithril.github.io/bs-scheduler/reference/#bs_scheduler.CosineAnnealingBS) - increases the batch size to a maximum batch size and decreases it again following a cyclic + cosine curve. +10. [IncreaseBSOnPlateau](https://ancestor-mithril.github.io/bs-scheduler/reference/#bs_scheduler.IncreaseBSOnPlateau) - increases the batch size each time a given metric has stopped improving for a given number + of steps. +11. [CyclicBS](https://ancestor-mithril.github.io/bs-scheduler/reference/#bs_scheduler.CyclicBS) - cycles the batch size between two boundaries with a constant frequency, while also scaling the + distance between boundaries. +12. [CosineAnnealingBSWithWarmRestarts](https://ancestor-mithril.github.io/bs-scheduler/reference/#bs_scheduler.CosineAnnealingBSWithWarmRestarts) - increases the batch size to a maximum batch size following a cosine curve, + then restarts while also scaling the number of iterations until the next restart. +13. [OneCycleBS](https://ancestor-mithril.github.io/bs-scheduler/reference/#bs_scheduler.OneCycleBS) - decreases the batch size to a minimum batch size then increases it to a given maximum batch size, + following a linear or cosine annealing strategy. +14. [SequentialBS](https://ancestor-mithril.github.io/bs-scheduler/reference/#bs_scheduler.SequentialBS) - calls a list of schedulers sequentially given a list of milestone points which reflect which + scheduler should be called when. +15. [ChainedBSScheduler](https://ancestor-mithril.github.io/bs-scheduler/reference/#bs_scheduler.ChainedBSScheduler) - chains a list of batch size schedulers and calls them together each step. diff --git a/docs/reference.md b/docs/reference.md new file mode 100644 index 0000000..4d93c2b --- /dev/null +++ b/docs/reference.md @@ -0,0 +1 @@ +::: bs_scheduler diff --git a/docs/tutorials.md b/docs/tutorials.md new file mode 100644 index 0000000..dcaae85 --- /dev/null +++ b/docs/tutorials.md @@ -0,0 +1,113 @@ +## Basic usage + +Integrating a Batch Size Scheduler inside a PyTorch training script is simple: + +```python +from torch.utils.data import DataLoader +from bs_scheduler import StepBS # We use StepBS in this example, but we can use any BS Scheduler + +# Define the Dataset and the DataLoader +dataset = ... +dataloader = DataLoader(..., batch_size=16) +scheduler = StepBS(dataloader, step_size=30, gamma=2) # Activates every 30 epochs and doubles the batch size. + +for _ in range(100): + train(...) + validate(...) + scheduler.step() + +# We expect the batch size to have the following values: +# epoch 0 - 29: 16 +# epoch 30 - 59: 32 +# epoch 60 - 89: 64 +# epoch 90 - 99: 128 +``` + +Full example: + +```python +import timm +import torch.cuda +import torchvision.datasets +from torch import nn +from torch.optim import SGD +from torch.utils.data import DataLoader +from torchvision.transforms import v2 + +from bs_scheduler import StepBS + +device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + +transforms = v2.Compose([v2.ToImage(), v2.ToDtype(torch.float32, scale=True)]) +train_loader = DataLoader( + torchvision.datasets.CIFAR10( + root="../data", + train=True, + download=True, + transform=transforms, + ), + batch_size=100, +) +val_loader = DataLoader( + torchvision.datasets.CIFAR10(root="../data", train=False, transform=transforms), + batch_size=500, +) +scheduler = StepBS(train_loader, step_size=10) + +model = timm.create_model("hf_hub:grodino/resnet18_cifar10", pretrained=False).to( + device +) +criterion = nn.CrossEntropyLoss() +optimizer = SGD(model.parameters(), lr=0.001) + + +def train(): + correct = 0 + total = 0 + + model.train() + for inputs, targets in train_loader: + inputs, targets = inputs.to(device), targets.to(device) + outputs = model(inputs) + loss = criterion(outputs, targets) + loss.backward() + optimizer.step() + optimizer.zero_grad() + + predicted = outputs.argmax(1) + total += targets.size(0) + correct += predicted.eq(targets).sum().item() + + return correct / total + + +@torch.inference_mode() +def val(): + correct = 0 + total = 0 + + model.eval() + for inputs, targets in val_loader: + inputs, targets = inputs.to(device), targets.to(device) + outputs = model(inputs) + + predicted = outputs.argmax(1) + total += targets.size(0) + correct += predicted.eq(targets).sum().item() + + return correct / total + + +def main(): + for epoch in range(100): + train_accuracy = train() + val_accuracy = val() + + scheduler.step() + + print(train_accuracy, val_accuracy) + + +if __name__ == "__main__": + main() +``` From f958efa269c268f1ca093f043eb505825cb8f8d2 Mon Sep 17 00:00:00 2001 From: ancestor-mithril Date: Wed, 4 Sep 2024 17:28:12 +0300 Subject: [PATCH 2/6] Moved comments on new line for better visibility --- docs/tutorials.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/tutorials.md b/docs/tutorials.md index dcaae85..112f98c 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -4,12 +4,14 @@ Integrating a Batch Size Scheduler inside a PyTorch training script is simple: ```python from torch.utils.data import DataLoader -from bs_scheduler import StepBS # We use StepBS in this example, but we can use any BS Scheduler +from bs_scheduler import StepBS +# We use StepBS in this example, but we can use any BS Scheduler # Define the Dataset and the DataLoader dataset = ... dataloader = DataLoader(..., batch_size=16) -scheduler = StepBS(dataloader, step_size=30, gamma=2) # Activates every 30 epochs and doubles the batch size. +scheduler = StepBS(dataloader, step_size=30, gamma=2) +# Activates every 30 epochs and doubles the batch size. for _ in range(100): train(...) From 2353dcb7a232d3a4381550b7a2bd6e3222c87e5c Mon Sep 17 00:00:00 2001 From: ancestor-mithril Date: Wed, 4 Sep 2024 17:31:07 +0300 Subject: [PATCH 3/6] Changed API reference link to Home Page --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7ceef13..5d0741a 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ A Batch Size Scheduler library compatible with PyTorch DataLoaders. ## Documentation -* [API Reference](https://ancestor-mithril.github.io/bs-scheduler/reference). +* [API Reference](https://ancestor-mithril.github.io/bs-scheduler). From e7b79b8e4c1577da4127866ac6b02bed0a765f84 Mon Sep 17 00:00:00 2001 From: ancestor-mithril Date: Wed, 4 Sep 2024 17:33:45 +0300 Subject: [PATCH 4/6] Updated README.md --- README.md | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/README.md b/README.md index 5d0741a..5a8d13b 100644 --- a/README.md +++ b/README.md @@ -8,8 +8,7 @@ A Batch Size Scheduler library compatible with PyTorch DataLoaders. * [API Reference](https://ancestor-mithril.github.io/bs-scheduler). - - +* [Examples](https://ancestor-mithril.github.io/bs-scheduler/tutorials) @@ -62,12 +61,6 @@ Please install [PyTorch](https://github.com/pytorch/pytorch) first before instal pip install bs-scheduler ``` -Or from git: - -``` -pip install git+https://github.com/ancestor-mithril/bs-scheduler.git@master -``` - ## Licensing The library is licensed under the [BSD-3-Clause license](LICENSE). From 66f25e6f72011d00e3425b4abbe61a469e313dd0 Mon Sep 17 00:00:00 2001 From: ancestor-mithril Date: Thu, 5 Sep 2024 11:50:46 +0300 Subject: [PATCH 5/6] Updated README with 'Why use? ' --- README.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 5a8d13b..abe8c18 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ A Batch Size Scheduler library compatible with PyTorch DataLoaders. * [API Reference](https://ancestor-mithril.github.io/bs-scheduler). -* [Examples](https://ancestor-mithril.github.io/bs-scheduler/tutorials) +* [Examples](https://ancestor-mithril.github.io/bs-scheduler/tutorials). @@ -16,7 +16,14 @@ A Batch Size Scheduler library compatible with PyTorch DataLoaders. ## Why use a Batch Size Scheduler? - +* Using a big batch size has several advantages: + * Better hardware utilization. + * Enhanced parallelism. + * Faster training. +* However, using a big batch size from the start may lead to a generalization gap. +* Therefore, the solution is to gradually increase the batch size, similar to a learning rate decay policy. +* See [Don't Decay the Learning Rate, Increase the Batch Size](https://arxiv.org/abs/1711.00489). + ## Available Schedulers From 4983e9e66527714e0c3cf7e8e2ffef8de4d725f2 Mon Sep 17 00:00:00 2001 From: ancestor-mithril Date: Thu, 5 Sep 2024 11:54:14 +0300 Subject: [PATCH 6/6] Added citation stub --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index abe8c18..3df4989 100644 --- a/README.md +++ b/README.md @@ -72,4 +72,8 @@ pip install bs-scheduler The library is licensed under the [BSD-3-Clause license](LICENSE). +## Citation + +To be added... +