Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🚚 Refactor Benchmarking Script #1216

Merged
merged 5 commits into from
Aug 4, 2023
Merged

🚚 Refactor Benchmarking Script #1216

merged 5 commits into from
Aug 4, 2023

Conversation

ashwinvaidya17
Copy link
Collaborator

Description

  • While this started as an exercise to simplify the script, it probably has become more complicated. But the outputs look pretty now πŸ€·β€β™‚οΈ
    image

  • All outputs are logged to runs/benchmark.logs.

  • I've removed throughput computations as it needs to be rafactored and we don't use it anywhere.

  • Also, I've tried to ensure that any exception does not break the benchmarking run so that the script keeps collecting results for the models that work.

Note: wandb and comet uploads haven't been tested yet

Changes

  • Bug fix (non-breaking change which fixes an issue)
  • Refactor (non-breaking change which refactors the code base)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Checklist

  • My code follows the pre-commit style and check guidelines of this project.
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing tests pass locally with my changes
  • I have added a summary of my changes to the CHANGELOG (not for minor changes, docs and tests).

@ashwinvaidya17 ashwinvaidya17 requested a review from djdameln July 25, 2023 11:10
djdameln
djdameln reviewed Jul 27, 2023
View reviewed changes
Copy link
Contributor

@djdameln djdameln left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think we could simplify the exception handling somehow? Catching the exceptions in various different places make it a bit hard to read the code.

Also, I'm wondering if it would be possible to log the results to wandb as they are collected, instead of all at once at the end of the script. In some cases could be useful to see the results come in as they are generated.

tools/benchmarking/benchmark.py Outdated


def hide_output(func):
"""Decorator to hide output of the function.
def redirect_output(func):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function could be moved to utils to keep this file a bit cleaner.

@github-actions github-actions bot added the Dependencies Pull requests that update a dependency file label Jul 28, 2023
@ashwinvaidya17 ashwinvaidya17 requested a review from yunchu as a code owner July 31, 2023 07:33
@github-actions github-actions bot added the Setup label Jul 31, 2023
@samet-akcay samet-akcay merged commit 784767f into openvinotoolkit:main Aug 4, 2023
samet-akcay added a commit that referenced this pull request Aug 6, 2023
@samet-akcay @ashwinvaidya17
Merge main to feature/benchmarking (#1236)
b730050 
* Configure readthedocs via `.readthedocs.yaml` file (#1229)

* 🚚 Refactor Benchmarking Script (#1216)

* New printing stuff

* Remove dead code + address codacy issues

* Refactor try/except + log to comet/wandb during runs

* pre-commit error

* third-party configuration

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* Update CODEOWNERS

---------

Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
samet-akcay added a commit that referenced this pull request Aug 7, 2023
@samet-akcay @ashwinvaidya17
Add DFM, EfficientAD, FastFlow and STFPM results on BTech, MVTecAD an…
f99b3f9 
…d Visa benchmarks (#1235)

* Fix metadata path

* Update readme files

* ignore runs directory

* ignore runs directory

* Add DFM MVTec AD results

* Add Fastflow MVTec AD results

* Fix typo

* F1 Score to F1

* Add STFPM MVTec AD results

* Add visa dataset results

* Visa - 3 decimal points

* MVTec - 3 decimal points

* Configure readthedocs via `.readthedocs.yaml` file (#1229)

* 🚚 Refactor Benchmarking Script (#1216)

* New printing stuff

* Remove dead code + address codacy issues

* Refactor try/except + log to comet/wandb during runs

* pre-commit error

* third-party configuration

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* Add efficient_ad mvtec results

* Add efficient_ad visa results

* Add efficient_ad btech results

* Add dfm btech results

* Add fastflow btech results

* Add stfpm btech results

* sort visa categories by alphabetical order in efficientad results

* Transform efficient-ad mvtec-ad results to title case

* Transform efficient-ad mvtec-ad results to title case and sort categories

---------

Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
samet-akcay added a commit that referenced this pull request Aug 7, 2023
@samet-akcay
Revert "🚚 Refactor Benchmarking Script (#1216)"
d4eaee3 
This reverts commit 784767f.
@samet-akcay samet-akcay mentioned this pull request Aug 7, 2023
Merged
samet-akcay added a commit that referenced this pull request Aug 7, 2023
@samet-akcay
Revert "🚚 Refactor Benchmarking Script" (#1239)
a62cd92 
Revert "🚚 Refactor Benchmarking Script (#1216)"

This reverts commit 784767f.
samet-akcay added a commit that referenced this pull request Aug 9, 2023
@samet-akcay @ashwinvaidya17 @djdameln
Merge main into feature/pimo (#1248)
1dd9434 
* Configure readthedocs via `.readthedocs.yaml` file (#1229)

* 🚚 Refactor Benchmarking Script (#1216)

* New printing stuff

* Remove dead code + address codacy issues

* Refactor try/except + log to comet/wandb during runs

* pre-commit error

* third-party configuration

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* Update CODEOWNERS

* Enable training with only normal images for MVTec (#1241)

* ignore mask check when dataset has only normal samples

* update changelog

* Revert "🚚 Refactor Benchmarking Script" (#1239)

Revert "🚚 Refactor Benchmarking Script (#1216)"

This reverts commit 784767f.

* Update benchmarking notebook (#1242)

* Fix metadata path

* Update benchmarking notebook

---------

Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Dick Ameln <[email protected]>
samet-akcay added a commit that referenced this pull request Aug 10, 2023
@samet-akcay @ashwinvaidya17
@djdameln @blaz-r @Seanny123
Merge main to feature/benchmarking (#1258)
6a8099d 
* Configure readthedocs via `.readthedocs.yaml` file (#1229)

* 🚚 Refactor Benchmarking Script (#1216)

* New printing stuff

* Remove dead code + address codacy issues

* Refactor try/except + log to comet/wandb during runs

* pre-commit error

* third-party configuration

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* Update CODEOWNERS

* Enable training with only normal images for MVTec (#1241)

* ignore mask check when dataset has only normal samples

* update changelog

* Revert "🚚 Refactor Benchmarking Script" (#1239)

Revert "🚚 Refactor Benchmarking Script (#1216)"

This reverts commit 784767f.

* Update benchmarking notebook (#1242)

* Fix metadata path

* Update benchmarking notebook

* Fix links to model architecture images (#1245)

* Fix links to architecture images

* Change links to raw files

* Wandb unwatch method belongs to experiment, not logger (#1246)

unwatch method belongs to experiment, not logger

---------

Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Dick Ameln <[email protected]>
Co-authored-by: BlaΕΎ Rolih <[email protected]>
Co-authored-by: Sean Aubin <[email protected]>
samet-akcay added a commit that referenced this pull request Aug 15, 2023
@samet-akcay @ashwinvaidya17
@djdameln @blaz-r @Seanny123 @JoaoGuibs
Merge main into feature/pimo (#1273)
f62fde0 
* Configure readthedocs via `.readthedocs.yaml` file (#1229)

* 🚚 Refactor Benchmarking Script (#1216)

* New printing stuff

* Remove dead code + address codacy issues

* Refactor try/except + log to comet/wandb during runs

* pre-commit error

* third-party configuration

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* Update CODEOWNERS

* Enable training with only normal images for MVTec (#1241)

* ignore mask check when dataset has only normal samples

* update changelog

* Revert "🚚 Refactor Benchmarking Script" (#1239)

Revert "🚚 Refactor Benchmarking Script (#1216)"

This reverts commit 784767f.

* Update benchmarking notebook (#1242)

* Fix metadata path

* Update benchmarking notebook

* Fix links to model architecture images (#1245)

* Fix links to architecture images

* Change links to raw files

* Wandb unwatch method belongs to experiment, not logger (#1246)

unwatch method belongs to experiment, not logger

* (Minor change) Added the tracer_kwargs to the TorchFXFeatureExtractor class (#1214)

* Added tracer_kwargs to torchfx

* Added tracer_kwargs on docstring

* Replace cdist in Patchcore (#1267)

* Ignore hidden directories when creating `Folder` dataset (#1268)

* Remove `config` from argparse in OpenVINO inference script. (#1257)

* Fix metadata path

* Remove leftover argument

* Update openvino entrypoint script

* Fix EfficientAD number of steps for optimizer lr change? (#1266)

* Fix metadata path

* Fix number of steps

---------

Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Dick Ameln <[email protected]>
Co-authored-by: BlaΕΎ Rolih <[email protected]>
Co-authored-by: Sean Aubin <[email protected]>
Co-authored-by: JoaoGuibs <[email protected]>
samet-akcay added a commit that referenced this pull request Aug 23, 2023
@jpcbertoldo @samet-akcay
@ashwinvaidya17 @djdameln
metrics[0]: add per-image overlap (pimo) (#1247)
103c22c 
* add per-image overlap (pimo)

* modif plot pimo curves

* add warning about memory

* tiny bug

* add tuto ipynb

* make image classes a return

* fix ipynb

* add tests for binclf curve

* add test to binclf

* add aupimo tests

* ruff

* Configure readthedocs via `.readthedocs.yaml` file (#1229)

* Update binclf_curve.py

* 🚚 Refactor Benchmarking Script (#1216)

* New printing stuff

* Remove dead code + address codacy issues

* Refactor try/except + log to comet/wandb during runs

* pre-commit error

* third-party configuration

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* Update CODEOWNERS

* Enable training with only normal images for MVTec (#1241)

* ignore mask check when dataset has only normal samples

* update changelog

* Revert "🚚 Refactor Benchmarking Script" (#1239)

Revert "🚚 Refactor Benchmarking Script (#1216)"

This reverts commit 784767f.

* Update benchmarking notebook (#1242)

* Fix metadata path

* Update benchmarking notebook

* add per-image overlap (pimo)

* modif plot pimo curves

* add warning about memory

* tiny bug

* add tuto ipynb

* make image classes a return

* fix ipynb

* add tests for binclf curve

* add test to binclf

* add aupimo tests

* ruff

* Update binclf_curve.py

* refactor from future pr

* correct tests

* add test

* fix test

* add plots tests

* correct codacy stuff

* correct codacy stuff

---------

Co-authored-by: Samet Akcay <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Dick Ameln <[email protected]>
samet-akcay added a commit that referenced this pull request Sep 12, 2023
@jpcbertoldo @samet-akcay
@ashwinvaidya17 @djdameln
metrics[1]: auc boxplot [GSoC 2023 @ OpenVINO] (#1294)
c7baf40 
* add per-image overlap (pimo)

* modif plot pimo curves

* add warning about memory

* tiny bug

* add tuto ipynb

* make image classes a return

* fix ipynb

* add tests for binclf curve

* add test to binclf

* add aupimo tests

* ruff

* Configure readthedocs via `.readthedocs.yaml` file (#1229)

* Update binclf_curve.py

* 🚚 Refactor Benchmarking Script (#1216)

* New printing stuff

* Remove dead code + address codacy issues

* Refactor try/except + log to comet/wandb during runs

* pre-commit error

* third-party configuration

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* Update CODEOWNERS

* Enable training with only normal images for MVTec (#1241)

* ignore mask check when dataset has only normal samples

* update changelog

* Revert "🚚 Refactor Benchmarking Script" (#1239)

Revert "🚚 Refactor Benchmarking Script (#1216)"

This reverts commit 784767f.

* Update benchmarking notebook (#1242)

* Fix metadata path

* Update benchmarking notebook

* add per-image overlap (pimo)

* modif plot pimo curves

* add warning about memory

* tiny bug

* add tuto ipynb

* make image classes a return

* fix ipynb

* add tests for binclf curve

* add test to binclf

* add aupimo tests

* ruff

* Update binclf_curve.py

* refactor from future pr

* add auc boxplot

* Apply suggestions from code review

* update demo nb

* correct tests

* add test

* fix test

* add plots tests

* add tests to pimo

* fix plt warning

* fix docstring warning

* add tests to common

* add tests for plot module and small fixes

* --amend

* clear ouputs in notebook

* correct typo

* correct codacy stuff

* correct codacy stuff

* merge

* fix kernel spec in 502_perimg_metrics.ipynb

* fix types in boxplot

* Update src/anomalib/utils/metrics/perimg/pimo.py

Co-authored-by: Samet Akcay <[email protected]>

---------

Co-authored-by: Samet Akcay <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Dick Ameln <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@djdameln djdameln djdameln left review comments

@samet-akcay samet-akcay samet-akcay approved these changes

@yunchu yunchu Awaiting requested review from yunchu

Assignees
No one assigned
Labels
Benchmarking Dependencies Pull requests that update a dependency file Setup Tools
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ashwinvaidya17 @samet-akcay @djdameln