Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace ImageNet ensemble baseline with Robustness Metrics. #9

Merged
merged 1 commit into from
May 18, 2021

Conversation

copybara-service[bot]
Copy link

@copybara-service copybara-service bot commented May 14, 2021

Replace ImageNet ensemble baseline with Robustness Metrics.

There are two approaches to implement ensembles.

  1. Load all SavedModels into a single model.
  • Pro: Simple to compute results.
  • Con: All models must fit in memory and compute can't parallelize across models.
  1. Eval each model in parallel, saving predictions. Then load predictions and compute metrics. (approach in Uncertainty Baselines)
  • Pro: Scales with compute and memory.
  • Con: Requires two stages (the first uses accelerators, the second is CPU-only). We're already doing the first stage to report non-ensemble results. So two stages is not that inconvenient.

This CL does #2.

Fixes google/uncertainty-baselines#63, google/uncertainty-baselines#71.

Note: I added 'ece' back to the imagenet_variants report.

TODOs in later PRs

  • Loading predictions is slow. Each file is at most 200MB with 50K predictions of 1000 float32 values, and read_predictions shouldn't take this long. np.load gets, say, read speeds of 200 MB/s (https://stackoverflow.com/a/30332316). It may be because we're loading batch_size=1?
  • Replace het_ensemble.py and sngp_ensemble.py.

@copybara-service copybara-service bot force-pushed the test_370938990 branch 4 times, most recently from ee9f2a7 to fa6cafc Compare May 18, 2021 18:30
PiperOrigin-RevId: 374466750
@copybara-service copybara-service bot merged commit 260bc1c into master May 18, 2021
@copybara-service copybara-service bot deleted the test_370938990 branch May 18, 2021 18:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add TPU/multi-GPU support for ensembles
1 participant