Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix iteration timing used in autotuning when gradient_accumulation_steps > 1 #2888

Merged
merged 12 commits into from
Sep 6, 2023
Merged

fix iteration timing used in autotuning when gradient_accumulation_steps > 1 #2888

Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
feb95e2
fix iteration timing when gas > 1
cli99 Feb 23, 2023
364c73c
fix formatting
cli99 Feb 24, 2023
2a2c4bd
Merge branch 'master' into flops-profiler-iter-timing
cli99 Feb 24, 2023
1cb113f
Merge branch 'master' into flops-profiler-iter-timing
tjruwase Mar 24, 2023
968591a
Merge branch 'master' into flops-profiler-iter-timing
cli99 Jun 23, 2023
607d8dd
Merge branch 'master' into flops-profiler-iter-timing
cli99 Jun 27, 2023
cf3d7a8
Merge branch 'master' into flops-profiler-iter-timing
cli99 Jun 28, 2023
7bfacf0
Merge branch 'master' into flops-profiler-iter-timing
loadams Aug 23, 2023
dd4199a
Merge branch 'master' into flops-profiler-iter-timing
loadams Aug 23, 2023
f3d33d4
Merge branch 'master' into flops-profiler-iter-timing
loadams Aug 24, 2023
ca97898
Merge branch 'master' into flops-profiler-iter-timing
loadams Sep 6, 2023
dcab379
Merge branch 'master' into flops-profiler-iter-timing
loadams Sep 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion deepspeed/runtime/engine.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2149,6 +2149,7 @@ def step(self, lr_kwargs=None):
if flops_profiler_active:
if self.autotuning_enabled():
self.flops = self.flops_profiler.get_total_flops() * 3
self.fwd_duration = self.flops_profiler.get_total_duration()
else:
self.flops_profiler.print_model_profile(
profile_step=self.global_steps,
Expand Down Expand Up @@ -2203,7 +2204,7 @@ def _autotuning_exit(self):
titer += msg[FORWARD_GLOBAL_TIMER] if FORWARD_GLOBAL_TIMER in msg else 0
titer += msg[BACKWARD_GLOBAL_TIMER] if BACKWARD_GLOBAL_TIMER in msg else 0
titer += msg[STEP_GLOBAL_TIMER] if STEP_GLOBAL_TIMER in msg else 0

titer *= self.gradient_accumulation_steps()
msg["latency"] = titer
msg["FLOPS_per_gpu"] = self.flops * 1_000_000 * self.gradient_accumulation_steps() / titer
msg["throughput"] = self.train_batch_size() * 1_000_000 / \
Expand Down