fix the grad_acc issue at epoch boundaries

[pacman100]

What does this PR do?

Should solve accumulating via epoch when using Accelerate (seen in #23935 (comment)). Requires huggingface/accelerate#1624

Fixes # (issue)

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[nivibilla]

I believe this PR also fixes #24245

[muellerzr]

Thanks!

[amyeroberts]

Thanks for fixing!

Happy with the changes. My only concern is things breaking as this relies on feature that's just been merged on accelerate's main branch. Is the next release of accelerate going to be before the next release of transformers?

@pacman100 @muellerzr You've been handling the switch using accelerate in trainer really well, and I can see our CI runs install accelerate from source, so happy if this follows the pattern you've using.

[muellerzr]

@amyeroberts we coordinate Accelerate releases to be a day or two before transformers, so there shouldn't be an issue there :)

(Though @pacman100 we should do the version check like we've done before with these fixes 😬 )

[sgugger]

Thanks for the work on this! There needs to be a couple of protections before this is merged (that we can remove at the next release of Accelerate when we move the pinned version up).

[Oxi84]

Hello,

How do i install this? I expected that PR means some kind of transformers update, in this case there should be install link such as git+https://github.com/huggingface/transformers@de9255de27abfcae4a1f816b904915f0b1e23cd9

[pacman100]

Hello @Oxi84, you can install this once it gets merged via pip install git+https://github.com/huggingface/transformers and pip install git+https://github.com/huggingface/accelerate

[philpax]

Just completed a training run with this PR and can confirm that the issue didn't occur. Thanks for the fix!