How do I get the deepspeed env file not to be re-generated?

[shing100]

There are a lot of nodes on the host, so when running as a multi-node, the deep speed env file continues to be created, causing problems.
Argument list too long: 'pdsh'

Is there a solution?

[NanoCode012]

Hello, could you clarify what you mean by deepspeed env file? Is it the zero*.json?

[shing100]

this file '.deepspeed_env'

[NanoCode012]

Thanks for clarifying. The issue is that your cli command being passed to pdsh is too long, am I understanding it right?

I'm not so familiar with this issue. Could you give me an example of how you run the command?

[shing100]

yes!

multi-node setting and run training (accelerate launch -m axolotl.cli.train your_config.yml)

[NanoCode012]

@shing100 , could you perhaps open a discussion on Deepspeed's side to see what their recommendation is? I don't think this issue is from axolotl, and they may have more expertise on this.

[shing100]

okay thanks!!

[NanoCode012]

I'm closing this for now but do update us if you do find a workaround :)