ImportError: cannot import name '_device_dtype_check_for_fused'

[jwm1969]

i was trying out the latest winglian/axolotl-cloud:0.5.2

Right after starting training used the adopt_adamw optimizer i get:

ImportError: cannot import name '_device_dtype_check_for_fused' from 'torch.optim.optimizer' (/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/optim/optimizer.py)

I fell back to adamw_torch and it works fine. I probably did not read something i was supposed to.

[winglian]

Try this tag. https://hub.docker.com/layers/axolotlai/axolotl-cloud/main-20241123-py3.11-cu124-2.5.1/images/sha256-15306041c9dddd1332204b2faf0ae6b04d5fde1d7215b0b4227a0947ff327028?context=explore I think the default tag is PyTorch 2.3.1 still and adopt requires a higher version of torch iirc.

[jwm1969]

ok thanks for info. FYI: i pip install torch which installed torch 2.5.1 and dependencies but same error; once i get to the next checkpoint i will try this tag and see if different result.

[jwm1969]

Try this tag. https://hub.docker.com/layers/axolotlai/axolotl-cloud/main-20241123-py3.11-cu124-2.5.1/images/sha256-15306041c9dddd1332204b2faf0ae6b04d5fde1d7215b0b4227a0947ff327028?context=explore I think the default tag is PyTorch 2.3.1 still and adopt requires a higher version of torch iirc.

I am finally doing this test and it is running fine now. I used -20241130- of the same tag you provided above. Question, it looks like adopt made a change on 11-22-2024 to add a Clip function to fix an issue (see below)... I was wondering which version of adopt this tag is using.

https://github.com/iShohei220/adopt

[winglian]

There is a new PR up that will get merged soon with updated ADOPT optimizer. It will also require torch 2.5.1