Minimal RAM requirements?

[shinandrew]

I have a RTX 3080 16GB with system RAM of 8 GB.

I know that this is not even remotely close to being able to run the model as is, so I tried reducing all batch sizes to 1, enabling gradient checkpointing, loading the model with float16, etc.

But I still get the following error.

ray.exceptions.OutOfMemoryError: Task was killed due to the node running low on memory.
Memory on the node (IP: 172.24.24.131, ID: 43022a1cd943460041e5c660f4cb788c1284d07c7d0c7a541170cc35) where the task (task ID: 9f448da775b9e7f6af4c9926742724f77b58f6a001000000, name=main_task, pid=32705, memory used=1.48GB) was running was 7.58GB / 7.61GB (0.9956), which exceeds the memory usage threshold of 0.99. Ray killed this worker (ID: 92fd5428ccbaab414ff5ae0f92082e71205d4e7301e79b86f2723bc5) because it was the most recently scheduled task; to see more information about memory usage on this node, use ray logs raylet.out -ip 172.24.24.131. To see the logs of the worker, use `ray logs worker-92fd5428ccbaab414ff5ae0f92082e71205d4e7301e79b86f2723bc5*out -ip 172.24.24.131. Top 10 memory users:

What would be the minimal possible configuration to run this experiment even at very low performance?

[AvisP]

If you check the discussion here you will see that it needs at least one 80 GB memory to train a Qwen 2.5 0.5B model and two of them for Qwen 2.5 3B. I found that the former needs about 60 GB of memory at least, and reducing batch_size won't solve the issue as when training the model takes 4 times the memory while training and then GRPO/PPO take a lot of memory. So you cannot train anything on 3080 with all the memory reductions you can think of. I would suggest renting GPUs on the cloud and try them out.