DeepSpeed Zero Init + PEFT Number of Parameters

[zanussbaum]

Thanks for a great package, this is great for training some large models.

As far as I can tell when using DeepSpeed Zero3 with Accelerate, the model by default gets loaded with empty weights. But if I were using this and ran the following code snippet

model = AutoModelForCausalLM.from_pretrained(config["model_path"])
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()

it would print something like
trainable params: 3670016 || all params: 3670016 || trainable%: 100.0

In a perfect world, it should print the true number of params (including empty ones)