ZeRO-Inference refresh #722
Conversation
tjruwase
requested review from
jeffra,
samyam,
ShadenSmith,
conglongli,
eltonzheng,
minjiaz,
RezaYazdaniAminabadi,
duli2012,
mrwyattii,
yaozhewei,
arashb and
xiaoxiawu-microsoft
as code owners
tjruwase
removed request for
duli2012,
samyam,
conglongli,
mrwyattii,
yaozhewei,
eltonzheng,
minjiaz,
RezaYazdaniAminabadi and
xiaoxiawu-microsoft
tjruwase
force-pushed
the
staging-zero-inference-v1
branch
from
9f072a8 to
9974256
Compare
tjruwase
force-pushed
the
staging-zero-inference-v1
branch
from
0e51a19 to
6baa73b
Compare
|
@tjruwase It's wonderful to see the optimizations you've integrated into Deepspeed ZeRO-inference, especially without the need for custom APIs. This is a great work! One minor suggestion is that if you used the ideas (e.g., cache offloading) from FlexGen, maybe it is better also to add FlexGen into the reference section. I'm also happy to discuss if you have any problems regarding other optimizations (Partial offloading, Cache quantization). |
|
@Ying1123, thanks for the kind words. You are correct that we should add FlexGen to reference since that was our inspiration for exploring cache offloading and weight quantization. Sorry about this oversight. Is the following the best reference? https://arxiv.org/abs/2303.06865 |
Yes, this is the most updated paper. Thanks! |
* Add zero inference * Fix scripts * Fix scripts * Fix scripts * Fix versioning text * Shrink figure * Shrink figure * Shrink figure * Generality * :q * Tweak repro scripts and README * Fix versions * Fix rqmts * README tweak * Cleanup * Rearrange README * Versioning * cleanup
Refresh with 2 new optimizations: weight quantization and kv cache offloading to CPU.
Companion DS PR: microsoft/DeepSpeed#4197