[DeepSpeed-Chat] Fix OOM issue in dataloader#841

Open

youkaichao wants to merge 1 commit intodeepspeedai:masterfrom youkaichao:fix_oom

+39-39

Commits on Jan 1, 2024

use online tokenizer to avoid oom
youkaichao
committed