[Misc]: How to access the KV cache directly?

[BDHU]

Anything you want to discuss about vllm.

I'm looking to conduct an experiment, which involves copying the contents of KV cache between nodes. I'm not super familiar with the codebase, is there any way to access the page table/KV cache directly? Where do I start? Any suggestions are helpful!

[duanzhaol]

Curios about this topic too, I want to implement a simple request transfer (including kv cache) between nodes. #2809 seems did it, but only support with infiniband, and has a dependency on MSCCL++.

[BDHU]

Any updates on this?

[tanejaaryan]

interested in this as well, can anyone guide a few first steps?

[CSEEduanyu]

just use cudaIPChandle and cudamemcopy

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[J1nLo]

Any updates on this?