[Fix] Fix async append result in different order issue

[kcz358]

Before you open a pull-request, please check if a similar issue already exists or has been closed before.

When you open a pull-request, please be sure to include the following

• A descriptive title: [xxx] XXXX
• A detailed description

Thank you for your contributions!

---

As stated in #243 , there is an error in current asyncio srt model. This PR tries to fix this and reproduce the result

[kcz358]

The result can be reproduced now by srt model. The result are mostly aligned with the paper result.

Noted that current sglang main seems to have some issue and can not reproduce the result. I have checked that you are be able to reproduce the result by

cd sglang
git checkout 2f1d92834f41df42e266ed6d7036b4add906d21f

My current result using srt api model on llava-ov-0.5B

AI2D

MME

When using current main branch sglang, I can only get 216/1066 which is relatively low. I will try to check what changes in sglang caused this issue.