Make sure that all workers are notified about end of execution loop (#…

…730)

Currently we will have a hang at the end of script when using TP>1 and
multistep scheduling. This is caused by lack of notification from driver
worker about ending the execution loop.
This is a workaround for this issue, by making sure that all workers are
notified at the end of `llm_engine` loop.
Other possible workaround could be modification of this check:
https://github.com/HabanaAI/vllm-fork/blob/habana_main/vllm/engine/llm_engine.py#L1379
with `or not self.has_unfinished_requests()`.

vllm/entrypoints/llm.py

@@ -1284,6 +1284,10 @@ def _run_engine(
 
         if use_tqdm:
             pbar.close()
+
+        # Make sure that all workers are finished.
+        self.llm_engine.stop_remote_worker_execution_loop()
+
         # Sort the outputs by request ID.
         # This is necessary because some requests may be finished earlier than
         # its previous requests.