Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"docker live stop" command failing due to unescaped '$' character #367

Closed
Grenite opened this issue Oct 2, 2023 · 1 comment
Closed
Labels
bug Something isn't working

Comments

@Grenite
Copy link

Grenite commented Oct 2, 2023

In my local algorithm deployments I frequently experienced an issue where the command "docker live stop" did not stop my live deployments and resulting in errors like this:

Context information:
  Python version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
  OS: Linux-6.2.0-33-generic-x86_64-with-glibc2.35
  Lean CLI version: 1.0.169
live.send_command(): stop - sending command. 
live.get_result(): stop -  waiting for results...
Traceback (most recent call last):
  File "/home/constantine/.local/lib/python3.10/site-packages/lean/main.py", line 94, in main
    lean.main(standalone_mode=False)
  File "/home/constantine/.local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/constantine/.local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/constantine/.local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/constantine/.local/lib/python3.10/site-packages/lean/click.py", line 159, in invoke
    result = super().invoke(ctx)
  File "/home/constantine/.local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/constantine/.local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/constantine/.local/lib/python3.10/site-packages/lean/commands/live/stop.py", line 37, in stop
    get_result(command_id, docker_container_name)
  File "/home/constantine/.local/lib/python3.10/site-packages/lean/commands/live/live.py", line 88, in get_result
    raise Exception((f"live.get_result(): {stack()[1].function} - "
Exception: live.get_result(): stop - Failed: to execute the command successfully. Failed to read result from docker file
result-command-6b31a490ac444ef88713ffbb6d082ee0.json within 30 seconds
Error: live.get_result(): stop - Failed: to execute the command successfully. Failed to read result from docker file
result-command-6b31a490ac444ef88713ffbb6d082ee0.json within 30 seconds

I have determined that in the file lean/commands/live/stop.py the issue lies upon the following variable "data":

    data = {
        "$type": "QuantConnect.Commands.AlgorithmStatusCommand, QuantConnect.Common",
        "Id": command_id,
        "Status": "Stopped"
    }

where the "$" character in the json member "$type" is not being escaped when it is passed to the docker exec command in function write_to_file() in lean-cli/lean/components/docker/docker_manager.py and thus fails the write the stop command to the docker container.
I found a workaround by adding a line after line 450 in the write_to_file() function of docker_manager.py to the following:

        data = data.replace('"','\\"')
        data = data.replace('$','\\$')
        command = f'docker exec {docker_container_name} bash -c "echo \'{data}\' > {docker_file.as_posix()}"'

And after adding that line the error no longer occurs

@Martin-Molinero
Copy link
Member

Hey @Grenite!
We believe this was due to a delay on lean engine shutting down, we've increased the command timeout because it was tight and improve the error log. If it happens again please run docker logs {container name} so we can look closer at what is lean doing, which is causing the delay. To get the docker container name you can run docker ps. Release in lean cli version 1.0.171
Thank you, closing the issue for now 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants