ConEmu leaves ConEmuANSI set to ON when a process spawns a child process in a new console

[rprichard]

Versions

ConEmu build: 170305 x64
OS version: W7 x64s 7 x64
Used shell version: cmd, ipython3 5.2.2

Summary

The ConEmuANSI variable is set to ON in a non-ConEmu console spawned from ConEmu, and if it's set in a process that doesn't have the ConEmu hook DLL in it, the color escapes are visible. ConEmu sometimes intervenes and sets ConEmuANSI to OFF, but it doesn't do that in a couple of situations.

Problem description

When I start a process inside ConEmu, with the default settings, ConEmu sets the ConEmuANSI environment variable to ON. This makes sense, because ConEmu has a hook DLL that adds ANSI escape sequence support.

If I spawn a child process in a new console, the hook DLL tends to stop being injected, and ConEmu sometimes sets ConEmuANSI to OFF.

To install ipython3

(I haven't tested this, but I think this is correct.)

1. Install Python3 somewhere
2. Run <install-dir>\Scripts\pip.exe install ipython

Scenario 1: using start cmd

1. Reset settings to default
2. Use the default {Shells} > {cmd} task.
3. Run start cmd in the shell
4. A new non-ConEmu console window appears with a cmd.exe shell in it.
5. The cmd.exe prompt is colored in this shell, indicating that the hook DLL is present.
6. In the new shell, the ConEmuANSI variable is OFF, but ConEmuHooks is Enabled.
7. In the new shell, run cmd
8. Now the cmd prompt's escapes are visible. ConEmuANSI is still OFF and ConEmuHooks is still Enabled.
9. Run ipython3. The prompt is colored.
10. Examine the process tree in sysinternals Process Explorer. Starting with python.exe, I see this ancestry:

ConEmuC64.exe

• cmd.exe (the original shell) - ConEmuHk64.dll is present
  • cmd.exe (the start cmd shell) - ConEmuHk64.dll is present
    • cmd.exe (the final cmd shell) - ConEmuHk64.dll is not present
      • ipython.exe - ConEmuHk64.dll is not present
        • python.exe - ConEmuHk64.dll is not present

It's curious that the start cmd shell has the hook, but its child doesn't.

Scenario 2: using start -new_console:z cmd

1. Reset settings to default
2. Use the default {Shells} > {cmd} task.
3. Run start -new_console:z cmd in the shell
4. A new non-ConEmu console window appears with a cmd.exe shell in it.
5. The cmd.exe prompt's escape sequences are visible, indicating the hook DLL is not present.
6. In the new shell, the ConEmuANSI variable is ON, and ConEmuHooks is Enabled.
7. Run ipython3. The REPL prints many escape sequences, probably because prompt_toolkit thinks ConEmu ANSI support is enabled.

Scenario 3: using winpty with ipython3

This is possibly winpty issue rprichard/winpty#102.

1. Reset settings to default
2. Install Cygwin 64bit and place the ConEmu Cygwin64 connector into C:\cygwin64\bin. Install winpty 0.4.2 from https://github.com/rprichard/winpty/releases/tag/0.4.2.
3. Add a task, C:\cygwin64\bin\conemu-cyg-64.exe.
4. Run winpty cmd
5. The ConEmuANSI variable is set to ON, and ConEmuHooks is Enabled.
6. Run ipython. As with scenario 2, the escape sequences are visible.
7. Examine the process tree in sysinternals Process Explorer. I see these important processes:

ConEmuC64.exe

• conemu-cyg-64.exe - ConEmuHk64.dll is present

winpty.exe - ConEmuHk64.dll is present

• winpty-agent.exe - ConEmuHk64.dll is not present
  • cmd.exe - ConEmuHk64.dll is not present
    • ipython.exe - ConEmuHk64.dll is not present
      • python.exe - ConEmuHk64.dll is not present

winpty.exe starts its winpty-agent.exe child process in a new, hidden console, which resembles scenarios 1 and 2. ConEmu does not install the hook DLL into the winpty-agent.exe child, but it also leaves ConEmuANSI set to ON, which then confuses ipython3. I could make winpty.exe clear ConEmuANSI, but:

1. ConEmu was responsible for setting the variable, and
2. ConEmu sometimes sets ConEmuAnsi back to OFF when a new console is spawned

I think ConEmu should set the variable to OFF in all three scenarios.

[rprichard]

This issue could be responsible for this VSCode issue (microsoft/vscode#22616). If I start VSCode from ConEmu (e.g. by running Code.exe), then:

• ConEmu does not hook the VSCode processes
• The ConEmuANSI variable is set to ON in the VSCode processes.
• When VSCode starts its integrated terminal, it uses winpty to create a hidden console (a non-ConEmu "RealConsole").
• winpty, unlike ConEmu, does not hook WriteConsole et al to add VT/100 color escape support.
• PHP-based components see that ConEmuANSI is set and write escapes, which are visible to the user.

A major caveat: I don't know if this scenario is the one causing the VSCode issue, and it might not be. It doesn't really matter, though, because I've reproduced it myself.

I'm probably going to end up filtering out ConEmuANSI in winpty, but I still think this is a ConEmu bug.

[maxnoe]

Any progress here? Still having the same problenm trying to lauch ipython in ConEmu using msys2