Clipboard corruption with clip.exe

[ivenhov]

ver
Microsoft Windows [Version 10.0.18363.592]

WSL distro

daniel@Daniel-Dell-XPS:~$ uname -a
Linux Daniel-Dell-XPS 4.4.0-18362-Microsoft #476-Microsoft Fri Nov 01 16:53:00 PST 2019 x86_64 x86_64 x86_64 GNU/Linux

daniel@Daniel-Dell-XPS:~$ cat /etc/*release (trimmed)
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"

• Test case:
  Attempting to copy string into clipboard using clip.exe.
  For certain text (possibly depends on the length) clipboard seems to have corrupt data
  Examples of the correct and corrupt clipboard (pasted content of the clipboard is on the second line, after the command)

This one fails

daniel@Daniel-Dell-XPS:~$ val="<redacted>"; echo -n $val | clip.exe
daniel@Daniel-Dell-XPS:~$ ㄵ㥢㤶㙥㐭ㅡⴰ㝥〰㔭㑣ⵦ㜹㌸づ㕢ぢぢ

With "x" in front of it, it works OK

daniel@Daniel-Dell-XPS:~$ val="<redacted>"; echo -n $val | clip.exe
daniel@Daniel-Dell-XPS:~$ <redacted>

In command line window it looks like:

• What's wrong / what should be happening instead:
  Expecting text to be copied to clipboard regardless of the text length

Possibly related to:
#3716

Update:
Added explanation of the second line on the command line (based on comment from @therealkenc)

[therealkenc]

Took a bit to figure out what "fail" meant. You are pasting after the clip. Okay. Confirmed here. Don't know whether it is the same as #3716, which celebrated a birthday last month. Probably.

[therealkenc]

One liner that is a little more succinct:

$ echo -n "51b969e6" | clip.exe; powershell.exe -noprofile -command Get-Clipboard
ㄵ㥢㤶㙥

The only real difference vs. #3716 is that over there we were working with the thesis a single character was the culprit. Appears the contents of the string (not the length) triggers.

$ echo -n "12345678" | clip.exe; powershell.exe -noprofile -command Get-Clipboard
12345678

These results are independently reproducible, not rando buffer corruption.

Now for the (very bad for us) news. I am pretty convinced the culprit is clip.exe proper. There is nothing in the WSL pipe marshaling that is going to deliberately peek at the string contents. If it was WSL interop, it'd fail but good, regardless of the string.

Winclip meanwhile is consistently correct.

Bad because, if this hypothesis is correct, the issue dies dreaded tag external.

[ivenhov]

Thanks for simplifying the test case and further investigation.

For some bizzare reason in regular CMD

echo 51b969e6623423423 | clip.exe

works as expected, ignoring appended \r\n
So it is on the borderline WSL and clip.exe

Regarding Winclip this seeps like the way to go. Is there any way to get hold of compiled version of it?

[therealkenc]

For some bizzare reason in regular CMD
echo 51b969e6623423423 | clip.exe

You're imagining semantic equivalencies because the strings echo and | "look the same". Those two scenarios are not semantically equivalent on too many levels to describe. And this is setting aside that the two aren't even the same syntactically, because of the implied CRLF (lack of -n) in the above CMD scenario. Your OP works fine if you drop the -n.

[Not that the CRLF is the point. It isn't. The two scenarios are semantically different with or without the CRLF.]

That said, I (double negative) can't prove there isn't a way for WSL interop to make Windows clip.exe happy (while simultaneously not breaking other things). Likewise there is no way to declare clip.exe's behavior broken, because its behavior can always be deemed correct by fiat.

Regarding Winclip this seeps like the way to go. Is there any way to get hold of compiled version of it?

I got it from here. No endorsement; I don't use it, I just found it in a quick google search yesterday. There are other third-party clip clones out there.

[ivenhov]

Thank you very much

After more fiddling I got what I wanted (including discarding newline)

$ echo -n 51b969e6623423423 | winclip.exe -c | powershell.exe -noprofile -command Get-Clipboard 51b969e6623423423 $ echo -n 51b969e662342342 | winclip.exe -c | powershell.exe -noprofile -command Get-Clipboard 51b969e662342342

But as you mentioned with clip.exe

$ echo -n 51b969e6623423423 | clip.exe | powershell.exe -noprofile -command Get-Clipboard 51b969e6623423423 $ echo -n 51b969e662342342 | clip.exe | powershell.exe -noprofile -command Get-Clipboard ㄵ㥢㤶㙥㈶㐳㌲㈴

So it seems it is somehow dependent on the length. Or the passed value is interpreted as some sort of encoding?
I don't know if that helps solving the issue.

[Gorthog]

Been using this tool for a bit and so far so good.

Install in Windows side:

dotnet tool install --global utf8clip

then in wsl side:
echo -n 51b969e6623423423 | utf8clip.exe; powershell.exe -noprofile -command Get-Clipboard

[ctrlcctrlv]

utf8clip works, and I thank @bluemarsh for providing it, but it's absolutely ridiculous that it's needed. Absolutely ridiculous.

Just add a /UTF8 flag. Or respect locale. Or something. Anything but what you're doing. What encoding is even default? 🤦

https://twitter.com/HW_BEAT_THAT/status/1270200908417167363

[BDisp]

If WSL is using this escape sequence "\x1b[?1h" probably is what is causing the clipboard corruption. I was using it before in the Linux to restore the terminal, but was causing the clipboard corruption. After I removed it the issue stopped. Only with WSL I still have this issue and perhaps is something like a escape sequence being causing all this.

Edit:
This may be a little confused but I mean using a ncurses application the clipboard on WSL doesn't work as expected. In a real Linux, even using VirtualBox, it works well. If anyone wants to check what I intend to demonstrate, please see this PR at gui-cs/Terminal.Gui#1319. Thanks.

[host1812]

Suddenly it start happening for me few weeks ago... I wonder if there is a working solution? I tried to install utf8clip but it is failing on Windows 11.
I found that you can just download binaries and add them to the path. You need .NET Core 3.0 be installed.

Works great! Thanks @Gorthog for suggestion.

[tianon]

This also reproduces similarly with Set-Clipboard via powershell.exe, no matter which UTF-8 encoding parameters I set in PowerShell before invoking it - I don't think Set-Clipboard is just a wrapper around clip.exe, right? (surely it's calling the .NET API for the clipboard or something?)

With WSLg, this is slightly better in that xclip inside WSL is able to read/write the clipboard successfully (xclip -selection clipboard -out, for example), but for some reason I haven't been able to figure out it unrelatedly hangs when you run xclip -selection clipboard -in over SSH (my use case involves setting clipboard contents remotely, and SSH directly to my WSL instance is the only way I've found so far to reliably do so without writing some custom bit of software).

I noticed with emoji, not any particular string (but was also able to reproduce with just 51b969e6 as noted above):

wsl-debian$ clip.exe <<<'🌸'; xclip -selection clipboard -out; echo
🌸

What's interesting IMO is that running the same from Git Bash on my host yields a different result, so unless I've horribly misunderstood, it seems like it's WSL related somehow, but I can't figure out exactly how:

git-bash$ clip.exe <<<'🌸'; debian run xclip -selection clipboard -out; echo
🌸

More fun examples, to show it's clipboard-specific:

wsl-debian$ /mnt/c/Program\ Files/Git/usr/bin/cat.exe <<<'🌸'
🌸
wsl-debian$ powershell.exe '$Input' <<<'🌸'
🌸
wsl-debian$ powershell.exe '$Input | Set-Clipboard' <<<'🌸'; xclip -selection clipboard -out; echo
🌸

Edit: very interestingly, it seems that the clipboard functionality in NeoVim is unaffected (even over SSH), so I guess I just need to stop using xclip or clip.exe, rely on WSLg / DISPLAY=:0 + NeoVim and let this go 😅

Edit 2: winning ticket:

$ ssh wsl 'DISPLAY=:0 nvim -Es +%\!xclip\ -out\ -selection\ clipboard\ -in' <<<'🙈'
$ ssh wsl DISPLAY=:0 xclip -selection clipboard -out
🙈

(no idea why nvim running xclip is fine but me running xclip directly hangs 🤷 I dug into the nvim implementation for clipboard handling and it just shells out, so I'm at a loss but happy to have a workaround that doesn't involve installing more tools)

[magiblot]

I was able to circumvent this issue by appending a null character at the end.

# Bug: clipboard returns 'ㄵ㥢㤶㙥㐭ㅡⴰ㝥〰㔭㑣ⵦ㜹㌸づ㕢ぢぢ'
echo -n 51b969e6-4a10-e700-5c4f-9783e0b5b0b0 | clip.exe
# Fix: clipboard returns '51b969e6-4a10-e700-5c4f-9783e0b5b0b0'
printf "51b969e6-4a10-e700-5c4f-9783e0b5b0b0\x00" | clip.exe

[ferdinandyb]

#11047 There was a solution here, setting Beta: Use Unicode UTF-8 for global language support in Windows. Although xclip doesn't work for me at all ...

[jalal-o]

i found a solution via iconv

tree | iconv -f utf8 -t utf16le | clip.exe