VM console not working in XO.

[bberndt]

OS Version: Rocky Linux 8.5
Node.js version: v14.19.1
Yarn version: 1.22.18

Server specs
amount of vCPUs and RAM on the machine where you attempt to install
Intel(R) Xeon(R) CPU E5620 @ 2.40GHz 4 core
16 GB ram

Issue
Not sure whats happening. Updated a week or so ago, (april 5 according to the log) and noticed today that most of the VMs will not display the console in Xen Orchestra. Updated again today to whatever version is in the master git repo for XO, and same problem. Reverted back to build 04913 5.89/ 5.94. and consoles now work again.

Installation logfile
Attach complete log as a file.
xo-install.log-202204051415.txt
xo-install.log-202204110948.txt
xo-install.log-202204110955.txt

[ronivay]

Hello,

What XCP-ng/XenServer you're running? My best guess would be this: vatesfr/xen-orchestra@c99120b

[bberndt]

We have a mix. Mostly XenServer 7.1, XCP 7.6, and a couple 8.2. Most of our hardware is limited to Xen or XCP 7.6 (raid card monitor software doesn't work in 8.2). And Sorry, I didn't notice what versions had the problem, but it does look like it -could- have been 7.1, and 7.6.

[ronivay]

Okay. If you can confirm it only happens with older versions, i'm pretty sure reason is above mentioned commit. If that's the case, not much that can be done in this repo unfortunately.

[bberndt]

So i set up another Rocky Linux VM, and then copied the EXISTING XenOrchestraInstallerUpdater script from my prod XO server. and installed the current XO. Looks like, indeed, anything but Xen XCP 8.2 doesn't show a console.
Dumb quesion: how do i git clone the latest XenOrchestraInstallerUpdater? I know im a bit behind.

[ronivay]

If you have recent enough version of the script, it’ll update itself during execution. Manually: just run git pull inside XenOrchestraInstallerUpdater directory to get newest version. Looking at logs you provided, your version already upgrades itself and is running latest.

[bberndt]

So far, not finding a common thread, between them showing up and not showing up. Happens in Firefox, and Chrome, Doesn't appear tob e related to Xen Guest Tools version, getting device drivers from MS or no, or version of XenServer.
As long as im on this old version of XO, it seems to work. Note, I need to retract the theory that its only older XenServer VMs having this problem; happens on 8.2 as well.

I do see these when i bring up a VM console that doesn't show, and DON'T see them when one does show.
Failed when connecting: Failed to connect to server ( (code: 1005)) util.js:44:54
Error util.js:44
_fail rfb.js:578
f rfb.js:233
onclose websock.js:305
open websock.js:307
_connect rfb.js:384
_updateConnectionState rfb.js:546
connect rfb.js:279
p react-novnc.js:106
componentDidMount react-novnc.js:111
React 12
listen EventListener.js:29
React 29

Tried changing state of a disconnected RFB object util.js:44:54
Error util.js:44
_updateConnectionState rfb.js:473
disconnect rfb.js:284
_clean react-novnc.js:66
p react-novnc.js:72
(Async: setTimeout handler)
_onUpdateState react-novnc.js:44
_updateConnectionState rfb.js:519
_fail rfb.js:588
f rfb.js:233
onclose websock.js:305
open websock.js:307
_connect rfb.js:384
_updateConnectionState rfb.js:546
connect rfb.js:279
p react-novnc.js:106
componentDidMount react-novnc.js:111
React 12
listen EventListener.js:29
React 29
trapBubbledEvent
listenTo
z
_updateDOMProperties
mountComponent
mountComponent
mountChildren
_createInitialChildren
mountComponent
mountComponent
mountChildren
_createInitialChildren
mountComponent
mountComponent
performInitialMount
mountComponent
mountComponent
performInitialMount
mountComponent
mountComponent
performInitialMount
mountComponent
mountComponent
performInitialMount
mountComponent
mountComponent
performInitialMount
mountComponent
mountComponent

[ronivay]

Yeah, like i mentioned before. There's a commit in XO repo that changed the console behaviour from direct TCP connection to websocket, which is most likely the reason why it doesn't work with older XCP-ng/XenServer.

You can verify this by building your XO with previous commit from the one that introduced the changes, like this:

Update your xo-install.cfg file and change line BRANCH="master" -> BRANCH="b9ff3db9b" and then run update with xo-install.sh. When update is done, check console functionality. If it works, you can then update to that commit which introduced changes by changing BRANCH="c99120b" and run update again. If console breaks at this point, you know for sure it's because of changes in commit c99120b. It is then up to XO developers if they can make some workaround for this. If not, you need to upgrade your hosts or run older version of XO.

E: missed that you mentioned having this issue with XCP-ng 8.2 as well. Well test the steps above anyway to pinpoint cause.

[bberndt]

This went exactly to your plan. It broke on the second commit. I double checked, and I think I might have mistaken that it was also broke for XCP 8.2. So anything older doesn't bring up the console. We could physically move to XCP 8.2, but the RAID monitor software doesn't work. (LSI/ 3Ware 9750's) I can't recall the exact problem, but the executable for the web-admin doesn't run.

[ronivay]

Nice. As this went forward in XCP-ng forum, linking the issue here where the compatibility will be fixed: vatesfr/xen-orchestra#6187

Keeping this one open as well until solved.

[github-actions]

This issue has been open for 14 days without activity. It will be closed in 5 days if not updated

[ronivay]

Fix was merged but it had some issues. New PR waiting to be merged. Issue still open in XO repository.

[ronivay]

Fixed merged and issue closed in XO repo. Confirmed to work.