No previews, tasks run forever

[info-sic]

I'm stuck :(
Website shows no previews, not even for plain .txt and tasks run forever. If I upload the same file to the public instance, a result shows up in seconds.
warning.log doesn't show anything specific, where can I investigate further???
I installed libreoffice/ppa as recommended,
22.04.1-Ubuntu, up2date
git describe v1.8.0-7-g11fdad2

[update]
A task with a .docx file finished relativly quick, but no preview. Here's the stdout:

`2024-02-08 16:46:02,492 preview ERROR:<Fault 1: '<class \'unoserver.converter.com.sun.star.lang.IllegalArgumentException\'>:Unsupported URL <file:///pandora/pandora/tasks/2024/02/3e3e98c9-7fdd-4dd8-a2cc-f68db7ee15ae/pandoratest.docx>: "type detection failed" at ./framework/source/loadenv/loadenv.cxx:189'>
Traceback (most recent call last):
  File "/pandora/pandora/pandora/workers/preview.py", line 15, in analyse
    task.file.convert()
  File "/pandora/pandora/pandora/file.py", line 276, in convert
    self.libreoffice_client.convert(inpath=str(self.path), outpath=f'{self.path}.pdf')
  File "/home/pandora/.cache/pypoetry/virtualenvs/pandora-bBUC7Dma-py3.10/lib/python3.10/site-packages/unoserver/client.py", line 87, in convert
    result = proxy.convert(
  File "/usr/lib/python3.10/xmlrpc/client.py", line 1122, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python3.10/xmlrpc/client.py", line 1464, in __request
    response = self.__transport.request(
  File "/usr/lib/python3.10/xmlrpc/client.py", line 1166, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib/python3.10/xmlrpc/client.py", line 1182, in single_request
    return self.parse_response(resp)
  File "/usr/lib/python3.10/xmlrpc/client.py", line 1354, in parse_response
    return u.close()
  File "/usr/lib/python3.10/xmlrpc/client.py", line 668, in close
    raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault 1: '<class \'unoserver.converter.com.sun.star.lang.IllegalArgumentException\'>:Unsupported URL <file:///pandora/pandora/tasks/2024/02/3e3e98c9-7fdd-4dd8-a2cc-f68db7ee15ae/pandoratest.docx>: "type detection failed" at ./framework/source/loadenv/loadenv.cxx:189'>
2024-02-08 16:46:02,494 preview WARNING:Unable to generate preview, this is suspicious: <Fault 1: '<class \'unoserver.converter.com.sun.star.lang.IllegalArgumentException\'>:Unsupported URL <file:///pandora/pandora/tasks/2024/02/3e3e98c9-7fdd-4dd8-a2cc-f68db7ee15ae/pandoratest.docx>: "type detection failed" at ./framework/source/loadenv/loadenv.cxx:189'>
2024-02-08 16:46:02,494 preview INFO:[3e3e98c9-7fdd-4dd8-a2cc-f68db7ee15ae] Runtime: 0.06s

`

Manu

[Rafiot]

I have no idea, the libreoffice errors are always a bit obscure. Just in case try re-running poetry install, maybe? If it is an issue with the python library that is somehow incompatible with the system-wide install of libreoffice?

Try running unoconv (from within the virtualenv): https://github.com/unoconv/unoserver
and see if there is any difference?

[info-sic]

Nope :(
:poetry install
Installing dependencies from lock file
No dependencies to install or update
Installing the current project: pandora (1.8.0) Nothing gets installed. Seems we're up to date.

:~/.cache/pypoetry/virtualenvs/pandora-bBUC7Dma-py3.10/bin$ ./unoconvert /pandora/pandora/tasks/2024/02/3e3e98c9-7fdd-4dd8-a2cc-f68db7ee15ae/pandoratest.docx /pandora/pandora/tasks/2024/02/3e3e98c9-7fdd-4dd8-a2cc-f68db7ee15ae/pandoratest.pdf

File "/usr/lib/python3.10/xmlrpc/client.py", line 668, in close raise Fault(**self._stack[0]) xmlrpc.client.Fault: <Fault 1: '<class \'unoserver.converter.com.sun.star.lang.IllegalArgumentException\'>:Unsupported URL <file:///pandora/pandora/tasks/2024/02/3e3e98c9-7fdd-4dd8-a2cc-f68db7ee15ae/pandoratest.docx>: "type detection failed" at ./framework/source/loadenv/loadenv.cxx:189'>

For some obscure reasons it seems that xmlrpc in client fails to handle the URI (the file does exist!)
Might a wrong pythonversion for the the virtual pandora env be the problem? How would I fix that?
head unoconvert #!/home/pandora/.cache/pypoetry/virtualenvs/pandora-bBUC7Dma-py3.10/bin/python For me, this looks good?

Maybe it is an ACL Problem? How can I find out and fix?

Feeling dumb,
Manu

[Rafiot]

It's just very obscure. Just making sure, can you run poetry run unoserver_launcher ?

[info-sic]

while pandora is stopped (after a poetry run stop)
2024-02-12 12:29:02,356 UnoserverLauncher INFO:Initializing UnoserverLauncher
2024-02-12 12:29:02,386 unoserver INFO:Starting unoserver.
2024-02-12 12:29:02,386 unoserver INFO:Command: libreoffice --headless --invisible --nocrashreport --nodefault --nologo --nofirststartwizard --norestore -env:UserInstallation=file:///tmp/tmpm7ez4wrk --accept=socket,host=127.0.0.1,port=2002,tcpNoDelay=1;urp;StarOffice.ComponentContext
2024-02-12 12:29:02,387 UnoserverLauncher INFO:Launching UnoserverLauncher
2024-02-12 12:29:02,889 UnoserverLauncher INFO:Shutting down UnoserverLauncher

whilst pandora is running (after a poetry run start)
File "/usr/lib/python3.10/socketserver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
But this seems reasonable.

[Rafiot]

Hmmm no, that doesn't look good:

2024-02-12 12:29:02,387 UnoserverLauncher INFO:Launching UnoserverLauncher
2024-02-12 12:29:02,889 UnoserverLauncher INFO:Shutting down UnoserverLauncher

It starts and stops immediately.

And the other one says that there is something running on the port.

Try the following:

1. run stop
2. edit start.py and comment out the line with unoserver_launcher
3. run start
4. run unoserver_launcher

If it doesn't keeps running, there is a problem.

[info-sic]

@Rafiot I really appreciate your help but I don't want to waste your time.
At a normal start unoserver runs fine.
2024-02-12 14:50:45,857 unoserver INFO:Starting unoserver.
2024-02-12 14:50:45,857 unoserver INFO:Command: libreoffice --headless --invisible --nocrashreport --nodefault --nologo --nofirststartwizard --norestore -env:UserInstallation=file:///tmp/tmpvew57ed5 --accept=socket,host=127.0.0.1,port=2002,tcpNoDelay=1;urp;StarOffice.ComponentContext 2024-02-12 14:50:45,881 UnoserverLauncher INFO:Launching UnoserverLauncher

I can see something listening there (with lsof and curl)
unoconvert can be run manually but throws the same Unsupported URL exception in xlmrpc.client

Could you send me the versionnumbers of the public instance, so I can crosscheck? Everything works ok there.
I did a reinstall, trashing /pandora/pandora ~/.cache before, but to no avail.
So I guess, I either revert to Version 1.7 or do a complete reinstall of the whole machine..

[Rafiot]

For yarahq please make sure you're on the last commit in the repo, I fixed a few things.

But looking at the first message again: Unsupported URL <file:///pandora/pandora/tasks/2024/02/3e3e98c9-7fdd-4dd8-a2cc-f68db7ee15ae/pandoratest.docx>

That path is incorrect, is should be /home/pandora/pandora/tasks/2024/02/3e3e98c9-7fdd-4dd8-a2cc-f68db7ee15ae/pandoratest.docx. What is the content of the .env file in the root of the project?

[info-sic]

`PANDORA_HOME=/pandora/pandora

I didn't originaly install it in the users homedirectory. Might that be the culprit?

I fixed the yarahq error with an mkdir /pandora/pandora/yara_repos/yara-hq-full myself. That works now.

[Rafiot]

yeah, PANDORA_HOME=/pandora/pandora is definitely incorrect. It should be

echo PANDORA_HOME="`pwd`" >> .env

see: https://github.com/pandora-analysis/pandora?tab=readme-ov-file#pandora-installation

So in your case /home/pandora, I think.

And for yarahq, I'm pushing a fix: the directory is supposed to be created by the script, but it isn't. Thanks!

[info-sic]

Since the doc didn't mention that you have to install pandora in ~, I decided to install everything in /pandora.
So the From the directory you cloned Pandora to, run: resulted in /pandora/pandora in the .env file. Which seems to be correct. Might this cause the problems? I have no problem trashing the whole thing and start from scratch, but you should clarify that in the readme...

[Rafiot]

Oh, right, I see why I thought it was in /home/pandora, it's because the virtualenv is there: File "/home/pandora/.cache/pypoetry/virtualenvs/pandora-bBUC7Dma-py3.10/lib/python3.10/site-packages/unoserver/client.py", line 87, in convert

So my guess is a permission issue. There is no requirement for the project to be installed inside the home directory, but it makes it a lot simpler with the permissions. If you ran any command besides apt-get, they are most probably in a weird state. The pandora user must have r/w access to everything in the repo, and all the commands must be run with that user, never as root/sudo.

[info-sic]

Ok, fixed it.
git describe v1.8.0-15-g94507ac

Trashed the /pandora/redis /pandora/kvrocks /pandora/pandora directories.
Reinstalled to /home/pandora/redis /home/pandora/kvrocks /home/pandora/pandora
Didn't work neither with libroffice-nogui nor with libreoffice/ppa
INSTALLED libreoffice/jammy,now 4:7.6.4-0ubuntu0.22.04.1~lo1 amd64 THAT DID THE TRICK.
Phew