Remote connection to target without JUL daemon does not handle error message gracefully (lttng 2.13)

[RyanFindley]

Recently, we updated our embedded target and apparently liblttng-ust is no longer being built with --enable-java-agent-jul.

When connecting to this target over ssh from TraceCompass, the following error occurs:

Error retrieving node configuration
    Command failed! Command: lttng --mi xml list -j

Actual output of this command when run on the target:

[15:56 ~]# lttng --mi xml list -j
Error: Unable to list jul events: Session daemon agent tracing is disabled
<?xml version="1.0" encoding="UTF-8"?>
<command xmlns="https://lttng.org/xml/ns/lttng-mi" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://lttng.org/xml/ns/lttng-mi https://lttng.org/xml/schemas/lttng-mi/4/lttng-mi-4.1.xsd" schemaVersion="4.1"><name>list</name><output/></command>

I suspect that the parser is choking on the bare error message emitted before the XML response content.

This can probably be solved by adding the --quiet arg to the request, to suppress the bare error output if the parser is not prepared to handle it:

[15:57 ~]# lttng --quiet --mi xml list -j        
<?xml version="1.0" encoding="UTF-8"?>
<command xmlns="https://lttng.org/xml/ns/lttng-mi" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://lttng.org/xml/ns/lttng-mi https://lttng.org/xml/schemas/lttng-mi/4/lttng-mi-4.1.xsd" schemaVersion="4.1"><name>list</name><output/></command>

or by redirecting stderr to /dev/null:

[16:05 ~]# lttng --mi xml list -j 2>/dev/null
<?xml version="1.0" encoding="UTF-8"?>
<command xmlns="https://lttng.org/xml/ns/lttng-mi" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://lttng.org/xml/ns/lttng-mi https://lttng.org/xml/schemas/lttng-mi/4/lttng-mi-4.1.xsd" schemaVersion="4.1"><name>list</name><output/></command>

To workaround in the short term: Is it possible to disable TraceCompass from querying for JUL domain session list? (via a properties or prefs file, command line arg, etc) Search didn't turn up any clues...

TraceCompass version: 10.2.0
lttng version on target:

[15:44 ~]# lttng --version
lttng (LTTng Trace Control) 2.13.9 - Nordicité

[RyanFindley]

For reference, the previous version of our target produced this output from the same command:

[17:45 ~]# lttng --version
lttng (LTTng Trace Control) 2.12.11 - (Ta) Meilleure
[17:46 ~]# lttng --mi xml list -j
<?xml version="1.0" encoding="UTF-8"?>
<command xmlns="https://lttng.org/xml/ns/lttng-mi" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://lttng.org/xml/ns/lttng-mi https://lttng.org/xml/schemas/lttng-mi/4/lttng-mi-4.0.xsd" schemaVersion="4.0">
<name>list</name>
<output><domains><domain><type>JUL</type><buffer_type>PER_PID</buffer_type><pids/></domain></domains></output></command>

I assume it's a difference in how we built, configured, or installed lttng on the target. (we have no java runtime on the target, so no need to log JUL events)

[bhufmann]

I think it fails because the command exits with a non zero value. Could you please confirm by printing the exit code (e.g. echo $?) and let us know?

[RyanFindley]

Good point, it returns exit code 1 with / without the --quiet flag

[14:51 ~]# lttng --mi xml list -j
Error: Unable to list jul events: Session daemon agent tracing is disabled
<?xml version="1.0" encoding="UTF-8"?>
<command xmlns="https://lttng.org/xml/ns/lttng-mi" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://lttng.org/xml/ns/lttng-mi https://lttng.org/xml/schemas/lttng-mi/4/lttng-mi-4.1.xsd" schemaVersion="4.1"><name>list</name><output/></command>
[14:51 ~]# echo $?
1
[14:51 ~]# lttng --quiet --mi xml list -j
<?xml version="1.0" encoding="UTF-8"?>
<command xmlns="https://lttng.org/xml/ns/lttng-mi" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://lttng.org/xml/ns/lttng-mi https://lttng.org/xml/schemas/lttng-mi/4/lttng-mi-4.1.xsd" schemaVersion="4.1"><name>list</name><output/></command>
[14:52 ~]# echo $?
1

[RyanFindley]

So the question remains: is it possible to configure TraceCompass to not query JUL sessions, or is this an implicit requirement of the design?

[bhufmann]

Thanks for confirming. No, it's not a implicit requirement. I think it's just a matter of ignoring any errors when checking for such UST provider (e.g. JUL, python etc). If it is not supported it should not prevent using other logger providers: [1] is the relevant code line. Instead of throwing an exception it could just return. Alternatively, we could ignore some of the errors like done for the default UST provider [2].

[1]

org.eclipse.tracecompass/lttng/org.eclipse.tracecompass.lttng2.control.ui/src/org/eclipse/tracecompass/internal/lttng2/control/ui/views/service/LTTngControlServiceMI.java

Line 1144 in 1801704

throw new ExecutionException(Messages.TraceControl_CommandError + command.toString());

[2]

org.eclipse.tracecompass/lttng/org.eclipse.tracecompass.lttng2.control.ui/src/org/eclipse/tracecompass/internal/lttng2/control/ui/views/service/LTTngControlServiceMI.java

Line 626 in 1801704

if (isError(result)) {

I'll try to submit a PR for this.

[RyanFindley]

Excellent, thanks!

[RyanFindley]

Some additional information:

We rolled back to previous versions of lttng and re-built the system. No other changes were made. It seems that this is a general incompatibility with lttng v2.13 specifically - it's now returning an error whereas previously in v2.12 it did not.

Turns out our system has never been built with --enable-java-agent-jul nor has it ever had any java runtime installed. The behavior of lttng in this situation is simply different between 2.12 and 2.13.

System built with lttng version we were using previously (no error)

lttng (LTTng Trace Control) 2.12.11 - (Ta) Meilleure
[19:03 ~]# lttng --mi xml list -j
<?xml version="1.0" encoding="UTF-8"?>
<command xmlns="https://lttng.org/xml/ns/lttng-mi" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://lttng.org/xml/ns/lttng-mi https://lttng.org/xml/schemas/lttng-mi/4/lttng-mi-4.0.xsd" schemaVersion="4.0"><name>list</name><output><domains><domain><type>JUL</type><buffer_type>PER_PID</buffer_type><pids/></domain></domains></output></command>
[19:03 ~]# echo $?
0

System built with lttng 2.12 (latest version) (also no error)

lttng (LTTng Trace Control) 2.12.16 - (Ta) Meilleure
[19:00 ~]# lttng --mi xml list -j
<?xml version="1.0" encoding="UTF-8"?>
<command xmlns="https://lttng.org/xml/ns/lttng-mi" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://lttng.org/xml/ns/lttng-mi https://lttng.org/xml/schemas/lttng-mi/4/lttng-mi-4.0.xsd" schemaVersion="4.0"><name>list</name><output><domains><domain><type>JUL</type><buffer_type>PER_PID</buffer_type><pids/></domain></domains></output></command>
[19:00 ~]# echo $?
0

I updated the title of the issue accordingly.

[bhufmann]

@RyanFindley The fix is integrated in the nightly build. Please let me know if it works in your environment. Thanks.
https://download.eclipse.org/tracecompass/master/rcp/

[RyanFindley]

@bhufmann Confirmed! Thanks for the quick turnaround. :)