quokka exporting data references from non-exported segments (src/segment.cpp)

[arizvisa]

I'm encountering an issue when using Quokka v0.5.5 (a3de4c3) with quarkslab/qbindiff@69bed0a. Essentially the symptom is that Quokka bails when trying to access a reference to a segment that wasn't exported. The backtrace can be found at the end of this issue.

Specifically, the following code from Quokka iterates through all the segments in a database and collects them into an array.

quokka/src/Segment.cpp

Lines 79 to 100 in a3de4c3

	int ExportSegments(quokka::Quokka* proto) {
	Timer timer(absl::Now());
	QLOG_INFO << "Start to export segments";
	std::vector<Segment> segments;
	segments.reserve(get_segm_qty());
	segment_t* seg = get_first_seg();
	while (seg != nullptr) {
	if (is_visible_segm(seg) && not is_ephemeral_segm(seg->start_ea)) {
	segments.emplace_back(seg);
	}
	seg = get_next_seg(seg->start_ea);
	}
	WriteSegments(proto, segments);
	QLOG_INFO << absl::StrFormat("Segments exported (took %.2fs)",
	timer.ElapsedSeconds(absl::Now()));
	return eOk;
	}

Segments are only collected by this ExportSegments function if they are "visible" (is_visible_segm) and not "ephemeral" (is_ephemeral_segm). However, header segments (SFL_HEADER) that are loaded by the PE loader can be both "ephemeral" (SFL_LOADER|SFL_DEBUG) and referenced by an instruction (perhaps other formats too, as it would depend on the loader used to build the database).

So, if a database containing a segment with both the SFL_HEADER and SFL_LOADER flags is exported by Quokka and a user of that exported data attempts to enumerate instructions that reference an SFL_HEADER segment, the quokka.Program.get_segment method will raise an exception when trying to call quokka.Segment.in_segment. The exception it raises is uncaught by qbindiff which'll cause qbindiff to abort when it tries to gather references with qbindiff.loader.backend.InstructionBackendQuokka._cast_references.

I temporarily fixed the fragility of qbindiff by catching the exception "inside" this loop, but the proper fix would be to include is_header_segm as one of the checks in ExportSegments from quokka. To reproduce this, you might be able to build a database for a portable executable file which should load a segment for the PE header labeled as "HEADER". If you examine the segment_t.flags for that "HEADER" segment, it should have both the SFL_LOADER and SFL_HEADER flag set.

    def _cast_references(
        self, references: list[quokka.types.ReferenceTarget]
    ) -> list[ReferenceTarget]:
...
        ret_ref: list[ReferenceTarget] = []
        for ref in references:
            match ref:
                case quokka.data.Data():
...
                    try:
                        value = ref.value
                    except Exception:
                        logging.warning("Skipping missing reference for address {:#x}".format(ref.address), exc_info=True)
                    else:
                        ret_ref.append(Data(data_type, ref.address, value))

---

This following is the uncaught exception in qbindiff being raised by Quokka. It's also probably worth formatting the address that is being raised by quokka/program.py:325 as hexadecimal using "{address:#x}" too, so that it's not being emitted in decimal.

Traceback (most recent call last):                                                        
  File "/usr/lib/python3.12/site-packages/quokka/addresser.py", line 65, in file
    segment = self.program.get_segment(offset)                   
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                            
  File "/usr/lib/python3.12/site-packages/quokka/program.py", line 325, in get_segment
    raise KeyError(f"No segment has been found for address 0x{address}")                                                                                                            
KeyError: 'No segment has been found for address 0x268435968'                             
                                                                                          
The above exception was the direct cause of the following exception:

Traceback (most recent call last):                                                        
  File "/home/user/.local/bin/qbindiff", line 8, in <module>                                                                                                                        
    sys.exit(main())                                                                      
...
  File "/usr/lib/python3.12/site-packages/qbindiff/visitor.py", line 263, in visit_instruction
    callback(program, instruction, collector) 
  File "/usr/lib/python3.12/site-packages/qbindiff/features/artefact.py", line 72, in visit_instruction
    for ref_type, references in instruction.references.items():
                                ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/functools.py", line 995, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/qbindiff/loader/instruction.py", line 65, in references
    return self._backend.references
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/functools.py", line 995, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/qbindiff/loader/backend/quokka.py", line 277, in references
    ref[convert_ref_type(ref_type)] = self._cast_references(references)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/qbindiff/loader/backend/quokka.py", line 227, in _cast_references
    ret_ref.append(Data(data_type, ref.address, ref.value))
                                                ^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/quokka/data.py", line 99, in value
    address = self.program.addresser.file(self.address)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/quokka/addresser.py", line 67, in file
    raise quokka.NotInFileError("Unable to find the segment") from exc
quokka.exc.NotInFileError: Unable to find the segment

[RobinDavid]

Thanks @arizvisa for such a detailed issue! With all these information we shall be able to fix that.
If you can post your sample it would be nice otherwise we will manage to find one to reproduce.

[patacca]

This has been fixed in #32
If there are still problem related to this feel free to reopen the issue

[arizvisa]

Thx, @patacca.