Unable to export: Failed to parse attachment row: Integer -51 out of range at index 0

[aaronk6]

I’m getting an error when running imessage-exporter with an end date before 2012-07-23:

$ imessage-exporter -f html -c clone -e 2012-07-22                                                                                                                        ✘ 127
Building cache...
  [1/4] Caching chats...
  [2/4] Caching chatrooms...
  [3/4] Caching participants...
  [4/4] Caching tapbacks...
Cache built!
Unable to export: Failed to parse attachment row: Integer -210 out of range at index 0

$ imessage-exporter -f html -c clone -e 2012-01-01
Building cache...
  [1/4] Caching chats...
  [2/4] Caching chatrooms...
  [3/4] Caching participants...
  [4/4] Caching tapbacks...
Cache built!
Unable to export: Failed to parse attachment row: Integer -51 out of range at index 0

It works fine when I use 2012-07-23 as the end date. Apparently, all newer dates also work fine: I tried 2013-01-01, and also omitted the date to export all messages, which worked fine, too.

Maybe also worth nothing:

• I do have attachment in 2011 and 2012.
• It works fine using text export.

$ imessage-exporter -V
iMessage Exporter 2.2.0

[ReagentX]

What is the output of imessage-exporter -d? Is the iMessage database from an older device?

[aaronk6]

$ imessage-exporter -d
Building cache...
  [1/4] Caching chats...
  [2/4] Caching chatrooms...
  [3/4] Caching participants...
  [4/4] Caching tapbacks...
Cache built!

iMessage Database Diagnostics

Message diagnostic data:
    Total messages: 256879
    Messages not associated with a chat: 299
    Messages belonging to more than one chat: 16
Attachment diagnostic data:
    Total attachments: 25121
        Data referenced in table: 35.47 GB
        Data present on disk: 54.80 GB
    Missing files: 438 (2%)
        No path provided: 229
        No file located: 209
Global diagnostic data:
    Total database size: 377.74 MB
    Duplicated contacts: 146
    Duplicated chats: 3

Environment Diagnostics

Detected converters:
    Image converter: sips
    Audio converter: afconvert
    Video converter: ffmpeg
Done!

[ReagentX]

Interesting, so this only happens when you use an end date filter with that specific date? Can you try and replicate the crash after running export RUST_BACKTRACE=1? The crash is happening somewhere inside of the attachment serializer, but it's hard to say where.

The only thing we parse as unsigned is this field:

imessage-exporter/imessage-database/src/tables/attachment.rs

Line 74 in e7d5ba0

pub total_bytes: u64,

I'm not sure how an attachment might get a negative size, so I'd like to know more about what happens when we get the crash. I don't think we serialize anything differently when querying with a filter, so the fact that it works without -e is surprising to me.

[aaronk6]

Interesting, so this only happens when you use an end date filter with that specific date?

It also happens with earlier dates, i.e. 2011-12-31, 2011-10-01, or 2012-06-01. The integer value is different for each of those dates (but always negative).

Can you try and replicate the crash after running export RUST_BACKTRACE=1?

I just did this, but it won’t give me additional output (tried the brew version and the binary from the releases section).

[ReagentX]

No problem, you'd probably have to git clone the repo and built it yourself to see any useful trace. I am investigating this, but if you are able to generate a trace I would be very appreciative.

The fact that it says "at index 0" suggests to me that it might be failing to parse the ROWID (but also, these access the field by name, not index, so maybe not):

imessage-exporter/imessage-database/src/tables/attachment.rs

Line 87 in e7d5ba0

rowid: row.get("rowid")?,

But that field is INTEGER PRIMARY KEY AUTOINCREMENT, so I don't think it could be < 0.

[aaronk6]

Even when running from the repo, I don’t get the stack trace:

$ cargo run -- -f html -c clone -e 2012-06-01
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.06s
     Running `target/debug/imessage-exporter -f html -c clone -e 2012-06-01`
Building cache...
  [1/4] Caching chats...
  [2/4] Caching chatrooms...
  [3/4] Caching participants...
  [4/4] Caching tapbacks...
Cache built!
Unable to export: Failed to parse attachment row: Integer -174 out of range at index 0

Any idea why not?

[ReagentX]

The only place we apply query filters is here:

imessage-exporter/imessage-database/src/tables/attachment.rs

Lines 260 to 294 in e7d5ba0

	/// Get the total attachment bytes referenced in the table
	pub fn get_total_attachment_bytes(
	db: &Connection,
	context: &QueryContext,
	) -> Result<u64, TableError> {
	let mut bytes_query = if context.has_filters() {
	let mut statement = format!("SELECT SUM(total_bytes) FROM {ATTACHMENT} a");
	if context.has_filters() {
	statement.push_str(" WHERE ");
	if let Some(start) = context.start {
	statement.push_str(&format!(
	" a.created_date >= {}",
	start / TIMESTAMP_FACTOR
	));
	}
	if let Some(end) = context.end {
	if context.start.is_some() {
	statement.push_str(" AND ");
	}
	statement
	.push_str(&format!(" a.created_date <= {}", end / TIMESTAMP_FACTOR));
	}
	}
	db.prepare(&statement).map_err(TableError::Attachment)?
	} else {
	db.prepare(&format!("SELECT SUM(total_bytes) FROM {ATTACHMENT}"))
	.map_err(TableError::Attachment)?
	};
	bytes_query
	.query_row([], |r| r.get(0))
	.map_err(TableError::Attachment)
	}

All it does is sum the total_bytes column, which is INTEGER DEFAULT 0; I am not sure how that could be negative.

[ReagentX]

Did you run export RUST_BACKTRACE=1 before replicating the crash?

I think it may occur here, because we do access an index 0:

imessage-exporter/imessage-database/src/tables/attachment.rs

Lines 291 to 293 in e7d5ba0

	bytes_query
	.query_row([], |r| r.get(0))
	.map_err(TableError::Attachment)

but I am not sure why it would be negative–but if it was, that line would fail with the error you mentioned.

[aaronk6]

Did you run export RUST_BACKTRACE=1 before replicating the crash?

Yes (just realized it was missing in the output I pasted).

[ReagentX]

If you replace this:

imessage-exporter/imessage-exporter/src/main.rs

Lines 29 to 31 in e7d5ba0

	if let Err(why) = app.start() {
	eprintln!("Unable to export: {why}");
	}

with

app.start().unwrap();

it should avoid the exception handler and crash "normally".

[ReagentX]

The problem is probably that you have some old row in the attachment table where the total_bytes is a large negative value; since get_total_attachment_bytes() performs SELECT SUM(total_bytes) (given some filters), when you filter it before a specific date, all of the values add up to be negative. Once you give it enough rows that the result of that calculation is positive, it no longer crashes.

The field is currently u64, but given your database proves it can be negative, it should probably get parsed as an i64 instead.

[aaronk6]

it should avoid the exception handler and crash "normally".

It does:

thread 'main' panicked at imessage-exporter/src/main.rs:29:33:
called `Result::unwrap()` on an `Err` value: DatabaseError(Attachment(IntegralValueOutOfRange(0, -174)))
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: core::result::Result<T,E>::unwrap
             at /private/tmp/rust-20241019-8559-8pur06/rustc-1.82.0-src/library/core/src/result.rs:1102:23
   4: imessage_exporter::main
             at ./imessage-exporter/src/main.rs:29:21
   5: core::ops::function::FnOnce::call_once
             at /private/tmp/rust-20241019-8559-8pur06/rustc-1.82.0-src/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[ReagentX]

If you change

imessage-exporter/imessage-exporter/src/app/runtime.rs

Lines 333 to 334 in e7d5ba0

	Attachment::get_total_attachment_bytes(&self.db, &self.options.query_context)
	.map_err(RuntimeError::DatabaseError)?;

to:

                Attachment::get_total_attachment_bytes(&self.db, &self.options.query_context)
                    .unwrap_or(0);

Does that resolve the crash for you?

[ReagentX]

Better yet, can you check if this branch (diff) resolves the crash?

[aaronk6]

Yes, it does!

[ReagentX]

Thank you for the report and going through the trouble to test the changes!

[aaronk6]

No worries at all! This is such a great tool and I was happy to help! Thanks for all your work on this and also for the amazingly fast response to my bug report.