Crashing in [RLMMigration verifyPrimaryKeyUniqueness]

[mgc]

Hiya. I'm hitting a crash during migration with Realm 0.91.1.

Looks like a stack overflow in tightdb::StringIndex::distinct called from [RLMMigration verifyPrimaryKeyUniqueness]

Thread 1Queue : com.apple.main-thread (serial)
#0  0x000000019467a5b0 in _nano_malloc_check_clear ()
#1  0x0000000194678f6c in nano_malloc_scribble ()
#2  0x00000001946693e4 in malloc_zone_malloc ()
#3  0x000000019466da04 in malloc ()
#4  0x000000019365a2e0 in operator new(unsigned long) ()
#5  0x00000001003f4a54 in tightdb::StringIndex::distinct(tightdb::Column&) const ()
#6  0x00000001003f4aec in tightdb::StringIndex::distinct(tightdb::Column&) const ()
[ ... ]
#2899   0x00000001003f4aec in tightdb::StringIndex::distinct(tightdb::Column&) const ()
#2900   0x00000001003f4aec in tightdb::StringIndex::distinct(tightdb::Column&) const ()
#2901   0x00000001004366f4 in tightdb::Table::get_distinct_view(unsigned long) ()
#2902   0x00000001002abcf4 in -[RLMMigration verifyPrimaryKeyUniqueness] at Realm/RLMMigration.mm:115

Full stack: http://pastebin.com/fVPHKskX

At the moment this is only happening on device (iPhone 6, iOS 8.2) with a database containing several thousand items. This is a debug build with malloc scribbling enabled, and it crashes reliably on start every time. I believe it is migrating from a database created using Realm 0.86.

I'll try to find other ways to repro, and share if possible.

[tgoyne]

Do you have strings with 10KB+ common prefixes? The depth of the recursion in StringIndex::distinct appears to be proportional to the length of the longest common prefix.

[mgc]

Hmm, nope. The primary key is a string URL, and I don't see any abnormally long ones. The only weirdness is that a handful of rows seem to have an empty string URL. It is possible that I managed to insert rows with empty primary keys, but it seems unlikely.

Here is the actual DB: https://dl.dropboxusercontent.com/u/20329687/RealmMigrationCrash.zip

Has it been corrupted somehow?

[mgc]

Incidentally, the older Realm browser also crashes when I try to change the contents of the empty string rows. http://pastebin.com/6y5SQUmf

[tgoyne]

It appears that the rows that are showing up in the browser as having empty URLs have embedded NUL bytes, which is something our string indexing code currently breaks on (and apparently the browser has issues with too). If this is actually the problem, then the simplest workaround while we work on fixing the problem would be to enumerate the objects in your migration block and delete the ones which have invalid URLs.

[mgc]

Thanks. I tried deleting and setting a new value in the migration, but both crash.

[migration enumerateObjects:OOURLCacheRecord.className block:^(RLMObject *oldObject, RLMObject *newObject) {
NSString *url = oldObject[@"url"];
if (url == nil || url.length == 0) {
[[newObject realm] deleteObject:newObject];
}
}];

index_string.cpp:551: [realm-core-0.88.5] Assertion failed: pos != values.size()
0 OBO Alpha 0x000000010efafc4c _ZN7tightdb4util9terminateEPKcS2_lbxx + 684
1 OBO Alpha 0x000000010f074dc8 _ZN7tightdb11StringIndex8DoDeleteEmNS_10StringDataEm + 2632
2 OBO Alpha 0x000000010f074a18 _ZN7tightdb11StringIndex8DoDeleteEmNS_10StringDataEm + 1688
3 OBO Alpha 0x000000010f074a18 _ZN7tightdb11StringIndex8DoDeleteEmNS_10StringDataEm + 1688
4 OBO Alpha 0x000000010f074a18 _ZN7tightdb11StringIndex8DoDeleteEmNS_10StringDataEm + 1688
5 OBO Alpha 0x000000010f074a18 _ZN7tightdb11StringIndex8DoDeleteEmNS_10StringDataEm + 1688
6 OBO Alpha 0x000000010f046f5e _ZN7tightdb11StringIndex5eraseINS_10StringDataEEEvmb + 78
7 OBO Alpha 0x000000010f053590 _ZN7tightdb20AdaptiveStringColumn17do_move_last_overEmm + 208
8 OBO Alpha 0x000000010f0bf46d _ZN7tightdb5Table17do_move_last_overEmb + 109
9 OBO Alpha 0x000000010f0bf36d _ZN7tightdb5Table27remove_backlink_broken_rowsERKNSt3__16vectorINS_10ColumnBase12CascadeState3rowENS1_9allocatorIS5_EEEE + 461
10 OBO Alpha 0x000000010f0c5eb5 _ZN7tightdb5Table14move_last_overEm + 245
11 OBO Alpha 0x000000010eee559f RLMDeleteObjectFromRealm + 383
12 OBO Alpha 0x000000010ef2fe91 -[RLMRealm deleteObject:] + 49

[migration enumerateObjects:OOURLCacheRecord.className block:^(RLMObject *oldObject, RLMObject *newObject) {
NSString *url = oldObject[@"url"];
if (url == nil || url.length == 0) {
newObject[@"url"] = [NSString stringWithFormat:@"stupid-hack-fix-%ld", (long)i];
i++;
}
}];

index_string.cpp:551: [realm-core-0.88.5] Assertion failed: pos != values.size()
0 OBO Alpha 0x000000010d8f0bdc _ZN7tightdb4util9terminateEPKcS2_lbxx + 684
1 OBO Alpha 0x000000010d9b5d58 _ZN7tightdb11StringIndex8DoDeleteEmNS_10StringDataEm + 2632
2 OBO Alpha 0x000000010d9b59a8 _ZN7tightdb11StringIndex8DoDeleteEmNS_10StringDataEm + 1688
3 OBO Alpha 0x000000010d9b59a8 _ZN7tightdb11StringIndex8DoDeleteEmNS_10StringDataEm + 1688
4 OBO Alpha 0x000000010d9b59a8 _ZN7tightdb11StringIndex8DoDeleteEmNS_10StringDataEm + 1688
5 OBO Alpha 0x000000010d9b59a8 _ZN7tightdb11StringIndex8DoDeleteEmNS_10StringDataEm + 1688
6 OBO Alpha 0x000000010d987eee _ZN7tightdb11StringIndex5eraseINS_10StringDataEEEvmb + 78
7 OBO Alpha 0x000000010d993917 _ZN7tightdb20AdaptiveStringColumn3setEmNS_10StringDataE + 199
8 OBO Alpha 0x000000010d9998f0 _ZN7tightdb20AdaptiveStringColumn10set_stringEmNS_10StringDataE + 32
9 OBO Alpha 0x000000010da0abb5 _ZN7tightdb5Table10set_stringEmmNS_10StringDataE + 165
10 OBO Alpha 0x000000010d804d70 _ZN7tightdb8RowFuncsINS_5TableENS_8BasicRowIS1_EEE10set_stringEmNS_10StringDataE + 96
11 OBO Alpha 0x000000010d7fd2eb _ZL11RLMSetValueP13RLMObjectBasemP8NSString + 107
12 OBO Alpha 0x000000010d7fc9fd _Z13RLMDynamicSetP13RLMObjectBaseP11RLMPropertyP11objc_objectm + 941
13 OBO Alpha 0x000000010d7fc54e _Z22RLMDynamicValidatedSetP13RLMObjectBaseP8NSStringP11objc_object + 1550
14 OBO Alpha 0x000000010d817ce0 -[RLMObjectBase setObject:forKeyedSubscript:] + 112
15 OBO Alpha 0x000000010d8147a3 -[RLMObject setObject:forKeyedSubscript:] + 115

My app is still alpha so I don't care much about migrating existing users. Any tips on how to prevent this from happening to begin with? Can you give me an example of a URL with null bytes? As far as I can tell my code will not let anything other than valid NSURL objects into the database, which makes me think this is memory or file corruption.

[jpsim]

What you're checking for in your code isn't for null bytes, but rather if the string is null. (url == nil || url.length == 0)

Here's one way to check if an NSString contains a null bytes:

NSString *url = @"myurl";
NSData *data = [url dataUsingEncoding:NSUTF8StringEncoding];

UInt8 asciiNull[] = { 0x0 };
NSData *zeroData = [NSData dataWithBytes:asciiNull
                                  length:sizeof(asciiNull)];

NSRange range = [data rangeOfData:zeroData
                          options:kNilOptions
                            range:NSMakeRange(0u, [data length])];

if (range.location != NSNotFound) {
    NSLog(@"Not safe to add this to a Realm: %@", url);
}

[alazier]

Hi Matt. Sorry you are running into issues.

I can easily reproduce the crashes you mention using your database. There are some entries containing embedded nulls in your db, but any attempt to delete or change these entries cases a crash as you described. Looking the offending urls, each string seems to be 100k+ bytes of garbage data. It isn't clear if this is the data being inserted or if the db is being corrupted.

We already have someone working on a proper fix for the indexing issues which we believe is the root cause of your problems, but there is not currently an eta for a fix. In the meantime we will probably release a workaround which no longer automatically creates indexes for primary key properties. I made a pr for this here #1623 if you want to try it out. This wont fix existing databases but new databases/installs will no longer use an index by default which should hopefully avoid the issue for the time being.

[mgc]

Hi Ari, np, I'm enjoying Realm so far.

NSURL returns nil if you init with a string containing nulls, but it is possible that I had/have a heap corrupting bug. Will see if it happens again.

FWIW as a user I'd expect primary keys to be automatically indexed. I'd like to be able to catch an exception to detect a corrupt db, delete, then create a new one, but for now I'll just check for successive crashes. Maybe worth fuzz testing the way SQLite does?

[jpsim]

Closing this as this should no longer happen with new databases.