"Bad Realm file header (#1)" error when opening Realm file

[mowens]

When attempting to read from a Realm DB file, I am occasionally getting a "Bad Realm file header (#1)" error. It appears that the DB files gets corrupted while the app is running in the background (we do perform writes in the background based on push events). The faulty code is as follows (with some omitted, but this likely won't be helpful at all

do {
   let realm = try Realm(configuration: config)
} catch let error as NSError {

}

Below is the corrupted realm file if that helps
corrupted.realm.zip

Realm version: 096.3
iOS: 9.1
Configuration: Debug

[bigfish24]

@mowens can you provide some more details about what writes are occurring in the background?

[jpsim]

@mowens is this Realm file supposed to be encrypted? We just identified an issue which may have caused that assertion to be hit for encrypted files under very specific scenarios (when a file is resized on open).

Also, if you can provide more details about what writes are occurring in the background, that could help us identify what's happening.

[mowens]

@bigfish24 The writes that occur in the background are pretty minimal. We basically update our cache in response to push notifications to sync resources between devices or from location changes. So basically a write in the background would be something like:

Option 1

1. Make HTTP request
2. Write resource to disk (disk)

Option 2

1. Publish resource over a socket
2. Write resource to disk (cache)

Nothing really fancy going on.

@jpsim The realm file is not configured for encryption

[jpsim]

@mowens your transaction may seem minimal and unfancy, but you may be attempting a combination of operations in an order which has slipped through our unit tests so far! So we'd appreciate if you could share this code, especially if there's a chance that we can trigger the assertion on our end, which would drastically improve the odds of us finding what's wrong here.

If you'd rather share that code privately, you can email [email protected] referencing this GitHub issue.

And thanks for letting us know that the file isn't encrypted.

[mowens]

@jpsim

There are 2 possible places where this can be happening on our code from a write:

1. DB compact on start up
2. Write in background

Here is our code that we run when the app launches to compress our DB file size:

public func compactPersistanceWithConfiguration(config: Configuration, deleteIfCorrupted: Bool = false) {
    dispatch_async(config.writeQueue) {
        autoreleasepool {

            do {

                // force migrate realm
                if let error = migrateRealm(config.createRealmConfiguration()) {
                    Log.error {
                        Log.event("PersistCompactFailed")
                            .message("Failed to migrate realm")
                            .field("path", value: config.path)
                            .error(error)
                    }
                }

                let realm = try Realm(configuration: config.createRealmConfiguration())

                // create random path to copy compact realm to
                let compactPath = NSTemporaryDirectory().stringByAppendingFormat("%0f.realm",
                    NSDate().timeIntervalSince1970 * 1000)

                // copy a compact version of the DB
                try realm.writeCopyToPath(compactPath, encryptionKey: nil)


                var url: NSURL?
                try NSFileManager.defaultManager().replaceItemAtURL(NSURL(string: "file:///\(config.path)")!,
                    withItemAtURL: NSURL(string: "file:///\(compactPath)")!,
                    backupItemName: nil,
                    options: .UsingNewMetadataOnly,
                    resultingItemURL: &url
                )
            } catch let error as NSError {
                Log.error {
                    Log.event("PersistCompactFailed")
                        .field("path", value: config.path)
                        .error(error)
                }
                if deleteIfCorrupted {
                    do {
                        try NSFileManager.defaultManager().removeItemAtPath(config.path)
                    } catch let error as NSError {
                        Log.error {
                            Log.event("PersistCompactCorrupted")
                                .message("Failed to remove corrupted realm file")
                                .field("path", value: config.path)
                                .error(error)
                        }
                    }
                }
            }
        }
    }
}

Our Configuration class is just a object that contains information about which write and callback queues should be used when accessing Realm and the Realm.Configration that we create for each store. We abstract our realm access behind a PersistentStore class. The Realm.Configuration we create is versioned w/ a migration block and specifies which Object types should be managed by the realm instance. (if need you to see this code, let me know)

Our Realm abstraction (e.g. the PersistentStore is a generic class that will manage a single object type. We encode/decode the objects to our own entity models so we do not have to have our resources extend the Realm instance. Here is a basic example:

class Foo: Resource {
    var id: String

    init(id: String) {
       self.id = id
    }
}

class FooPersistent: Object, PersistentType {
   dynamic var id: String = ""

   class func fromModel(model: AnyObject) -> FooPersistent {
       let object = FooPersistent()
       if let model = model as? Foo {
            object.id = model.id
       } 
       return object
   } 

   func toModel() -> AnyObject {
         return Foo(id: id)
    }
}

Then each PersistentStore will manage a pair of these:

let fooStore = PersistentStore<Foo, FooPersistent>(config: configuration.createRealmConfiguration())

Now we can store instances of Foo using the fooStore and the PersistentStore will handle the encode/decode of to the actual Realm Object that is stored. Here is the code for performing a write using the PersistentStore:

    public func add(object: T, closure: ((T) -> Void)?) {
        dispatch_async(config.writeQueue) {
            autoreleasepool {
                Log.verbose {
                    Log.event("Adding object to database")
                        .field("db", value: NSStringFromClass(T.self))
                        .field("path", value: self.config.path)
                        .field("size", value: self.realmSize())
                        .field("object", value: "\(object)")
                }

                let realm = self.createRealm()
                // convert from Foo to FooPersistent
                let realmObject = U.fromModel(object)

                do {
                    try realm?.write {
                        realm?.add(realmObject, update: true)
                    }

                    self.notify(.Added, model: object)

                    // we convert our objects so we can always invalidate the realm objects in memory
                    realm?.invalidate()

                } catch let error as NSError {
                    Log.error {
                        Log.event("PersistentFailure")
                            .message("Failed to write to realm db")
                            .field("db", value: NSStringFromClass(T.self))
                            .field("object", value: "\(object)")
                            .error(error)
                    }
                }

                // always on the main thread
                if let closure = closure {
                    self.invokeCallback {
                        closure(object)
                    }
                }
            }
        }
    }

The code for creating our realm instance:

    private func createRealm() -> Realm? {
        do {
            let realm = try Realm(configuration: config.createRealmConfiguration())
            return realm
        } catch let error as NSError {
            Log.error {
                Log.event("PersistRealm")
                    .message("Failed to allocate an instance of realm. Current DB action will not be performed!")
                    .field("db", value: NSStringFromClass(T.self))
                    .error(error)
            }
        }
        return nil
    }

So a write would be invoked like so:

let foo = Foo(id: "foobar")
fooStore.add(foo) { storedFoo in

}

Hope this helps!

[jpsim]

Thanks for sharing those details! I'll try to reproduce and get back to you.

[jpsim]

@mowens I'm afraid there's too much code that wasn't shared for me to stub in without making so many assumptions as to void the initial conditions that led to the corruption that you've reported to us.

Without having sufficient sample code from you for us to be able to build and run, it'll be impossible to make more progress on this because we won't have anything actionable. Is there any chance you could email your project privately to us at [email protected] so we can attempt to reproduce? We could sign a mutual NDA enforcing that we can only use your code privately and with the sole goal of improving the Realm framework, if that makes you feel more comfortable.

There's nothing that jumps out at me in the snippets you've shared as being able to lead to corruption, although I do wonder why all the wrappers around Realm db connections, writes and models are necessary. If you find yourself wrapping many of our APIs to provide a nicer interface, I'd love to hear more about your rationale to see if our API could be improved.

Thanks!

[mowens]

@jpsim I would have to check with my VP to see if he is willing to travel down the NDA road. We haven't seen this crash in a while as we added recover code that will detect this and rebuild everything from the network. I will continue to monitor and see if I can ever get a better crash dump of this.

As for why we are wrapping Realm API's with our own, this is purely for abstraction reasons. The idea is that we don't let the Realm API bleed out of our PersistentStore API this way if for any reason we needed to switch to a different DB we could just by refactoring the PersistentStoreStore implementation.

We are also using promises (PromiseKit) heavily so we wrap our code (which isn't illustrated in the above code samples), but with wrappers around Realm API's we can provide promise based writes which is very nice for caching purposes:

For example:

firstly {
    // make an HTTP GET at /foo/:id and decode HTTP body into a Foo instance
    httpClient.get("/foo/\(id)", Foo.self)
}.then { foo in
   // cache Foo instance in the store
   return store.add(foo)
}.then { foo in
   // do something with the Foo instance
}.error { error in
   // handle errors
}

So with promises it allows us to do some really nice chaining, but this is really an implementation detail of some decisions we have made :) I think the Realm API is pretty damn awesome 👍

[jpsim]

I understand that you may not be able to share your full code with us, but please also understand that without being able to reproduce this issue ourselves, it's very unlikely we'll be able to fix it. The other option would be for you to take the time to make a clean-room sample project that triggers the same error without any proprietary information and share that with us.

As for your rationale behind the wrappers, I certainly see the value in having promise helpers. Hiding Realm usage behind an agnostic wrapper, however, will only complicate your codebase without simplifying a potential future move to a different data layer in my opinion, but I understand the motivations. Thanks for sharing!

[jpsim]

@jpsim I would have to check with my VP to see if he is willing to travel down the NDA road. We haven't seen this crash in a while as we added recover code that will detect this and rebuild everything from the network. I will continue to monitor and see if I can ever get a better crash dump of this.

Hi @mowens, any news on this? We'd love to continue investigating what may be causing this corruption, but don't have enough to go on at the moment. Let me know if there's anything I can do to help, like talking to your VP for example.

[mowens]

Hey @jpsim I think we figured out what the issue was and it is completely irrelevant to Realm. What I believe to have happened was we had 2 threads running (one doing a write) and one doing something else and that something else crashed. Being that this was running the background at the time, next boot up would show that Realm had been corrupted (which makes sense).

Thanks for looking into this and sorry for taking you down a rabbit hole on this one.

[jpsim]

@mowens Realm has measures in place to prevent corruption from ever happening, even when write transactions are interrupted by the process crashing. It seems those safeguards failed in this case, which would still be considered a Realm bug.

[NguyenVV]

Hi there, I still faced this issue on Realm 2.2.0. Which action need to do with this issue.
Exception with message :

EXCEPTION: io.realm.exceptions.RealmFileException
LOCATION : SourceFile line -2 in SharedRealm.nativeGetSharedRealm()
TIME : 12/21/16, 7:32
MESSAGE : Unable to open a realm at path '/data/data/com.planday.ninetofiveapp/files/346b6fdd431da7ed53942b847aa7afea531fb9f7.realm': Bad Realm file header (#1). in /Users/cm/Realm/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_SharedRealm.cpp line 92

[jpsim]

@NguyenVV please file a new issue taking care to fill out the template.