Seeing "Realm file decryption failed" errors

[JoshHrach]

Background
Our iOS app is live with over one hundred thousand users. Of those, we have an issue happening with approximately 80 users. We're unable to reproduce this locally, but this is what we're able to piece together from Crashlytics.

Details
We store sensitive customer data in an encrypted Realm. To accomplish this, we have extended Realm with a static function that returns our encrypted Realm, or throws an error if it fails.

For 99% of users, this works fine. For less than 1%, we're seeing the "Realm file decryption failed" error. From investigating, it seems that this error is caused by trying to access a Realm with either the wrong key, or by trying to decrypt an unencrypted Realm.

Right now, with how we're generating the key, I'm inclined to think that that is not the issue.

Code Sample
We have a Realm extension with the following:

static func encryptedRealm() throws -> Realm {
    // Fetches key from Keychain if one exists, otherwise creates new Data key, stores in Keychain, returns it here.
        let key: Data = getEncryptionKey() 
        
        // Create default config
        let config = Realm.Configuration(encryptionKey: key)
        
        // Return Realm for config
        do {
            let realm = try Realm(configuration: config)
            return realm
        } catch let error as NSError {
            throw error
        }
  }

The first place Realm is hit in our app is via our User Settings manager. Simple version:

class SettingsManager {
    /// Shared instance
    static let shared = SettingsManager()
    
    /// Realm instance
    var realm: Realm {
        do {
            return try Realm.encryptedRealm()
        } catch let error as NSError {
            // Some logging to Crashlytics happening here
            fatalError()
        }
    }

   // Other methods/properties here as convenience accesses to user data
}

Expected results
While we accounted for the fact that something could go wrong, I was expecting to see errors pertaining to file I/O, disk space, etc.

Actual results
For less than 1% of our users, we're seeing "Realm file decryption failed". As all hits to Realm are happening through the Realm.encryptedRealm() function, I don't know how a non-encrypted Realm could be created. So I'm at a loss as to what might be going on.

The Crashlytics trace has our realm property getting above, followed by:

libswiftCore.dylib
__hidden#23281_ line 134
specialized _assertionFailure(StaticString, String, file : StaticString, line : UInt, flags : UInt32) -> Never

Looking up errors for that has shown me potentially issues with Realm, background threads, and the need to use autoreleasepool. But the error I'm catching before the fatalError call seems to imply otherwise.

Any guidance would be welcome at this point. Thanks.

Version of Realm
2.10.2 (Unable to upgrade at this time)

Xcode version: 9.2

iOS/OSX version:
Bug has been reported on iOS 10 and 11

Dependency manager + version: CocoaPods 1.4.0

[Spenders]

Hi @JoshHrach,

Any news on this topic ? I have the same problem with an App I'm working on.
Same version 2.10.2 as yours, same error message.

Implementation has been made by a team with a singleton pattern which seems not to be the best way for realm. Is it also you're case?

Thanks,
Matthieu

[JoshHrach]

@Spenders No luck on our end. I ended up doing an early check of the Realm (within the AppDelegate) and am deleting the database if it hits this. It's cut down on crashes in our app, but it hasn't addressed the ultimate problem.

[corteggo]

We're also experiencing this issue as well with the latest available version of RealmSwift. It's quite annoying since encryption key is being generated properly and stored for later uses.

[nwainwright]

Hey @JoshHrach @Spenders @corteggo @bmunkholm, we're having this happen for a small number of our users (including me). It's very intermittent and we haven't been able to nail it down yet. Has anyone worked this out or is it fixed in a later version of RealmSwift?

[jguerinet]

I'm having the same issue as well (Realm v3.7.1, issue reported on iOS 11 and 12 so far). We never had an unencrypted Realm DB (It was encrypted from the start) so I don't think it's trying to decrypt an unencrypted DB. My best guess is that somehow something went wrong with the key? For now, I am checking and deleting the DB when the app starts (like @JoshHrach mentioned), but this is less than ideal as in some situations the DB containus user inputted data that it not backed anywhere.

Received error:

1 | \| | 00:30:00:176 (UTC) | \| | 20:30:00.175 ERROR RealmManager.initRealm():41 - Error opening Realm: Unable to open a realm at path '/var/mobile/Containers/Data/Application/CEBB28E1-957F-4812-922A-DC64991C90E7/Documents/default.realm': Realm file decryption failed.
-- | -- | -- | -- | --
2 | \| | 00:30:00:285 (UTC) | \| | 20:30:00.286 ERROR RealmManager.initRealm():42 - Error Domain=io.realm Code=2 "Unable to open a realm at path '/var/mobile/Containers/Data/Application/CEBB28E1-957F-4812-922A-DC64991C90E7/Documents/default.realm': Realm file decryption failed." UserInfo={Error Code=2, NSFilePath=/var/mobile/Containers/Data/Application/CEBB28E1-957F-4812-922A-DC64991C90E7/Documents/default.realm, Underlying=Realm file decryption failed, NSLocalizedDescription=Unable to open a realm at path '/var/mobile/Containers/Data/Application/CEBB28E1-957F-4812-922A-DC64991C90E7/Documents/default.realm': Realm file decryption failed.}

Initialization code:

let key = RealmManager.getKey() as Data
let objectTypes = [...]

Realm.Configuration.defaultConfiguration = Realm.Configuration(encryptionKey: key,
    schemaVersion: RealmManager.schemaVersion,
    migrationBlock: { migration, oldSchemaVersion in
        RealmManager.performMigration(migration, from: oldSchemaVersion, to: RealmManager.schemaVersion)
    }, 
    objectTypes: objectTypes)
        
do {
    // Create an instance of the Realm now to perform any necessary migrations
    _ = try Realm()
} catch {
    SwiftyBeaver.error("Error opening Realm: \(error.localizedDescription)")
    SwiftyBeaver.error("\(error)")
}

Key Getter Code (attempts to load a key from the keychain, if none found it generates one and then stores it back to the keychain):

private static func getKey() -> NSData {
    // Identifier for our keychain entry
    let keychainIdentifier = ...
    let keychainIdentifierData = keychainIdentifier.data(using: String.Encoding.utf8, allowLossyConversion: false)!
        
    // First check in the keychain for an existing key
    var query: [NSString: AnyObject] = [
        kSecClass: kSecClassKey,
        kSecAttrApplicationTag: keychainIdentifierData as AnyObject,
        kSecAttrKeySizeInBits: 512 as AnyObject,
        kSecReturnData: true as AnyObject
    ]
        
    var dataTypeRef: AnyObject?
    var status = withUnsafeMutablePointer(to: &dataTypeRef) { SecItemCopyMatching(query as CFDictionary, UnsafeMutablePointer($0)) }
    if status == errSecSuccess {
        return dataTypeRef as! NSData
    }
        
    // No pre-existing key, so generate a new one
    let keyData = NSMutableData(length: 64)!
    let result = SecRandomCopyBytes(kSecRandomDefault, 64, keyData.mutableBytes.bindMemory(to: UInt8.self, capacity: 64))
    assert(result == 0, "Failed to get random bytes")
        
    // Store the key in the keychain
    query = [
        kSecClass: kSecClassKey,
        kSecAttrApplicationTag: keychainIdentifierData as AnyObject,
        kSecAttrKeySizeInBits: 512 as AnyObject,
        kSecValueData: keyData
    ]
        
    status = SecItemAdd(query as CFDictionary, nil)
    assert(status == errSecSuccess, "Failed to insert the new key in the keychain")
        
    return keyData
}

Just like the others, this is happening only for a small amount of users, seemingly randomly. We have not been able to reproduce it ourselves. Maybe there's something going on with the keychain? Are you all storing your keys there? If not the only answer I see is that it's within Realm itself.

[nwainwright]

Hey @JoshHrach @Spenders @corteggo @bmunkholm @jguerinet I think the issue might be about waking from a suspended state and Realm not starting fast enough to reply to the request to open the database file. We turned off all iOS background processing in the XCode project settings and it solved our main issue. We also think we've found a situation when we call the keychain for credentials for an encrypted database and it's returning the wrong credentials (maybe older ones), so we're now in the process of seeing if we can get multiple credentials and then cycle through each one to ensure we find the one that's valid. I hope this is helpful to you!

[tgoyne]

One thing I notice about your getKey() function is that it assumes that the only two possible results are "successfully read key" and "key not found". Perhaps in some rare situations the key is in the keychain but not readable (due to something like the device being locked?), in which case your app will instead generate a new key and then fail to open the existing Realm.

[jguerinet]

@nwainwright Thanks for the update. I don't believe this is the case for me, as this is happening on the main thread upon opening the completely-terminated app. I also don't know how the wrong credentials could have affected this, as we have only ever used one. Regardless, thanks for the theories and good luck!

@tgoyne This was a case I had originally considered as well, except that in that situation this would crash:

status = SecItemAdd(query as CFDictionary, nil)
assert(status == errSecSuccess, "Failed to insert the new key in the keychain")

Since we're asking to add and not to update, if the key already existed in any capacity the result would be errSecDuplicateItem and crash the app in itself (I tested this). And if for some reason it came back as errSecSuccess, the original key would have been lost since it seemingly no longer exists, and there wouldn't be much I could anyway. I don't believe this to be the case as I don't see where the key would be deleted in the first place, and there are clearly other people facing the same bug. Thank you for your suggestion though!

[JoshHrach]

@nwainwright Thanks for the tip. I can check what we're doing with our app, though we did want to support background processing.

@tgoyne Your point about the keychain lines up with what I was thinking. I was trying to see if perhaps there was a bad key being used with our users. I was originally thinking that, somehow, a new key was generated, but I had no way to replicate it to know. But your post gives me something to work on. If it truly can't get it from the keychain, then I might need to do a catch for that.

[corteggo]

We just realized what was the problem and I thought it might be helpful for you to know what we did in case it helps to fix your issue as well.

Our scenario is the following one:

• We do have multiple environments (Debug, QA, Production)
• We wanted to use a specific encription key for Debug mode and a runtime random generated key for QA and Production environments.
• We are saving encription key to Keychain once it is generated the first time. We always recover it in later executions.
• We used KeychainSwift library to make things easier when interacting with Keychain (https://github.com/evgenyneu/keychain-swift).
• We do have Apple CarPlay support and device is automatically locked once the user connects his/her device to the infotainment system.

We were experiencing a random issue where the encription key failed to open the database. There was no usage pattern to reproduce it and it happened also in Production for some users.

What we did is:

• Ensuring we had different keychain keys for each of our environments (e.g. com.ourcompany.ourappname.databaseEncriptionKeyDebug, com.ourcompany.ourappname.databaseEncriptionKeyQa, com.ourcompany.ourappname.databaseEncriptionKeyProd). Some users were exchanging beta and production releases and a wrong key was being used.
• Ensuring the database encription key was being saved with permissions to access it while the device was locked (see Potential solution to the background access problem evgenyneu/keychain-swift#78 for more information). When they were connected to Apple CarPlay, Keychain item was not accessible at all if they didn't open the app in their phone screen first.
• Using barriers (GCD) to perform only a one read/write operation at a time.

After all of those changes, there were still a few users having problems with database access. We realized there was a situation where the user was using iCloud to make backups of their phone but wasn't using iCloud Keychain. He/she changed or made a factory reset to his/her phone and restored the iCloud backup. Once the app was being started, Keychain returned nil for com.ourcompany.ourappname.databaseEncriptionKeyProd key and database failed to open (obviously). We added a specific control for this situation and created an empty database with a brand new encription key saved to the new Keychain.

Hope it helps.

[cucamacuca]

So how you handle a new phone with iTunes backup and no iCloud Keychain turned on? New database?

[corteggo]

@cucamacuca Unfortunately the answer is "yes" at this point. We're looking for other solutions for the future like having a backup of encription keys remotely that can be downloaded using a second-factor authentication or something like that but there is no decision yet.

[nunojfg]

Hi so the problem I have experienced is related to the source code that Realm provides for the generation of the key:

private static func getKey() -> NSData {
    // Identifier for our keychain entry
    let keychainIdentifier = ...
    let keychainIdentifierData = keychainIdentifier.data(using: String.Encoding.utf8, allowLossyConversion: false)!
        
    // First check in the keychain for an existing key
    var query: [NSString: AnyObject] = [
        kSecClass: kSecClassKey,
        kSecAttrApplicationTag: keychainIdentifierData as AnyObject,
        kSecAttrKeySizeInBits: 512 as AnyObject,
        kSecReturnData: true as AnyObject
    ]
        
    var dataTypeRef: AnyObject?
    var status = withUnsafeMutablePointer(to: &dataTypeRef) { SecItemCopyMatching(query as CFDictionary, UnsafeMutablePointer($0)) }
    if status == errSecSuccess {
        return dataTypeRef as! NSData
    }
        
    // No pre-existing key, so generate a new one
    let keyData = NSMutableData(length: 64)!
    let result = SecRandomCopyBytes(kSecRandomDefault, 64, keyData.mutableBytes.bindMemory(to: UInt8.self, capacity: 64))
    assert(result == 0, "Failed to get random bytes")
        
    // Store the key in the keychain
    query = [
        kSecClass: kSecClassKey,
        kSecAttrApplicationTag: keychainIdentifierData as AnyObject,
        kSecAttrKeySizeInBits: 512 as AnyObject,
        kSecValueData: keyData
    ]
        
    status = SecItemAdd(query as CFDictionary, nil)
    assert(status == errSecSuccess, "Failed to insert the new key in the keychain")
        
    return keyData
}

The problem is that SecItemAdd can fail if the keyChain is inacessible for some reason (device pass locked) so if this happens the keyChain will not save the encryptionKey thus creating a Realm Database with an encryptionKey that the app no longer can recover from, so the solution for this is verifying the result of status like so:

if status != errSecSuccess {
     throw DatabaseError // you should throw an error or return nil so this can fail gracefully
}

[JoaoPinho]

I had this same issue in some users and i was able to understand why and how to fix it.
In my case i was using SwiftKeychainWrapper and in some rare cases the stored encryption key in keychain was replaced, maybe you guys have some similar.

Take a look at this suggestion that i made to KeychainWrapper and see if your issue is related too:
Storage improvement for unchanged keys

[varyP]

What the problem was:
The problem is not with Realm, but with the encryption key it is being passed.
As others have mentioned, the problem might be if you are saving the key in KeyChain & just before you initialize Realm (and fetch encrytion key from KeyChain to do so), the user locks up his phone & then Keychain doesn't allow you to read that key anymore & your fallback logic creates a new key which is obviously not same as the old key & boom Realm exception!

Solution:

1. Check what accessibility option does your keychain wrapper provide you, to reproduce run a debug build on your phone & lock(must have passcode) as soon as the app launches.
2. Should reproduce your issue
3. Check what error does your Keychain wrapper give you while trying to fetch the key.
   Most likely it is 'errSecInteractionNotAllowed', fix this by using accessible option as `afterFirstUnlock/kSecAttrAccessibleAfterFirstUnlock' while saving/updating the key into keychain.

PS : Old keys might not be accessible, so migrate to a new key perhaps.

Thankyou guys & specially @TimOliver(we discussed this at iOSConfSG) for pointing in the right direction. No more annoying crashes!

[TimOliver]

Yay! Glad I was able to help! 😁

[RealmBot]

➤ Lee Maguire commented:

Looks like this has been solved.