secure cell x32-x64 compatibility fix #279

vixentael · 2018-01-16T13:55:38Z

The problem described in #220 appears to be SecureCell incompatibility between x32 and x64 platforms. SecureCell is working correctly on both platforms, however, ciphertext calculations are platform-dependent.

Where is the problem
Message length and context length are used in encryption and decryption in every mode of SecureCell, and they are defined as size_t types (secure_cell.c#L47, secure_cell.c#L29 and below).

During encryption-decryption process sizeof(message_length) is calculated (sym_enc_message.c#L184) which is equal to sizeof(size_t).

Result of sizeof(size_t) is platform dependent: 4 bytes on x32 platforms, 8 bytes on x64 platforms.

This value is used inside soter_kdf function for calculating hmac.

Therefore, resulting ciphertext uses different amount of bytes on different platforms.

Fix
Real fix is to use platform-independent types, like uint32_t, instead of size_t. uint32_t variables are already used among many data structures themis_auth_sym_message_hdr_type and themis_sym_message_hdr_type

Changing sizeof(message_length) to sizeof(uint32_t) results in using 4 bytes on every platform, making encryption-decryption compatible among x32 and x64.

Compatibility fix
Unfortunately using 4 bytes for x64 leads to unability to decrypt messages encrypted by themis 0.9.6 x64.

Therefore on x64 platform if decrypting fails first time, we try to re-decrypt message using 8 bytes as context length for the second time.

Testing
I made manual integration testing among x32 and x64 Ubuntu 17.04 for themis 0.9.6 and 0.9.7. I encrypt data in every Secure Cell mode, write it to the files, and encrypt on another platform.

Results

fully compatible (can be encrypted and decrypted both ways)
0.9.7 х64 <-> 0.9.7 х32
0.9.7 х64 <-> 0.9.6 х32
0.9.7 х32 <-> 0.9.6 х32
almost fully compatible
0.9.7 x64 <-> 0.9.6 x64 (messages encrypted with context imprint mode can't be decrypted on 0.9.7)
not compatible
0.9.6 x64 can't decrypt messages encrypted by 0.9.7 x64/x32. Users should update to 0.9.7.
0.9.7 x32 can't decrypt messages encrypted by 0.9.6 x64. Users should update to 0.9.7.

ignatk

Generally, I would add a COMPAT compile time switch (enabled on compile time by adding -DCOMPAT and replace hardcoded THEMIS_097_SECURE_CELL_X64_COMPATIBILITY_FIX) with something like

#ifdef COMPAT
/* below will likely be evaluated compile time, so for other cases it will be a no-op */
if (sizeof(size_t) == sizeof(uint64_t) {
  /* the compat code */
}
#endif

ignatk · 2018-01-16T14:43:43Z

src/themis/secure_cell.h

+// since 0.9.7 Secure Cell context size is equal sizeof(uint32_t) for both platforms,
+// causing compatibility issues on x64
+// this define makes SecureCell
+#define THEMIS_097_SECURE_CELL_X64_COMPATIBILITY_FIX


Why do we hardcode this? I would make this a compile time flag instead.

ignatk · 2018-01-16T14:45:38Z

src/soter/soter_error.h

@@ -27,7 +27,7 @@
 #include <stdint.h>

 /** @brief return type */
-typedef int soter_status_t;
+typedef int32_t soter_status_t;


This is just an error code. If we don't depend on its size anywhere, better leave it as int, that is platform natural type.

to be consistent with
https://github.com/cossacklabs/themis/blob/master/src/themis/themis_error.h#L31

what if we decide to use large numbers as error codes in future? I think platform independent types are better

The point here not the size, but that the types are "native" to the platform. Therefore the resulting code is better optimised. So, for error codes, when they are not used in any serialisation structures (and they shouldn't be), this is the preferred approach.

wouldn't we have platform inconsistency using int, if someone decided to use 0x7FFFFFFFFFFFFFFF as error code?

Most projects do not have any issues with that - including Linux userspace API :)

ignatk · 2018-01-16T14:48:27Z

src/themis/sym_enc_message.c

@@ -181,7 +181,7 @@ themis_status_t themis_auth_sym_encrypt_message(const uint8_t* key,
 						 size_t* encrypted_message_length){
  uint8_t key_[THEMIS_AUTH_SYM_KEY_LENGTH/8];
  THEMIS_CHECK_PARAM(message!=NULL && message_length!=0);
-  THEMIS_STATUS_CHECK(themis_sym_kdf(key,key_length, THEMIS_SYM_KDF_KEY_LABEL, (uint8_t*)(&message_length), sizeof(message_length), in_context, in_context_length, key_, sizeof(key_)),THEMIS_SUCCESS);
+  THEMIS_STATUS_CHECK(themis_sym_kdf(key,key_length, THEMIS_SYM_KDF_KEY_LABEL, (uint8_t*)(&message_length), sizeof(uint32_t), in_context, in_context_length, key_, sizeof(key_)),THEMIS_SUCCESS);


It is not a good practice to do sizeof(type) at this point. Better declare message_length with the type we want and use sizeof(variable): if we want to update the message_length type in the future again, we would have to remember to update this line as well, otherwise we can get a buffer overflow.

If you use sizeof(var) approach - the code "adjusts itself"

agree with you

unfortunately, we can't easily declare message_length as uint32_t because it requires changing function signatures all over the project (changing params from size_t to uint32_t), which requires updating most of the wrappers. (pile of work, many mistakes :doge:)

from another perspective, current code already uses uint32_t cast in some places like
https://github.com/cossacklabs/themis/blob/master/src/themis/sym_enc_message.c#L166 so I would say that changing message_length type in future is not an easy task anyway

This just creates a potential landmine for the future (and maybe even a security incident). If it is possible, better change it now than suffer the consequences in the future.

I'm on your side totally.

However i'm trying to fix 3yrs old mistake of using sizeof(size_t) without need of re-writing half of Themis core and introducing more mistakes. Believe me, first thing I tried was changing a type of message_length, but I faced tons of changed code, including secure message, secure session, tests, wrappers, everywhere.

Can we define uint32_t somewhere in this file and use define instead of actual type? :)

The point is this is used to calculate buffer sizes, therefore enforces buffer safety. Our primary goal is secure code and this type of fix makes it worse in that regard. Even if it is much more work - it is justified IMO. Otherwise we might be breaking a "security promise" for Themis users.

sizeof(type) in buffer length calculation and using an actual value of a different type is just too typical for a potential security issue we should be making in a security oriented project.

Thank you for your comments! You're absolutely right in terms of security and 'best practices' perspective.

We will update Themis core to use uint32_t or even uint64_t for message calculations when we will have automated integration testing between platforms, wrappers and core versions. Until then we cannot guarantee that significant code changes won't break core functionality.

Unfortunately the code already has force casts like this

const size_t message_length ... hdr->message_length=(uint32_t)message_length;

I don't see how adding

size_t* encrypted_message_length ... sizeof((uint32_t)(encrypted_message_length))

is better than sizeof(uint32_t).

Taking into account limited resources and severity of the problem we're trying to solve, I believe that this temporary solution works before proper refactoring. We already working on integration test suit.

Can you please explain how using sizeof(uint32_t) decreases current security guarantees, except the point that we will forget to change this code during future refactoring.

Unfortunately the code already has force casts like this

That troubles me as well: casting up is fine, casting down usually creates issues (unless you're sure they are not)

Can you please explain how using sizeof(uint32_t) decreases current security guarantees

The point is in "forgetting to change" - for example you have code which serialises secure cell structure and you usually do it by iterating each variable usually, something like

uint8_t *p = malloc(hdr + len); memcpy(p, &hdr); p += sizeof(hdr); memcpy(p, &len); p += sizeof(len); ...

there are two potential problems here if you get incorrect size forgetting to update sizeof(len) with proper type:

if you update it in serialisation code, but not in the malloc code - it will allocate smaller buffer and you'll write data past the buffer boundary

even if you update it in both places, but actual length is 64 bit, but we just use sizeof(uint32_t) as a storage buffer (cast down) - this code will break on big-endian systems (which are still common nowadays), because it will write zeroes at best and garbage at worst

Now these type of errors are very easy to make, they are very hard to spot during code review, they will likely not be caught by automatic test suite.

Thank you, got it!

How about next plan:

merge fix as is;

document these problems (down-casting and sizeof(uint32_t)) as places of potential risks (add "warning-note-for-future-us" inside code);

check code with static analyzers (afaik we are doing this manually time to time, not sure whether analyzers trigger warnings about down-casting);

finish automated between-platforms integration tests (we are working on this right now, any ideas how to test on big-endian systems?);

create refactoring plan, paying attention on risky places, change size_t variable to uint32_t (uint64_t? whatever we decide is better);

refactor core and wrappers, keeping an eye on compatibility.

I believe that your suggestion #270 to update C code style will help us a lot!

What do you think?

Probably adding a TODO in the comment above these "tricky lines" will suffice for now.

ignatk · 2018-01-16T14:50:03Z

src/themis/sym_enc_message.c

+  if (decryption_result != THEMIS_SUCCESS && decryption_result != THEMIS_BUFFER_TOO_SMALL) {
+
+    // we are on x64, should sizeof(uin64_t) for backwards compatibility with themis 0.9.6 x64
+    if (sizeof(size_t) == sizeof(uint64_t)) {


This is resolved on compile time, so can be converted to MACROS and added to THEMIS_097_SECURE_CELL_X64_COMPATIBILITY_FIX above

you mean, the MACROS code may look smth like

#ifdef COMPAT #define THEMIS_SCELL_COMPAT(func){ \ /* below will likely be evaluated compile time, so for other cases it will be a no-op */ if (sizeof(size_t) == sizeof(uint64_t) { \ // compatibility code {func;} \ } \ } #else #define THEMIS_SCELL_COMPAT(func) {} #endif }

usage code

if (decryption_result != THEMIS_SUCCESS && decryption_result != THEMIS_BUFFER_TOO_SMALL) { THEMIS_SCELL_COMPAT({ THEMIS_STATUS_CHECK(themis_sym_kdf(key,key_length, THEMIS_SYM_KDF_KEY_LABEL, (uint8_t*)(&encrypted_message_length), sizeof(uint64_t), NULL, 0, key_, sizeof(key_)),THEMIS_SUCCESS); decryption_result = themis_sym_decrypt_message_u_(key_,sizeof(key_),context,context_length,encrypted_message,encrypted_message_length,message,message_length); }); }

Well I don't think we actually need #define THEMIS_SCELL_COMPAT(func) at all, just

#ifdef COMPAT if (sizeof(size_t) == sizeof(uint64_t) { ... } #endif

the code will be empty if either COMPAT is not defined, or compiler evaluates sizeof(size_t) == sizeof(uint64_t) to false.

ignatk · 2018-01-16T14:50:20Z

src/themis/sym_enc_message.c

@@ -262,7 +277,7 @@ themis_status_t themis_sym_encrypt_message_u(const uint8_t* key,
 					     uint8_t* encrypted_message,
 					     size_t* encrypted_message_length){
  uint8_t key_[THEMIS_SYM_KEY_LENGTH/8];
-  THEMIS_STATUS_CHECK(themis_sym_kdf(key,key_length, THEMIS_SYM_KDF_KEY_LABEL, (uint8_t*)(&message_length), sizeof(message_length), NULL, 0, key_, sizeof(key_)),THEMIS_SUCCESS);  
+  THEMIS_STATUS_CHECK(themis_sym_kdf(key,key_length, THEMIS_SYM_KDF_KEY_LABEL, (uint8_t*)(&message_length), sizeof(uint32_t), NULL, 0, key_, sizeof(key_)),THEMIS_SUCCESS);


ignatk · 2018-01-16T14:50:35Z

src/themis/sym_enc_message.c

+  if (decryption_result != THEMIS_SUCCESS && decryption_result != THEMIS_BUFFER_TOO_SMALL) {
+
+    // we are on x64, should sizeof(uin64_t) for backwards compatibility with themis 0.9.6 x64
+    if (sizeof(size_t) == sizeof(uint64_t)) {


ditto adding to macros

mobrio · 2018-01-16T17:52:44Z

I entirely appreciate the desire for a safe and minimal intervention and the fine detail of that is not my competence.

However, my understanding (which may be entirely wrong) from #220 is that x32 primarily (if not solely) occurs for the Android emulator. Similarly I would expect x64 to be the majority case in server or indeed desktop contexts.

So some thoughts/questions:

Should we not then consider x32 to be the exception that is the focus for compatibility fixing and x64 the norm?

For example does constraining x64 platforms to use 4 bytes via sizeof(uint32_t) rather than 8 result in any performance loss?

Similarly if there is a case when our compatibility strategy is to "fail and retry with a different size" would it not be better for that performance hit to occur in a x32 emulator rather than x64 real world devices?

Also, as I recall it the C preprocessor knows all about stuff like SIZEOF_INT so perhaps there's the opportunity for natural adaption rather than some explicit -DCOMPAT switch?

vixentael · 2018-01-17T03:05:57Z

@mobrio, actually saying, x64 code should be considered as "broken" implementation, and x32 as "correct".

The encryption-decryption code uses uint32_t structures mostly, except that variables defined as size_t. On x32 platforms sizeof(size_t) equals sizeof(uint32_t) equals 4 bytes. All calculations are respective and correct. On x64 sizeof(size_t) == sizeof(uint64_t) == 8 bytes, making the code a mix of uint_32t structures and 8-bytes long buffers :(

Android emulator was just the trigger to find this issue. It affects any x32<->x64 platforms. We believe that most users are using x64 "broken" themis version. That's why adding "fail and retry with a different size" for x64 is important: we should allows users to decrypt their data, previously encrypted by "broken" code. The "double decryption" strategy should be enabled by default, allowing users to disable it manually.

Changing data types for uint64_t, removing explicit sizeof(uint32_t) and "double decryption" strategy, is our "next step" update. However, in my opinion, we should release quick fix, set up automated integration tests (who knows how many similar issues are still undetected), detect those issues, understand the problem landscape better, and perform proper refactoring only after that.

vixentael · 2018-01-17T03:07:45Z

In latest commit I've changed code to

#ifdef SCELL_COMPAT

  // we are on x64, should sizeof(uin64_t) for backwards compatibility with themis 0.9.6 x64
  if (sizeof(size_t) == sizeof(uint64_t)) {
    if (decryption_result != THEMIS_SUCCESS && decryption_result != THEMIS_BUFFER_TOO_SMALL) {
        THEMIS_STATUS_CHECK(themis_sym_kdf(key,key_length, THEMIS_SYM_KDF_KEY_LABEL, (uint8_t*)(&encrypted_message_length), sizeof(uint64_t), NULL, 0, key_, sizeof(key_)),THEMIS_SUCCESS);
        decryption_result = themis_sym_decrypt_message_u_(key_,sizeof(key_),context,context_length,encrypted_message,encrypted_message_length,message,message_length);
    }
  }
#endif

and enable -DSCELL_COMPAT by default in Makefile.

To disable compatibility fix, users should pass NO_SCELL_COMPAT parameter into make, like this

make NO_SCELL_COMPAT=y all install

vixentael · 2018-01-17T18:03:48Z

Added warning lines.
Created refactoring issue and describe plan there.
Waiting for tests to complete and merge this issue.

Thank you a lot for this discussion, it was very useful and strategic-minded!

vixentael added 3 commits January 16, 2018 01:32

fix for x32/x64 compatibility: make context platform-independent

eb7d82b

same fix for context imprint

8e52329

small scripts handy to encrypt & decrypt bunch of secure cell files

e2dcc0d

vixentael added the compatibility Backward and forward compatibility, platform interoperability issues, breaking changes label Jan 16, 2018

vixentael self-assigned this Jan 16, 2018

vixentael requested review from ignatk and Lagovas January 16, 2018 13:55

ignatk reviewed Jan 16, 2018

View reviewed changes

add SCELL_COMPAT flag to compiler options and makefile

183e3af

vixentael added 2 commits January 17, 2018 19:35

add todo to mark potential insecure places

14a0459

fix typo

a2cb8c7

vixentael merged commit dd758bd into master Jan 17, 2018

vixentael deleted the vixentael/new_x32_fix branch January 17, 2018 20:28

vixentael mentioned this pull request Jan 18, 2018

Can't decrypt messages between iOS and Android #220

Closed

ilammy mentioned this pull request Mar 26, 2020

Test and deprecate Themis 0.9.6 compatibility path #614

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

secure cell x32-x64 compatibility fix #279

secure cell x32-x64 compatibility fix #279

vixentael commented Jan 16, 2018 •

edited

Loading

ignatk left a comment •

edited

Loading

ignatk Jan 16, 2018

vixentael Jan 16, 2018

ignatk Jan 16, 2018

vixentael Jan 16, 2018

ignatk Jan 16, 2018

vixentael Jan 16, 2018

ignatk Jan 17, 2018

ignatk Jan 16, 2018

vixentael Jan 16, 2018

ignatk Jan 16, 2018

vixentael Jan 16, 2018

ignatk Jan 16, 2018

vixentael Jan 17, 2018 •

edited

Loading

ignatk Jan 17, 2018

vixentael Jan 17, 2018

ignatk Jan 17, 2018

vixentael Jan 17, 2018

ignatk Jan 16, 2018

vixentael Jan 16, 2018

ignatk Jan 16, 2018 •

edited

Loading

ignatk Jan 16, 2018

ignatk Jan 16, 2018

mobrio commented Jan 16, 2018

vixentael commented Jan 17, 2018

vixentael commented Jan 17, 2018

vixentael commented Jan 17, 2018

secure cell x32-x64 compatibility fix #279

secure cell x32-x64 compatibility fix #279

Conversation

vixentael commented Jan 16, 2018 • edited Loading

ignatk left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vixentael Jan 17, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ignatk Jan 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mobrio commented Jan 16, 2018

vixentael commented Jan 17, 2018

vixentael commented Jan 17, 2018

vixentael commented Jan 17, 2018

vixentael commented Jan 16, 2018 •

edited

Loading

ignatk left a comment •

edited

Loading

vixentael Jan 17, 2018 •

edited

Loading

ignatk Jan 16, 2018 •

edited

Loading