Error transcrypt smudge context=default' failed 1 since version 2.2.1

[marian-finary]

General Context

We are using transcrypt in our mobile app repository. This mobile app works with Expo. We use Expo pipelines to create builds. One of the step during the build process is to use transcrypt for some files

Current Behavior

Since 2.2.1, run transcrypt fails randomly but mostly all the time, but only during creation of Android build.
For the pipeline that creates iOS build, there is not the issue at all.

Context (Environment)

Expo uses this docker image: ubuntu-20.04-jdk-11-ndk-r21e in the Android pipelines
You could find more information about it here: https://docs.expo.dev/build-reference/infrastructure/#ubuntu-2004-jdk-11-ndk-r21e-alias-default

Detailed Description

We use this script to install and run transcrypt, at some point in the pipeline:

#!/bin/bash

set -e

git clone https://github.com/elasticdog/transcrypt.git
${PWD}/transcrypt/transcrypt -p $1 -y
rm -rf transcrypt/

and we run it with that command: ./decryptFiles.sh my_transcrypt_password

So, since the 2.2.1 was released, we face that issue :

$ ./decryptFiles.sh xxxxxxxxxxxxxxxx
Cloning into 'transcrypt'...
.git/crypt/transcrypt: line 244: warning: command substitution: ignored null byte in input
error: external filter '"$(git config transcrypt.crypt-dir 2>/dev/null || printf %s/crypt ""$(git rev-parse --git-dir)"")"/transcrypt smudge context=default' failed 1
error: external filter '"$(git config transcrypt.crypt-dir 2>/dev/null || printf %s/crypt ""$(git rev-parse --git-dir)"")"/transcrypt smudge context=default' failed
fatal: eas.json: smudge filter crypt failed
error Command failed with exit code 128.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
yarn exited with non-zero code: 128

If we force to install the 2.2.0, by modifying our script with:

#!/bin/bash

set -e

git clone https://github.com/elasticdog/transcrypt.git --branch v2.2.0
${PWD}/transcrypt/transcrypt -p $1 -y
rm -rf transcrypt/

we don't have the issue at all and the trace looks like:

$ ./decryptFiles.sh xxxxxxxxxxxxxxxx
Cloning into 'transcrypt'...
Note: switching to 'ac99a937ccd7e04663f4bb0cdd9536d059ffd2db'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

git switch -c <new-branch-name>

Or undo this operation with:

 git switch -

Turn off this advice by setting config variable advice.detachedHead to false
The repository has been successfully configured by transcrypt.
Done in 2.41s.

So for now, we have set the version to 2.2.0

[jmurty]

Hi @marian-finary thanks for the feedback. I suspect the damage-undoing code I added as part of the 2.2.1 release in b380f2c might be to blame. Using version 2.2.0 specifically is a reasonable workaround for now.

I'll try to do some testing on that Docker image you linked. Does the Android build that fails have different encrypted files? Or do you think the problem is more likely caused by the runtime environment?

Apart from the original issue, I'd recommend some changes to your pipeline scripts to make it quicker and more reliable:

• Be aware that grabbing the main branch with git clone https://github.com/elasticdog/transcrypt.git could be risky because that branch includes the latest features, it is not the same as the latest stable release. Using a specific tagged release version will always be safer
• Cloning the whole repo is excessive, you could get just the specific version of transcrypt script you need with a simple fetch using e.g. wget https://raw.githubusercontent.com/elasticdog/transcrypt/v2.2.0/transcrypt
  Then chmod the download to run it, or use bash ./transcrypt

[jmurty]

I haven't been been able to reproduce this issue after doing some basic testing in the ubuntu:focal-20210921 Docker image. But the problem is probably due to a combination of this image and the specific file being decrypted.

I have made a change based on a hunch that might help, in commit 8be50df which is now the latest on main branch. At the very least it should solve the warning: command substitution: ignored null byte in input

It would be great if you could test this and confirm or deny the fix.

[rotarur]

I'm facing the same issue, I had to use the version 2.1.0 instead. is there any ETA on this fix?

[jmurty]

Hi @rotarur I haven't been able to reproduce this issue myself, so I'm not in a position to fix it.

Could you try the version in commit 8be50df to see if it helps? And can you confirm whether you are using the official 2.2.1 release, or the latest code on the main branch which is not the same thing?

Other than that, any info you could provide to narrow down the root problem would be helpful.

[rotarur]

I'll try that commit hash and see if it works.

I'll use the transcrypt script directly from that commit

[rotarur]

it worked

The repository has been successfully configured by transcrypt.

I've used your version transcrypt 2.3.0-pre

[jmurty]

Thanks very much for confirming that fix @rotarur and for the initial report @marian-finary.

I have released a new version 2.2.2 that includes the fix from the commit mentioned above, but without the other unstable features on the main branch.

I encourage everyone to use the latest official release (now 2.2.2) instead of the main branch.

[AtomicFS]

Hi, I have encountered this problem too in my Arch Linux system, with both v2.2.1 and v2.2.2.

I just cloned my dotfiles repository with transcrypt encrypted files in it, and tried to setup transcrypt.

• git clone <repo>
• transcrypt -> ERROR

*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
error: external filter '"$(git config transcrypt.crypt-dir 2>/dev/null || printf %s/crypt ""$(git rev-parse --git-dir)"")"/transcrypt smudge' failed 1
error: external filter '"$(git config transcrypt.crypt-dir 2>/dev/null || printf %s/crypt ""$(git rev-parse --git-dir)"")"/transcrypt smudge' failed
fatal: .mozilla/firefox/atomicfs-profile/storage/default/moz-extension+++380c1289-2b9c-4a2e-b1d5-f87e69c0b375^userContextId=4294967295/idb/3647222921wleabcEoxlt-eengsairo.sqlite: smudge filter crypt failed

In the error message above it fails on sqlite database, but I had it fail also on plain-text file too (UTF-8) but I can't replicate that at the moment.

Another way to replicate the problem:

• git clone --branch empty <repo>
• transcrypt
• git checkout master -> ERROR

Update

After further testing, I found that this issue is indeed dependent on the file. I just replicated the problem on the text file with load of URL links. I will do further testing.

Update 2

After some more testing, this is some seriously weird bug which is difficult to replicate. Total pain in the ass. But somehow I managed to replicate it in brand new repository with simple text file: transcrypt_bug repository

Just clone the repo and try to setup transcrypt, and you should get error:

$ transcrypt --cipher=aes-256-cbc --password="qLdF5e0xkI6WP+zlzDczNOqHALotjfD/EwLsvKol"

Repository metadata:

  GIT_WORK_TREE:  /tmp/transcrypt_bug
  GIT_DIR:        /tmp/transcrypt_bug/.git
  GIT_ATTRIBUTES: /tmp/transcrypt_bug/.gitattributes

The following configuration will be saved:

  CIPHER:   aes-256-cbc
  PASSWORD: 5elnshPnNS04NYGvI8PBiHLwlSuLtD40FsNf9T02

Does this look correct? [Y/n] 

error: external filter '"$(git config transcrypt.crypt-dir 2>/dev/null || printf %s/crypt ""$(git rev-parse --git-dir)"")"/transcrypt smudge' failed 1
error: external filter '"$(git config transcrypt.crypt-dir 2>/dev/null || printf %s/crypt ""$(git rev-parse --git-dir)"")"/transcrypt smudge' failed
fatal: lorem-ipsum.txt: smudge filter crypt failed

Used transcrypt 2.2.2.

It might have something to do with the size of the file maybe? When I tried it on lorem-ipsum.txt file with 9.2kB, the problem was not there. However when I increased the size to 56kB, the problem appeared.

[jmurty]

Hi @AtomicFS thanks so much for digging deep into this bug. As you say it's been a total pain to replicate, since it is file-dependent and the output gives no clues about the underlying problem.

Worse still, it must also be system-dependent, because I followed the steps in your transcrypt_bug repository but it did not reproduce the problem on my MacOS 13.2.1 system – the lorem-ipsum.txt file decrypted without problems for me.

I do have a couple of ideas for things to try next, if you're willing to continue digging?

hexdump requirement

The first is to confirm whether you have the hexdump command available on the system you are testing? This is a new requirement introduced in version 2.2.2, which isn't obvious and its absence would certainly break the smudge operation.

Undoing an ill-judged feature

The second thing is for me to completely remove a feature I added to the smudge operation in 2.2.1 that was supposed to automatically fix faulty encrypted files caused by #147. It's very likely that my auto-fixing code is causing these failures, since it does a lot of work and is therefore complicated. And it's also likely that adding this code was a bad idea in the first place, since it solves a problem that no-one may actually have in reality.

I have pushed a commit that removes the auto-fixing code from the smudge operation in commit d5b4bab (branch 158-smudge-errors). Could you please test the version of transcrypt in that commit/branch in your transcrypt_bug repository to see if it solves the problem on your system?

If the over-complicated auto-fixing smudge code is the root problem, I'll nuke it and make a new release, to hopefully finally fix this nasty lingering bug.

[AtomicFS]

Hi @jmurty

Oh, so the plot thickens ... even system-dependent :/ ... this sucks.

Just out of curiosity I tried to make a fresh installation of ArchLinux and I could replicate the problem on the transcrypt_bug repository. Maybe it has something to do the with one of the libraries? I am running openssl 3.0.8-1.

Regarding hexdump, I do have it (from util-linux 2.38.1). But shouldn't util-linux be listed in dependencies in the PKGBUILD?

The problem does not manifest when using commit d5b4bab.

[jmurty]

Thanks for following up @AtomicFS, I appreciate your effort.

I have released a new version 2.2.3 based on the trial commit above which will hopefully solve this once and for all.

I don't think it's worth the effort, and pain for users, to keep trying to make the problematic auto-fixing feature work across all systems for all files. If anyone does have a faulty encrypted file that could have been fixed by that feature – if/when it worked properly – they can instead ask for and get help here.

[AtomicFS]

@jmurty glad I can help.

[jmurty]

This issue has been quiet for a while now, so it looks like release 2.2.3 did finally fix the incompatibility issues.