Docker-sync with unision strategy broken for newer MacOS / homebrew installation

[endam]

Error/Feature Requestion/Docs

Error

Docker Driver

Docker for Mac 2.1.2.0(38030)

Sync strategy

unison

your docker-sync.yml

version: '2'
syncs:
  sync-app:
    src: './backend/'
    sync_strategy: unison

OS

OSX Version: 10.14.6

docker-sync version

0.5.11

Problem

Running docker-sync start result in "error during unmarshaling (input_value: ill-formed message)"
Full log:

$ docker-sync start
ok  Starting unison for sync sync-app
error  Error starting sync, exit code 0
message  Warning: No archive files were found for these roots, whose canonical names are:
{docker-sync.yml root dir}/backend
//4bd40753d696//app_sync
This can happen either
because this is the first time you have synchronized these roots,
or because you have upgraded Unison to a new version with a different
archive format.

Update detection may take a while on this run if the replicas are
large.

Unison will assume that the 'last synchronized state' of both replicas
was completely empty.  This means that any files that are different
will be reported as conflicts, and any files that exist only on one
replica will be judged as new and propagated to the other replica.
If the two replicas are identical, then no changes will be reported.

If you see this message repeatedly, it may be because one of your machines
is getting its address from DHCP, which is causing its host name to change
between synchronizations.  See the documentation for the UNISONLOCALHOSTNAME
environment variable for advice on how to correct this.

Donations to the Unison project are gratefully accepted:
http://www.cis.upenn.edu/~bcpierce/unison


message  Unison 2.51.2 (ocaml 4.08.1): Contacting server...
Connected [//4bd40753d696//app_sync -> //K0529-171219.local/{docker-sync.yml root dir}/backend]
Looking for changes
Waiting for changes from server
Reconciling changes
Fatal error: Server: Fatal error during unmarshaling (input_value: ill-formed message),
possibly because client and server have been compiled with differentversions of the OCaml compiler.
success  Unison server started
error  Error starting sync, exit code 0
message  Warning: No archive files were found for these roots, whose canonical names are:
{docker-sync.yml root dir}/backend
//4bd40753d696//app_sync
This can happen either
because this is the first time you have synchronized these roots,
or because you have upgraded Unison to a new version with a different
archive format.

Update detection may take a while on this run if the replicas are
large.

Unison will assume that the 'last synchronized state' of both replicas
was completely empty.  This means that any files that are different
will be reported as conflicts, and any files that exist only on one
replica will be judged as new and propagated to the other replica.
If the two replicas are identical, then no changes will be reported.

If you see this message repeatedly, it may be because one of your machines
is getting its address from DHCP, which is causing its host name to change
between synchronizations.  See the documentation for the UNISONLOCALHOSTNAME
environment variable for advice on how to correct this.

Donations to the Unison project are gratefully accepted:
http://www.cis.upenn.edu/~bcpierce/unison


message  Unison 2.51.2 (ocaml 4.08.1): Contacting server...
Connected [//4bd40753d696//app_sync -> //K0529-171219.local/{docker-sync.yml root dir}/backend]
Looking for changes
Waiting for changes from server
Reconciling changes
Fatal error: Server: Fatal error during unmarshaling (input_value: ill-formed message),
possibly because client and server have been compiled with differentversions of the OCaml compiler.
success  Starting Docker-Sync in the background

unison was installed with homebrew.
I would appreciate it very much if you could point out anything to me.

[EugenMayer]

Fatal error: Server: Fatal error during unmarshaling (input_value: ill-formed message),
possibly because client and server have been compiled with differentversions of the OCaml compiler

https://unix.stackexchange.com/questions/309669/unison-fatal-error-during-unmarshaling

Most probably either the wrong unison package installed or the current one in homebrew is not compatible.

The release was adjusted very recently https://github.com/Homebrew/homebrew-core/blob/master/Formula/unison.rb

And there might be already an fix for that https://github.com/Homebrew/homebrew-core/blob/master/Formula/unison.rb#L18

maybe brew upgrade and test again.

[endam]

Thanks, but the same error happened after even brew upgrade or downgrade....
I gave up unison at once and will switch to another sync strategy.

[EugenMayer]

@endam well i cannot do anything about that right now - unison strategy being depenend on the "hosts unison" ( not dockerized ) and that adds an extra layer of complexity - this is one reason why we have chosen native_osx as the default ( there are more ). Unison and unox under OSX / brew tend to break more then once. Once it is some OCALM, then it's python madness, then something else.

I keep this issue open for anybody running into this issue with unison. I expect this to be resolved by homebrew "auotmatically" by fixing the build

[JParkinson1991]

Also experiencing the same problem. #688

Just to clear things up, is it docker-sync using 4.06.1 to compile? and brew using 4.08.1 to compile?

Edit: by docker-sync i mean the eugenmayer/unison Dockerfile

[EugenMayer]

@JParkinson1991 docker sync does use the unison docker image at https://github.com/EugenMayer/docker-image-unison

https://github.com/EugenMayer/docker-image-unison/blob/master/Dockerfile#L5 and the unison version https://github.com/EugenMayer/docker-image-unison/blob/master/Dockerfile#L3

OCALM is picked from testing, so could be anything - but it is not rebuild all the time.

You can just try to rebuild the docker image locally using the right tag https://github.com/EugenMayer/docker-sync/blob/master/lib/docker-sync/sync_strategy/unison.rb#L25

eugenmayer/unison:2.51.2.1

docker build . -t eugenmayer/unison:2.51.2.1

And then test again. If that works we need to do that officially, release .2.51.2.2 with the updated OCALM version

[JParkinson1991]

I'll have a dig around today @EugenMayer

Im hoping doing a quick update of the ocaml version used to compile in the unison image should fix this. Looking at alpine packages it seems there default ocaml version is 4.08.1 https://pkgs.alpinelinux.org/packages?name=ocaml&branch=edge so that should match the brew version.

Question though, why does this not affect native_osx? Both native_osx and unison syn strategies build ontop of the docker-image-unsion right?

[EugenMayer]

native_osx does not use unison on the host ( OSX ) but rather osxfs and has no dependencies - unison is just used inside the container locally and thus is consistent in its build

[JParkinson1991]

I see, explains why there is no error. I wonder if it's worth (assuming the ocaml version bump fixes this) maintain both the unison image and the required host binary.

Bringing both 'in house' would eliminate these compiler mismatch errors etc.

Either way, ill take a look at rebuilding the image locally today and see where that gets me.

[JParkinson1991]

@EugenMayer

Managed to update the docker-image-unsion Dockefile to compile Unison with OCaml 4.08.1.

As suggested, tagged this locally with 2.51.2.2 and updated the unison.rb file in docker-sync gem to reference and all works absolutely fine!

Opened up a pull request for you to have a look at.

[EugenMayer]

thank you for your work!

You do not need to match the unison.rb file yet in terms of using a custom image version, use the docker-sync.yml setting for that:
https://github.com/EugenMayer/docker-sync/blob/master/lib/docker-sync/sync_strategy/unison.rb#L23

https://docker-sync.readthedocs.io/en/latest/getting-started/configuration.html#references

options:
  unison_image: 'eugenmayer/unison:2.51.2.2'

Beside that, it would make sense to patch the unsion.rb file for the next release - could you open a PR here to? Would be great!

I already build and published eugenmayer/unison:2.51.2.2 - would need some more testers before i can merge in the unison.rb PR though

[JParkinson1991]

That unsion_image configuration makes things alot easier for me and other mac users for the time being. Thanks for publishing the newly tagged image so quickly!

Ive opened the PR for the update to the default unison image, defeinitely needs more tests. #689

Obviously the big one here is that Unison needs patching to be compiled with ocaml 4.08.1 but that shouldnt cause any problems for other operating systems.

[EugenMayer]

will be fixed with 0.5.12

[armellarcier]

Guys this is great, thanks a lot for the time you put in docker-sync. But really, how should I upgrade please ? It's not clear to me... sorry!

[JParkinson1991]

@Benew

The updated unison image (OCAML 4.08.1 compatible) is now used by default in docker-sync core from version 0.5.13

Theres no need to change any options as detailed above, all you need to do is update the docker-sync gem.

gem update docker-sync

Hope that helps

[inPhoenix]

@JParkinson1991 The update command didnt work with for me. I'm on macOS Catalina 10.15.3.

[JParkinson1991]

@inPhoenix working fine here, on same OS, can only assume this is a problem at your end and not with this package.

[joeRinehart]

@inPhoenix - A coworker and I hit this today: on 10.15.3, unison up to date, all of that jazz. What we eventually found was unison in the existing sync containers was still on the old OCaml compiler.

Destroying the containers and letting docker-sync re-create them brought everything into unison (pun intended) and much joy was had.

[hibikiosawa4388]

I have same problem.
I get a temporary measure.

unison -version # => unison version 2.51.3.70 (ocaml 4.12.0)

cd /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula
git log unison.rb
git checkout 0edb12c144c6f64e6c35bdd4f6afb7d56d10bf60
brew uninstall unison
HOMEBREW_NO_AUTO_UPDATE=1 brew install unison

unison -version # => unison version 2.51.3 (ocaml 4.10.0)

it's worked docker-sync after executing the command.

[rommeras]

@hibikiosawa4388 Thanks bro! You made my day!!!

[donnykurnia]

@hibikiosawa4388 trying it today, but it not working anymore

$ HOMEBREW_NO_AUTO_UPDATE=1 brew install unison
Warning: Treating unison as a formula. For the cask, use homebrew/cask/unison
==> Downloading https://ghcr.io/v2/homebrew/core/unison/manifests/2.51.3

curl: (22) The requested URL returned error: 404
Error: Failed to download resource "unison_bottle_manifest"
Download failed: https://ghcr.io/v2/homebrew/core/unison/manifests/2.51.3

[donnykurnia]

For temporary solution, I download unison v2.51.3 with ocaml-4.08.1 from https://github.com/bcpierce00/unison/releases/tag/v2.51.3 extract the content, and put the unison executable into bin folder in my home and use it for docker-sync.

[hanie21]

For temporary solution, I download unison v2.51.3 with ocaml-4.08.1 from https://github.com/bcpierce00/unison/releases/tag/v2.51.3 extract the content, and put the unison executable into bin folder in my home and use it for docker-sync.

I did the process but still has the same issue when runnig "brew install Elixir" on Mac

[donnykurnia]

@hanie21 I'm not using elixir so I guess that's is a different issues.

[artemsheludko1]

@EugenMayer Can you please provide new version for OCAML 4.12.0?

[EugenMayer]

@artemsheludko1 if you like, run a PR for EugenMayer/docker-image-unison#18 for that purpose - i cannot currently work on this myself but i can maintain PRs

[artemsheludko1]

@EugenMayer Unfortunately I am not very familiar with this stuff.

[donnykurnia]

@EugenMayer EugenMayer/docker-image-unison#20

[donnykurnia]

For those who need docker-image-unison with the same OCAML version with the latest unison on homebrew, you can try my image:

https://hub.docker.com/r/donnykurnia/unison

To use this image, follow this guide: https://docker-sync.readthedocs.io/en/latest/getting-started/configuration.html#sync-strategy-image

[EugenMayer]

Be aware it is 2.54 and not 2.53 - as far as i'am aware this will most probably create issues for macos users, since those are still on 2.53 https://formulae.brew.sh/formula/unison

[hibikiosawa4388]

How to brew upgrade unison.

step 1:

Add the following to your docker-sync.yml.

options:
  unison_image: 'eugenmayer/unison:2.51.3-4.12.0-AMD64'

ATTENTION:
You will need to check the unison version and choose the appropriate docker image for it.
( You should check version by unison -version )

link : https://github.com/EugenMayer/docker-image-unison

step 2:

$ brew upgrade unison

$ cd $PROJECT_DIR
$ ls | grep docker-sync.yml
# => docker-sync.yml


# ↓ Important:  
$ docker-sync clean

# ↓Make sure unison_image is already selected.
$ cat docker-sync.yml | grep -B1 unison_image  
# => options:
# =>  unison_image: 'eugenmayer/unison:2.51.3-4.12.0-AMD64'

# ↓Verify that the version displayed is the same.
$ unison  -version 
# => unison version 2.51.3 (ocaml 4.12.0)

$ docker-sync start 
# => ...
# => UNISON 2.51.3 (OCAML 4.12.0) started propagating 
# => ...

@EugenMayer @donnykurnia
Thx for fix it.

[torson]

To downgrade Unison to 2.51.3 , use this :
https://www.noulakaz.net/2021/03/21/installing-a-previous-version-of-a-program-in-homebrew/

brew tap-new avinash/homebrew-old
brew extract --version 2.51.3 unison avinash/homebrew-old
brew install avinash/homebrew-old/[email protected]

Though this still didn't work for me, initial sync works but then background sync doesn't .

The only strategy that works for me currently on Mac M1 using Colima (can't use Docker Desktop because some amd64 containers don't work there as on arm64 it's not doing proper virtualization, and Colima does full software amd64 virtualization) is with using a custom native_osx_image from this post: #346 .
It uses up 1 CPU core (around 80%) all the time with my current codebase because it's polling all the time, but at least it's working, both directions.

[torson]

Ah correction, the unison strategy started working for me after I've installed the new file-system watcher instead of unox :
https://github.com/EugenMayer/docker-sync/releases/tag/0.7.0

brew unlink unox
brew install autozimu/homebrew-formulas/unison-fsmonitor

[bertBruynooghe]

@torson Ha, you really made my day! I ran in to the same issues as you after switching to an M1 Mac, and the new file-system watcher made everything work again!