Pacman 5.2.1-7 breaks upgrade from older installation (e.g. appveyor)

[hessenfarmer]

Describe the bug

it is not possible to upgrade from older Versions to newer version anymore as to decompress 5.2.1-7 it is necessary to use zstd. Support to use this properly came only a couple of days before with 5.2.1-6 any older version will fail to decompress the new pacman.

Steps to Reproduce the Problem

run Pacman -Sy pacman on any older installation

Some regression code to update older versions might be good otherwise one have to do a stepped upgrade first to 5.2.1-6 and then in the next run to the newest pacman

[mati865]

You can use newer installer from https://github.com/msys2/msys2-installer/releases/ or download libzstd, older pacman and ztsd from http://repo.msys2.org/msys/x86_64 with curl and install with pacman -U.

[sn-lurman]

Please resolve this issue as soon as you can. We have windows build fail due to this problem

[sn-lurman]

Log from our build:

NFO: Processing execute[upgrade msys2 database and core packages] action run (C:/Users/ADMINI~1/AppData/Local/Temp/kitchen/cache/cookbooks/mingw/resources/msys2_package.rb line 41)

    [execute] :: Synchronizing package databases...
              downloading mingw32.db...
              downloading mingw32.db.sig...
              downloading mingw64.db...
              downloading mingw64.db.sig...
              downloading msys.db...
              downloading msys.db.sig...
              :: Starting core system upgrade...
              resolving dependencies...
              looking for conflicting packages...
              warning: terminate other MSYS2 programs before proceeding
  
              Packages (8) bash-4.4.023-2  filesystem-2020.02-2  libzstd-1.4.4-2  mintty-1~3.1.4-1  msys2-runtime-3.1.4-2  pacman-5.2.1-7  pacman-mirrors-20200329-1  zstd-1.4.4-2
  
              Total Download Size:   19.95 MiB
              Total Installed Size:  66.40 MiB
              Net Upgrade Size:      15.85 MiB
  
              :: Proceed with installation? [Y/n]
              :: Retrieving packages...
              downloading msys2-runtime-3.1.4-2-x86_64.pkg.tar.zst...
              downloading bash-4.4.023-2-x86_64.pkg.tar.xz...
              downloading filesystem-2020.02-2-x86_64.pkg.tar.xz...
              downloading mintty-1~3.1.4-1-x86_64.pkg.tar.xz...
              downloading pacman-mirrors-20200329-1-any.pkg.tar.xz...
              downloading libzstd-1.4.4-2-x86_64.pkg.tar.xz...
              downloading zstd-1.4.4-2-x86_64.pkg.tar.xz...
              downloading pacman-5.2.1-7-x86_64.pkg.tar.zst...
              checking keyring...
              checking package integrity...
              loading package files...
              error: could not open file /var/cache/pacman/pkg/msys2-runtime-3.1.4-2-x86_64.pkg.tar.zst: Unrecognized archive format
              error: could not open file /var/cache/pacman/pkg/pacman-5.2.1-7-x86_64.pkg.tar.zst: Unrecognized archive format
              error: failed to commit transaction (cannot open package file)
              Errors occurred, no packages were upgraded.
  
    ================================================================================
    Error executing action `run` on resource 'execute[upgrade msys2 database and core packages]'
    ================================================================================

[mati865]

@sn-lurman it's resolved in newer installers: https://github.com/msys2/msys2-installer/releases/

[sn-lurman]

I'm using omnibus and it's not working due to this change.
I do not have permission to fix omnibus.
How you can help me?

[mati865]

This might work for you:

base_url='http://repo.msys2.org/msys/x86_64'
curl ${base_url}/libzstd-1.4.4-2-x86_64.pkg.tar.xz -o libzstd-1.4.4-2-x86_64.pkg.tar.xz
curl ${base_url}/pacman-5.2.1-6-x86_64.pkg.tar.xz -o pacman-5.2.1-6-x86_64.pkg.tar.xz
curl ${base_url}/zstd-1.4.4-2-x86_64.pkg.tar.xz -o zstd-1.4.4-2-x86_64.pkg.tar.xz
pacman -U --noconfirm libzstd-1.4.4-2-x86_64.pkg.tar.xz pacman-5.2.1-6-x86_64.pkg.tar.xz zstd-1.4.4-2-x86_64.pkg.tar.xz
rm libzstd-1.4.4-2-x86_64.pkg.tar.xz pacman-5.2.1-6-x86_64.pkg.tar.xz zstd-1.4.4-2-x86_64.pkg.tar.xz

[shlomif]

This might work for you:
Hi!

First off all note that it is affecting my builds too: https://ci.appveyor.com/project/shlomif/fc-solve/builds/33023327 .

base_url='http://repo.msys2.org/mingw/x86_64/'
curl ${base_url}/libzstd-1.4.4-2-x86_64.pkg.tar.xz -o libzstd-1.4.4-2-x86_64.pkg.tar.xz
curl ${base_url}/pacman-5.2.1-6-x86_64.pkg.tar.xz -o pacman-5.2.1-6-x86_64.pkg.tar.xz
curl ${base_url}/zstd-1.4.4-2-x86_64.pkg.tar.xz -o zstd-1.4.4-2-x86_64.pkg.tar.xz
pacman -U --noconfirm libzstd-1.4.4-2-x86_64.pkg.tar.xz pacman-5.2.1-6-x86_64.pkg.tar.xz zstd-1.4.4-2-x86_64.pkg.tar.xz
rm libzstd-1.4.4-2-x86_64.pkg.tar.xz pacman-5.2.1-6-x86_64.pkg.tar.xz zstd-1.4.4-2-x86_64.pkg.tar.xz

Which language is it? Is it sh?

[mati865]

@shlomif execute it in MSYS2 shell directly or put into shell script.

[shlomif]

@mati865 : ok, thanks! The script does not work nor does it fix the later issue: https://ci.appveyor.com/project/shlomif/fc-solve/builds/33029422

[mati865]

@shlomif there was a typo, edited the comment.

[shlomif]

@mati865 : I tried your edited version and it still does not work: https://ci.appveyor.com/project/shlomif/fc-solve/builds/33033336 . The only change there seems to be the removal of a double slash in the URIs.

[mati865]

Ok, now that's embarrassing. Should work this time.

[shlomif]

@mati865 : thanks! Seems to work: https://ci.appveyor.com/project/shlomif/fc-solve/builds/33036043 - but i have yet to see if the build success will be restored.

[shlomif]

Update: the build is successful. For the record, here is a refactored (and not yet tested) code (my changes are under CC0 ):

set -e -x

base_url='http://repo.msys2.org/msys/x86_64/'
packages="libzstd-1.4.4-2-x86_64.pkg.tar.xz pacman-5.2.1-6-x86_64.pkg.tar.xz zstd-1.4.4-2-x86_64.pkg.tar.xz"
for p in $packages
do
    curl "${base_url}$p" -o "$p"
done
pacman -U --noconfirm $packages
rm -f $packages

[lazka]

appveyor should work again now. Can anyone confirm?

[hessenfarmer]

Appveyor works if you first upgrade to the previous pacman version and then update the whole thing. But everybody using appveyor MSYS2 has to include this workaround in his .yml as long as either there is a fix in MSYS2 or there is an MSYS2 update on appveyor.
Might be good to ask them to upgrade their environments with the latest version of the MSYS2 installation.

[mati865]

@hessenfarmer the fix done by @lazka was uploaded recently. He is just waiting on confirmation everything works.

[lazka]

some of our own appveyor jobs ran through fine now, so I assume this is fixed.

[sn-lurman]

Thank you very much.
Now it's working.

[sn-lurman]

Hi Lazka,

For some reason the same problem is up again:

: Proceed with installation? [Y/n]
16:05:58 :: Retrieving packages...
16:05:58 downloading msys2-runtime-3.1.4-3-x86_64.pkg.tar.xz...
16:05:58 downloading bash-4.4.023-2-x86_64.pkg.tar.xz...
16:05:58 downloading filesystem-2020.02-2-x86_64.pkg.tar.xz...
16:05:58 downloading mintty-1~3.1.6-1-x86_64.pkg.tar.xz...
16:05:58 downloading pacman-mirrors-20200329-1-any.pkg.tar.xz...
16:05:58 downloading libzstd-1.4.5-1-x86_64.pkg.tar.zst...
16:05:58 downloading zstd-1.4.5-1-x86_64.pkg.tar.zst...
16:05:58 downloading pacman-5.2.1-8-x86_64.pkg.tar.xz...
16:05:58 checking keyring...
16:05:58 checking package integrity...
16:05:58 loading package files...
16:05:58 Errors occurred, no packages were upgraded.
16:05:58 STDERR: warning: terminate other MSYS2 programs before proceeding
16:05:58 error: could not open file /var/cache/pacman/pkg/libzstd-1.4.5-1-x86_64.pkg.tar.zst: Unrecognized archive format
16:05:58 error: could not open file /var/cache/pacman/pkg/zstd-1.4.5-1-x86_64.pkg.tar.zst: Unrecognized archive format
16:05:58 error: failed to commit transaction (cannot open package file)
16:05:58 ---- End output of .\bin\bash.bat -c 'pacman -Syu --noconfirm' ----
16:05:58 Ran .\bin\bash.bat -c 'pacman -Syu --noconfirm' returned 1

[lazka]

yeah, I've posted something here: https://www.msys2.org/news/#2020-05-31-update-fails-with-could-not-open-file