Ubuntu 18.04 (Bionic) MS repo contains hash mismatches

[ogreenz]

While I was trying to sync my repository from https://packages.microsoft.com/ubuntu/18.04/prod it seemed to fail because there are md5 hash mismatches.

Error message:

Downloading https://packages.microsoft.com/ubuntu/18.04/prod/dists/bionic/InRelease...
Downloading https://packages.microsoft.com/ubuntu/18.04/prod/dists/bionic/Release...
Downloading https://packages.microsoft.com/ubuntu/18.04/prod/dists/bionic/Release.gpg...
openpgp: Signature made Sat, 16 Jan 2021 04:35:06 IST using RSA key ID EB3E94ADBE1229CF
openpgp: Good signature from "Microsoft (Release signing) <[email protected]>"
Downloading & parsing package files...
Downloading https://packages.microsoft.com/ubuntu/18.04/prod/dists/bionic/main/binary-amd64/Packages.bz2...
Building download queue...
Download queue: 1 items (6.17 MiB)
Downloading https://packages.microsoft.com/ubuntu/18.04/prod/pool/main/a/adutil-preview/adutil-preview_0.7.022_amd64.deb...
ERROR: unable to update: download errors:
  https://packages.microsoft.com/ubuntu/18.04/prod/pool/main/a/adutil-preview/adutil-preview_0.7.022_amd64.deb: md5 hash mismatch "5a7e403baf89f827a538c69c933bf722" != "99722ef91df57fac1723ce50450ce029

The file https://packages.microsoft.com/ubuntu/18.04/prod/dists/bionic/main/binary-amd64/Packages was changed at January 16th, 2020, and seems to contains now a wrong hash. I was able to sync a week ago.

I also was able to confirm it manually by downloading the file, compute the hash, and compare it to the one found inside the "Packages" file.

Solution:

The "Packages" file in the mirror needs to be re-generated.

Please let me know if you need any more information.

I'm cc'ing @dagood because I saw you've handled a similar issues #3988 & #2608 in the past.

Best,

Ofir.

[dagood]

adutil-preview/adutil-preview_0.7.022_amd64.deb

This package is unrelated to .NET and I'm not familiar with it--it would be ideal if you can report this to whoever maintains that package so they are alerted to the problem and can get the packages.microsoft.com team to fix it.

That said, if we can help (@rbhanda), the next piece of info we need is which mirror you're hitting.

@ogreenz Can you provide the full output of one of these commands (in particular to get the IP)?
ping packages.microsoft.com
wget https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb

[ogreenz]

Sure here's the output I'm getting:

$ ping packages.microsoft.com
PING csd-apt-weu-d-1.westeurope.cloudapp.azure.com (13.80.99.124) 56(84) bytes of data.

--- csd-apt-weu-d-1.westeurope.cloudapp.azure.com ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

$ wget https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb
--2021-01-19 21:54:03--  https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb
Resolving packages.microsoft.com (packages.microsoft.com)... 13.80.99.124
Connecting to packages.microsoft.com (packages.microsoft.com)|13.80.99.124|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3132 (3.1K) [application/octet-stream]
Saving to: 'packages-microsoft-prod.deb'

packages-microsoft-prod.deb                     100%[=====================================================================================================>]   3.06K  --.-KB/s    in 0s      

2021-01-19 21:54:04 (251 MB/s) - 'packages-microsoft-prod.deb' saved [3132/3132]

[dagood]

Thanks, that looks like the right info. I still think you should try contacting support for that package (SQL Server, Azure?) in case we're not able to help.

[rbhanda]

Hi @ogreenz I checked with our packages.microsoft.com team and they are not able to repro this issue including the mirror. They are suggesting to try to clear the cache and retry to see if that works.

[ogreenz]

Hi @rbhanda I don't think it's a cache issue. I just tried this from a US server and got the following address:

$ ping packages.microsoft.com
PING csd-apt-eus-d-2.eastus.cloudapp.azure.com (40.117.131.251) 56(84) bytes of data.

The exact same issue is happening there too.

you can just download these files and see for yourself:

http://40.117.131.251/ubuntu/18.04/prod/dists/bionic/main/binary-amd64/Packages

http://40.117.131.251/ubuntu/18.04/prod/pool/main/a/adutil-preview/adutil-preview_0.7.022_amd64.deb

the relevant package is defined in line 918 in the Packages file, in it you could see the MD5 is:
99722ef91df57fac1723ce50450ce029

while the downloaded file has the md5 hash of:
5A7E403BAF89F827A538C69C933BF722

This is from 2 separate instances of the same repo now.

[rbhanda]

Hi @ogreenz I followed up on this with our internal team but they were not able to repro this . Are you still seeing this issue?

[ogreenz]

Hi @rbhanda, yes downloading the files again yield the exact same result.
Are you telling me you're downloading:

http://40.117.131.251/ubuntu/18.04/prod/pool/main/a/adutil-preview/adutil-preview_0.7.022_amd64.deb

and getting MD5 of
99722ef91df57fac1723ce50450ce029
as written in the Packages file, line 928.

I think it's pretty binary either they are the same hash or they aren't.

[rbhanda]

Thanks for reporting @ogreenz . Since this package is not owned by .NET, I am not sure how else I can help here. Are you able to contact support for this package as suggested

Thanks, that looks like the right info. I still think you should try contacting support for that package (SQL Server, Azure?) in case we're not able to help.
#5860 (comment)

[dagood]

It looks like the latest version of the package isn't affected, only the old one. (I don't have insight into what the packages.microsoft.com team did to try but fail to repro the problem, though.) Here's a simple repro using Docker for a fresh start:

docker run -it --rm ubuntu:18.04 bash -c '
  set -x
  apt update
  apt install -y sudo wget
  wget https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
  sudo dpkg -i packages-microsoft-prod.deb
  apt update
  apt install -y adutil-preview=0.7.022'

[...]
+ wget https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
--2021-02-05 17:22:19--  https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb
Resolving packages.microsoft.com (packages.microsoft.com)... 40.117.131.251
Connecting to packages.microsoft.com (packages.microsoft.com)|40.117.131.251|:443... connected.
[...]
Get:2 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 adutil-preview amd64 0.7.022 [6470 kB]
Err:2 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 adutil-preview amd64 0.7.022
  Hash Sum mismatch
  Hashes of expected file:
   - SHA512:8c6dd6a665ca304a85213cf3cea33e174fddfbb9858dbadfbf86b1cfbaa8f5d2fe361abc472ea20175dc3d0388445e8c8b4ce8deec410d44fe2cd466cb886948
   - SHA256:be217cd0aebb6828b15943acbc0206fbea1068d9092cd3891e640a59da92a21c
   - SHA1:f657b73dd8d6c1486a0862f53010a4562e287cb5 [weak]
   - MD5Sum:99722ef91df57fac1723ce50450ce029 [weak]
   - Filesize:6470148 [weak]
  Hashes of received file:
   - SHA512:745910d8c27c57bb59167a02efaf606014524c390691a0426c4c12a0306f35994259197c4ee10f99cbe20d1e3299e3b0c5115c33cdb099d6686594664809af4c
   - SHA256:87cdec15a4d25a7739fc370e8c5e3e0da2c0d2a44215c348d68d83e0b7edee3a
   - SHA1:89283b83f411664e29b3dffc649d63b4c9f0596c [weak]
   - MD5Sum:5a7e403baf89f827a538c69c933bf722 [weak]
   - Filesize:6470148 [weak]
  Last modification reported: Wed, 18 Nov 2020 23:10:30 +0000

This might be useful for a report to SQL/Azure to narrow it down.

[ogreenz]

Hi @rbhanda, the package isn't owned by .NET but this repo is where you update .NET core from, and it's broken.
Another option here is just to remove the package if it's not used anywhere or, fix the hash file.

[teottin]

@dagood
There is currently a hash mismatch with the package servicefabric_sdkcommon_1.4.2.deb
https://packages.microsoft.com/ubuntu/18.04/prod/pool/main/s/servicefabricsdkcommon/servicefabric_sdkcommon_1.4.2.deb: md5 hash mismatch "fa8ef8b014dab9409464f20238e8c67f" != "cf389033d5f91dceb7afe362511876e7"

I downloaded the file manually and checked the hash against the one in the Packages file and there is indeed a mismatch

[dagood]

@teottin, can you please file a new issue for that package and/or report the problem to whoever pointed you at that package?

[teottin]

@teottin, can you please file a new issue for that package and/or report the problem to whoever pointed you at that package?

@dagood It seems the problem has been resolved now, I guess someone resposible for packages.microsoft.com updated the pacakge index.
No one pointed me to the package in question, it was just a random package, but I have set up a local mirror of packages.microsoft.com ubuntu18 repo and when there is a mismatch in a package checksum and the package index, it will not sync. I am guessing Microsoft has someone responsible for maintaining the debian repositories as a whole? Who should I contact if this happens again?

[dagood]

I am guessing Microsoft has someone responsible for maintaining the debian repositories as a whole?

Not as far as I know (unfortunately). In my experience, the team that maintains packages.microsoft.com defers a lot of responsibility onto the individual product teams that push packages into it. However, I don't work in this area anymore--the best way to get your issue routed correctly is to file a new issue. The team wasn't able to repro the original issue in this thread (even though I was able to), but maybe with a new package/report they would have been able to repro and fix it proactively.

What might have happened in this case is that another version of the package happened to be published, which regenerated the index without anyone really being aware of this problem.

Whether or not we're able to recommend mirroring the repo is still up in the air: #2882. (I assume it's fine, but I don't know about the licenses of every package potentially being redistributed.)