github actions are broken for "ubuntu-18.04" bionic. "Mirror sync in progress?" error

[pestophagous]

Describe the bug
A continuous integration action is failing at an early step when I call sudo apt-get update

The error:

E: Failed to fetch https://packages.microsoft.com/repos/microsoft-ubuntu-bionic-prod/dists/bionic/main/binary-amd64/Packages.bz2  File has unexpected size (89974 != 89668). Mirror sync in progress? [IP: 40.76.35.62 443]
   Hashes of expected file:
    - Filesize:89668 [weak]
    - SHA512:239b3775157309c1f1ff14624d2d0044c5a0c57f6bf9a431f628de1c9f19c91e107c4bc78e1d12bce44121e5866fdf67328e6482151b6edd3456f8c541dc5739
    - SHA256:d737bf9c5418556ef337f43f0124a3db6e79c9f7209deb140ea87644169168b1
    - SHA1:de9e09d12abdb967a62954611af31655b3a8b5d9 [weak]
    - MD5Sum:9e9321f39e46dccc97d928ec9bffc87a [weak]
   Release file created at: Thu, 23 Jan 2020 17:58:24 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.
##[error]Process completed with exit code 100.

Area for Triage:
Containers + Packages + Ubuntu bionic + apt

Question, Bug, or Feature?:
Bug

Virtual environments affected

• macOS 10.15
• Ubuntu 16.04 LTS
• [x ] Ubuntu 18.04 LTS
• Windows Server 2016 R2
• Windows Server 2019

Expected behavior
This command should succeed:
sudo apt-get update

Actual behavior
A description with steps to reproduce the issue. If your have a public example or repo to share,
please provide the link.

1. Make a .github/workflows/main.yml that uses runs-on: ubuntu-18.04
2. During the action, try to sudo apt-get update

E: Failed to fetch https://packages.microsoft.com/repos/microsoft-ubuntu-bionic-prod/dists/bionic/main/binary-amd64/Packages.bz2  File has unexpected size (89974 != 89668). Mirror sync in progress? [IP: 40.76.35.62 443]
   Hashes of expected file:
    - Filesize:89668 [weak]
    - SHA512:239b3775157309c1f1ff14624d2d0044c5a0c57f6bf9a431f628de1c9f19c91e107c4bc78e1d12bce44121e5866fdf67328e6482151b6edd3456f8c541dc5739
    - SHA256:d737bf9c5418556ef337f43f0124a3db6e79c9f7209deb140ea87644169168b1
    - SHA1:de9e09d12abdb967a62954611af31655b3a8b5d9 [weak]
    - MD5Sum:9e9321f39e46dccc97d928ec9bffc87a [weak]
   Release file created at: Thu, 23 Jan 2020 17:58:24 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.
##[error]Process completed with exit code 100.

It seems this has happened in the recent past, then resolves in about a day:

dotnet/core#3221
dotnet/core#3988

Sorry to belabor the obvious, but this is disruptive to repositories that block merging of any PR until the actions succeed.

[traversaro]

As this is happening so often and it is not planned to be fixed any time soon (see dotnet/core#4167), probably the best thing for downstream GitHub Actions users that are not interested in apt packages offered by https://packages.microsoft.com/repos is to remove those from the apt sources before running apt-get update, see https://github.community/t5/GitHub-Actions/ubuntu-latest-Apt-repository-list-issues/m-p/41138/highlight/true#M4501 for example.

It would be great if the use of these apt repos in GitHub Actions could opt-in, instead of opt-out, to reduce the amount of boilerplate GitHub Actions necessary in every repo.

[traversaro]

Example step to remove the apt repos, from https://github.community/t5/GitHub-Actions/ubuntu-latest-Apt-repository-list-issues/m-p/41138/highlight/true#M4501 :

    # Remove apt repos that are known to break from time to time 
    # See https://github.com/actions/virtual-environments/issues/323  
    - name: Remove broken apt repos [Ubuntu]
      if: matrix.os == 'ubuntu-latest'
      run: |
        for apt_file in `grep -lr microsoft /etc/apt/sources.list.d/`; do sudo rm $apt_file; done

[pestophagous]

@traversaro that works for me. Thank you for this tip.

I'll leave this open for github to triage (or escalate/transfer to Microsoft teams) as they see fit.

After all, the fix that works for you and for me is still unacceptable for anyone who does need https://packages.microsoft.com/repos/microsoft-ubuntu-bionic***

[al-cheb]

Hello, @pestophagous
As I see dotnet/core#4179 issue was fixed. Please let me know can we close the issue?

[pestophagous]

Hey @al-cheb, thanks for taking an interest.

Indeed dotnet/core#4179 is closed (though I would not say fixed), but it is worth copy-pasting the last thing noted on that ticket:

We (.NET Core) have been trying very hard to work with the team responsible for this service since what I'd consider the landmark incident December 12, 2019 (#3988). That effort to get some improvement is tracked at dotnet/core#4167.

dotnet/core#4167 is still open.

The flakiness is the bug. And in that sense, the bug still exists. The fact that packages.microsoft.com is up and serving apt packages today does not mean the flakiness is gone.

However, the bug seems to belong to a non-github Microsoft team and not to github directly, so I agree with you that closing this github ticket for now is kind of our only option.

Lastly, however, I will also echo what @traversaro said here earlier:

It would be great if the use of these apt repos in GitHub Actions could opt-in, instead of opt-out, to reduce the amount of boilerplate GitHub Actions necessary in every repo.