RuntimeError: DNF error: Problems in request: #67

cmoulliard · 2024-10-18T16:40:44Z

Issue

When we try to grab the urls of RPMs for fedora, we got such an error

Failed to set locale, defaulting to C.UTF-8
INFO:root:Running solver for x86_64
INFO:root:$ skopeo --override-arch=amd64 copy docker://fedora dir:/tmp/tmp8kt_7dmw
Getting image source signatures
Copying blob f7bb57d05c2a done   |
Copying config eca85b75dc done   |
Writing manifest to image destination
INFO:root:Extracting rpmdb from layer sha256:f7bb57d05c2a01f21aeee5873ebc9afe04304b3f5ea7b4c47e0ce18ac2d38124
INFO:dnf:Added updates repo from
INFO:dnf:Added updates-debuginfo repo from
INFO:dnf:Added updates-source repo from
INFO:dnf:Added fedora repo from
INFO:dnf:Added fedora-debuginfo repo from
INFO:dnf:Added fedora-source repo from
INFO:dnf:Last metadata expiration check: 0:00:01 ago on Fri Oct 18 12:31:09 2024.
ERROR:dnf:No match for argument: golang
ERROR:dnf:No match for argument: podman
ERROR:dnf:No match for argument: jq
ERROR:root:Problems in request:
missing packages: golang, podman, jq
Traceback (most recent call last):
  File "/home/cloud-user/.local/lib/python3.9/site-packages/rpm_lockfile/__init__.py", line 155, in resolver
    base.install_specs(solvables)
  File "/usr/lib/python3.9/site-packages/dnf/base.py", line 2053, in install_specs
    raise dnf.exceptions.MarkingErrors(no_match_group_specs=no_match_group_specs,
dnf.exceptions.MarkingErrors: Problems in request:
missing packages: golang, podman, jq

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cloud-user/.local/bin/rpm-lockfile-prototype", line 8, in <module>
    sys.exit(main())
  File "/home/cloud-user/.local/lib/python3.9/site-packages/rpm_lockfile/__init__.py", line 439, in main
    process_arch(
  File "/home/cloud-user/.local/lib/python3.9/site-packages/rpm_lockfile/__init__.py", line 244, in process_arch
    packages, sources, module_metadata = resolver(
  File "/home/cloud-user/.local/lib/python3.9/site-packages/rpm_lockfile/__init__.py", line 158, in resolver
    raise RuntimeError(f"DNF error: {exc}")
RuntimeError: DNF error: Problems in request:
missing packages: golang, podman, jq

Command executed

git clone https://github.com/redhat-buildpacks/paketo-container
cd paketo-container
podman run --rm -v "$PWD:$PWD:z" -w "$PWD" fedora cp -r /etc/yum.repos.d/. .
rpm-lockfile-prototype -f Containerfile rpms.in.yaml --outfile rpms.lock.yaml

Repo File

rpms.in.yaml
contentOrigin:
  repofiles:
    - ./fedora-updates.repo
    - ./fedora.repo
packages:
  - golang
  - podman
  - jq
  - gettext

fedora.repo

cat fedora.repo
[fedora]
name=Fedora $releasever - $basearch
#baseurl=http://download.example/pub/fedora/linux/releases/$releasever/Everything/$basearch/os/
metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch
enabled=1
countme=1
metadata_expire=7d
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
skip_if_unavailable=False

[fedora-debuginfo]
name=Fedora $releasever - $basearch - Debug
#baseurl=http://download.example/pub/fedora/linux/releases/$releasever/Everything/$basearch/debug/tree/
metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-debug-$releasever&arch=$basearch
enabled=0
metadata_expire=7d
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
skip_if_unavailable=False

[fedora-source]
name=Fedora $releasever - Source
#baseurl=http://download.example/pub/fedora/linux/releases/$releasever/Everything/source/tree/
metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-source-$releasever&arch=$basearch
enabled=0
metadata_expire=7d
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
skip_if_unavailable=False

The text was updated successfully, but these errors were encountered:

lubomir · 2024-10-21T06:37:33Z

The error is that dnf doesn't find the requested packages in the provided repo. That's strange, as Fedora does package all of them.

I tried replicating this locally, but it does work for me.

This is quite possibly caused by the metalink. Maybe it's getting resolved to a mirror that is out dated. Does it happen consistently over time?

In general, metalinks are only good for experimenting and playing around with the tool. It seems to me that's what you are doing here, given the fedora image with no tag. You could try pointing directly at a Fedora compose by setting baseurl to one of the following:
https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-41/compose/Everything/$basearch/os
https://kojipkgs.fedoraproject.org/compose/rawhide/latest-Fedora-Rawhide/compose/Everything/$basearch/os
There's no compose for the released versions though.

cmoulliard · 2024-10-21T07:20:39Z

This is quite possibly caused by the metalink. Maybe it's getting resolved to a mirror that is out dated. Does it happen consistently over time?

Yes. I did a test right now and got the same error

cmoulliard · 2024-10-21T07:22:04Z

In general, metalinks are only good for experimenting and playing around with the tool. It seems to me that's what you are doing here, given the fedora image with no tag. You could try pointing directly at a Fedora compose by setting baseurl to one of the following:

Are you saying that such an issue could be resolved if I specify the fedora version to be used from this command podman run --rm -v "$PWD:$PWD:z" -w "$PWD" fedora cp -r /etc/yum.repos.d/. . ?

cmoulliard · 2024-10-21T07:30:01Z

Here is a new test which is failing too using fedora:40

[cloud-user@rhel-9 paketo-container]$ podman run --rm -v "$PWD:$PWD:z" -w "$PWD" fedora:40 cp -r /etc/yum.repos.d/. .
Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.fedoraproject.org/fedora:40...
Getting image source signatures
Copying blob 311336741819 done   |
Copying config 8bf3872d7a done   |
Writing manifest to image destination

$ rpm-lockfile-prototype -f Containerfile rpms.in.yaml --outfile rpms.lock.yaml
Failed to set locale, defaulting to C.UTF-8
INFO:root:Running solver for x86_64
INFO:root:$ skopeo --override-arch=amd64 copy docker://fedora dir:/tmp/tmpjipyr8kd
Getting image source signatures
Copying blob f7bb57d05c2a done   |
Copying config eca85b75dc done   |
Writing manifest to image destination
INFO:root:Extracting rpmdb from layer sha256:f7bb57d05c2a01f21aeee5873ebc9afe04304b3f5ea7b4c47e0ce18ac2d38124
INFO:dnf:Added updates repo from
INFO:dnf:Added updates-debuginfo repo from
INFO:dnf:Added updates-source repo from
INFO:dnf:Added fedora repo from
INFO:dnf:Added fedora-debuginfo repo from
INFO:dnf:Added fedora-source repo from
ERROR:dnf:No match for argument: podman
ERROR:dnf:No match for argument: golang
ERROR:dnf:No match for argument: jq
ERROR:root:Problems in request:
missing packages: podman, golang, jq
Traceback (most recent call last):
  File "/home/cloud-user/.local/lib/python3.9/site-packages/rpm_lockfile/__init__.py", line 155, in resolver
    base.install_specs(solvables)
  File "/usr/lib/python3.9/site-packages/dnf/base.py", line 2053, in install_specs
    raise dnf.exceptions.MarkingErrors(no_match_group_specs=no_match_group_specs,
dnf.exceptions.MarkingErrors: Problems in request:
missing packages: podman, golang, jq

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cloud-user/.local/bin/rpm-lockfile-prototype", line 8, in <module>
    sys.exit(main())
  File "/home/cloud-user/.local/lib/python3.9/site-packages/rpm_lockfile/__init__.py", line 439, in main
    process_arch(
  File "/home/cloud-user/.local/lib/python3.9/site-packages/rpm_lockfile/__init__.py", line 244, in process_arch
    packages, sources, module_metadata = resolver(
  File "/home/cloud-user/.local/lib/python3.9/site-packages/rpm_lockfile/__init__.py", line 158, in resolver
    raise RuntimeError(f"DNF error: {exc}")
RuntimeError: DNF error: Problems in request:
missing packages: podman, golang, jq

lubomir · 2024-10-21T08:12:08Z

Are you saying that such an issue could be resolved if I specify the fedora version to be used from this command podman run --rm -v "$PWD:$PWD:z" -w "$PWD" fedora cp -r /etc/yum.repos.d/. . ?

Nope, I meant replacing the metalink option in the repo file with a baseurl. The URLs I provided would work for 41 and Rawhide, but I don't have one for 40. Alternatively you could put this into the input yaml file (instead of the current section):

contentOrigin:
  repos:
    - repoid: fedora
      baseurl: https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-41/compose/Everything/$basearch/os

cmoulliard · 2024-10-21T12:18:30Z

That works using the following config

cat rpms.in.yaml
contentOrigin:
  repos:
    - repoid: fedora
      baseurl: https://kojipkgs.fedoraproject.org/compose/40/latest-Fedora-40/compose/Everything/x86_64/os/
packages:
  - golang
  - podman
  - jq
  - gettext
  - 
...
rpm-lockfile-prototype -f Containerfile rpms.in.yaml --outfile rpms.lock.yaml
Failed to set locale, defaulting to C.UTF-8
INFO:root:Running solver for x86_64
INFO:root:$ skopeo --override-arch=amd64 copy docker://fedora dir:/tmp/tmpvqc3xp1l
Getting image source signatures
Copying blob f7bb57d05c2a done   |
Copying config eca85b75dc done   |
Writing manifest to image destination
INFO:root:Extracting rpmdb from layer sha256:f7bb57d05c2a01f21aeee5873ebc9afe04304b3f5ea7b4c47e0ce18ac2d38124
INFO:dnf:Added fedora repo from https://kojipkgs.fedoraproject.org/compose/40/latest-Fedora-40/compose/Everything/x86_64/os/
INFO:dnf:Last metadata expiration check: 0:00:21 ago on Mon Oct 21 07:00:31 2024.
WARNING:root:No sources found for perl-IPC-Open3-1.22-506.fc40.noarch
WARNING:root:No sources found for shadow-utils-subid-2:4.15.1-1.fc40.x86_64
WARNING:root:No sources found for perl-Pod-Escapes-1:1.07-503.fc40.noarch
WARNING:root:No sources found for nftables-1:1.0.9-3.fc40.x86_64
WARNING:root:No sources found for libutempter-1.2.1-13.fc40.x86_64
WARNING:root:No sources found for catatonit-0.1.7-22.fc40.x86_64
WARNING:root:No sources found for libmd-1.1.0-4.fc40.x86_64
WARNING:root:No sources found for perl-Pod-Perldoc-3.28.01-503.fc40.noarch
WARNING:root:No sources found for perl-Pod-Simple-1:3.45-6.fc40.noarch
...

cmoulliard · 2024-10-21T12:34:05Z

Hmmmm. I got another problem during image build as cachi2 reports such an error

Adding /tmp/cachi2/output/deps/rpm/x86_64/repos.d/cachi2.repo to fetched.repos.d
Adding the entitlement to the build
[1/2] STEP 1/14: FROM fedora AS builder
[1/2] STEP 2/14: RUN . /cachi2/cachi2.env &&     dnf -y install golang
Repository 'fedora' is missing name in configuration, using id.
fedora                                           13 MB/s |  62 kB     00:00    
Error: 
 Problem: package golang-1.22.1-1.fc40.x86_64 from fedora requires golang-bin = 1.22.1-1.fc40, but none of the providers can be installed
  - package golang-bin-1.22.1-1.fc40.x86_64 from fedora requires gcc, but none of the providers can be installed
  - conflicting requests
  - nothing provides libgomp = 14.0.1-0.15.fc40 needed by gcc-14.0.1-0.15.fc40.x86_64 from fedora
(try to add '--skip-broken' to skip uninstallable packages)
subprocess exited with status 1
subprocess exited with status 1

lubomir · 2024-10-21T15:11:18Z

The URL for Fedora 40 repo points to the gold compose used at GA time back in April. It contains gcc-14.0.1-0.15.fc40.x86_64 which requires libgomp-14.0.1-0.15.fc40.
The container image has been updated since, and includes libgomp-14.1.1-4.fc40.x86_64.
The installation basically requires a downgrade.
There's a bug in the lockfile generation that somehow omits the downgraded packages from the result.

lubomir · 2024-10-21T15:32:14Z

Maybe it's not a bug in the tool. It seems that the problem is using the bare fedora as base image. In the build process podman is interpreting it as registry.fedoraproject.org/fedora, but skopeo used by the resolver turns it into docker.io/library/fedora.

The prefetch and build works for me with your yaml and this containerfile:

FROM registry.fedoraproject.org/fedora:40
RUN dnf install -y golang podman jq gettext

cmoulliard · 2024-10-21T16:40:08Z

That fails when Tekton prefetch pod download the bits using fedora:40 too

rpm-lockfile-prototype -f Containerfile rpms.in.yaml --outfile rpms.lock.yaml

where

cat rpms.in.yaml
contentOrigin:
  repos:
    - repoid: fedora
      baseurl: https://kojipkgs.fedoraproject.org/compose/40/latest-Fedora-40/compose/Everything/x86_64/os/
packages:
  - golang
  - podman
  - jq
  - gettext
  
 and 
 
 cat rpms.lock.yaml | grep golang
  - url: https://kojipkgs.fedoraproject.org/compose/40/latest-Fedora-40/compose/Everything/x86_64/os/Packages/g/golang-1.22.1-1.fc40.x86_64.rpm
    name: golang
    sourcerpm: golang-1.22.1-1.fc40.src.rpm
  - url: https://kojipkgs.fedoraproject.org/compose/40/latest-Fedora-40/compose/Everything/x86_64/os/Packages/g/golang-bin-1.22.1-1.fc40.x86_64.rpm
    name: golang-bin
    sourcerpm: golang-1.22.1-1.fc40.src.rpm
  - url: https://kojipkgs.fedoraproject.org/compose/40/latest-Fedora-40/compose/Everything/x86_64/os/Packages/g/golang-src-1.22.1-1.fc40.noarch.rpm
    name: golang-src
    sourcerpm: golang-1.22.1-1.fc40.src.rpm
    
BUT BUT     


[build] Trying to pull registry.fedoraproject.org/fedora:40...
[build] Getting image source signatures
[build] Copying blob sha256:311336741819859e92064b81199a47397788d736d764f561a89cba9583600387
[build] Copying config sha256:8bf3872d7acb45c472225350d631e1bf091ea3b1e1f466521e78102f7deea864
[build] Writing manifest to image destination
[build] 8bf3872d7acb45c472225350d631e1bf091ea3b1e1f466521e78102f7deea864
[build] 8bf3872d7acb45c472225350d631e1bf091ea3b1e1f466521e78102f7deea864
[build] Build will be executed with network isolation
[build] Prefetched content will be made available
[build] Adding /tmp/cachi2/output/deps/rpm/x86_64/repos.d/cachi2.repo to fetched.repos.d
[build] Adding the entitlement to the build
[build] [1/2] STEP 1/14: FROM fedora:40 AS builder
[build] [1/2] STEP 2/14: RUN . /cachi2/cachi2.env &&     dnf -y install golang
[build] Repository 'fedora' is missing name in configuration, using id.
[build] fedora                                           12 MB/s |  62 kB     00:00  
[build] Error: 
[build]  Problem: package golang-1.22.1-1.fc40.x86_64 from fedora requires golang-bin = 1.22.1-1.fc40, but none of the providers can be installed
[build]   - package golang-bin-1.22.1-1.fc40.x86_64 from fedora requires gcc, but none of the providers can be installed
[build]   - conflicting requests
[build]   - nothing provides libgomp = 14.0.1-0.15.fc40 needed by gcc-14.0.1-0.15.fc40.x86_64 from fedora
[build] (try to add '--skip-broken' to skip uninstallable packages)
[build] subprocess exited with status 1
[build] subprocess exited with status 1
[build] Error: building at STEP "RUN . /cachi2/cachi2.env &&     dnf -y install golang": exit status 1

cmoulliard · 2024-10-21T17:01:24Z

Using [2/2] STEP 1/13: FROM registry.fedoraproject.org/fedora:40 fixed the issue

If user specifies a base image with no registry information, things may seem to work as skopeo will use some default value. But it may lead to confusing breakage later if the build system picks a different registry. For example, right now using `fedora` as image spec will make skopeo download `docker.io/library/fedora:40`, but podman uses `registry.fedoraproject.org/fedora:40`. There's no good way to automatically fix it, so let's print a big warning at least. Relates: #67

lubomir · 2024-10-22T06:58:11Z

Great to hear that.

To summarize, we have found two potential issues here:

Mirrorlist and metalink isn't very reliable here. It works great for bare dnf, so maybe the tool is doing something wrong.
Using image name without registry can lead to confusing errors. Warn about images with no registry #68 should help there by printing a big warning.

If user specifies a base image with no registry information, things may seem to work as skopeo will use some default value. But it may lead to confusing breakage later if the build system picks a different registry. For example, right now using `fedora` as image spec will make skopeo download `docker.io/library/fedora:40`, but podman uses `registry.fedoraproject.org/fedora:40`. There's no good way to automatically fix it, so let's print a big warning at least. Relates: #67

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RuntimeError: DNF error: Problems in request: #67

RuntimeError: DNF error: Problems in request: #67

cmoulliard commented Oct 18, 2024

lubomir commented Oct 21, 2024

cmoulliard commented Oct 21, 2024

cmoulliard commented Oct 21, 2024

cmoulliard commented Oct 21, 2024

lubomir commented Oct 21, 2024

cmoulliard commented Oct 21, 2024

cmoulliard commented Oct 21, 2024

lubomir commented Oct 21, 2024

lubomir commented Oct 21, 2024

cmoulliard commented Oct 21, 2024 •

edited

Loading

cmoulliard commented Oct 21, 2024

lubomir commented Oct 22, 2024

RuntimeError: DNF error: Problems in request: #67

RuntimeError: DNF error: Problems in request: #67

Comments

cmoulliard commented Oct 18, 2024

Issue

lubomir commented Oct 21, 2024

cmoulliard commented Oct 21, 2024

cmoulliard commented Oct 21, 2024

cmoulliard commented Oct 21, 2024

lubomir commented Oct 21, 2024

cmoulliard commented Oct 21, 2024

cmoulliard commented Oct 21, 2024

lubomir commented Oct 21, 2024

lubomir commented Oct 21, 2024

cmoulliard commented Oct 21, 2024 • edited Loading

cmoulliard commented Oct 21, 2024

lubomir commented Oct 22, 2024

cmoulliard commented Oct 21, 2024 •

edited

Loading