Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rhel10: move os package sets to yaml files #1104

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

rhel10: move os package sets to yaml files #1104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
33d46bf
spec: introduce a new declative way to define image properties
ondrejbudai Dec 11, 2024
2cdfdbf
rhel10: convert os package sets to image specs
ondrejbudai Dec 11, 2024
2214543
rhel: allow overrdiding the spec directory
ondrejbudai Dec 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions cmd/spec-flatten/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
package main

import (
"os"
"path"

"github.com/osbuild/images/pkg/spec"
)

func main() {
dir := os.DirFS(".")
path := path.Clean(os.Args[1])
merged, err := spec.MergeConfig(dir, path)
if err != nil {
panic(err)
}

os.Stdout.Write(merged)
}
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ require (
golang.org/x/tools v0.24.0
google.golang.org/api v0.201.0
gopkg.in/ini.v1 v1.67.0
gopkg.in/yaml.v3 v3.0.1
)

require (
Expand Down Expand Up @@ -180,5 +181,4 @@ require (
google.golang.org/grpc v1.67.1 // indirect
google.golang.org/grpc/stats/opentelemetry v0.0.0-20240907200651-3ffb98b2c93a // indirect
google.golang.org/protobuf v1.35.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
47 changes: 47 additions & 0 deletions pkg/distro/rhel/imagetype.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
package rhel

import (
"bytes"
"fmt"
"io/fs"
"math/rand"
"os"

"slices"

Expand All @@ -18,6 +21,9 @@ import (
"github.com/osbuild/images/pkg/osbuild"
"github.com/osbuild/images/pkg/platform"
"github.com/osbuild/images/pkg/rpmmd"
"github.com/osbuild/images/pkg/spec"
"github.com/osbuild/images/specs"
"gopkg.in/yaml.v3"
)

const (
Expand Down Expand Up @@ -284,6 +290,47 @@ func (t *ImageType) Manifest(bp *blueprint.Blueprint,
staticPackageSets[name] = getter(t)
}

var itSpec struct {
Spec struct {
Packages []string `yaml:"packages"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe:

Packages struct {
   Include []string
   Exclude []string
}

it would be a deeper nesting but also more symmetric? Plus it seems most image types have some excludes (even tar!)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to clarify: the current merge rules and their implementation merge lists just for top-level keys, so moving Include and Exclude under Packages would break everything. However, we can alter the merge rules:

Dictionaries:
If both values are dictionaries, perform a recursive merge:
For each key:
If the key exists in both, merge the corresponding values.
If the key exists in only one, use that value.

Lists:
If both values are lists, merge them:
Append items from the second list to the first.
Deduplicate items, maintaining all unique entries.

Scalars (Strings, Numbers, Booleans, Null):
If both values are scalars, the value from the second structure overrides the first.

Type Mismatch:
If the types of values for a given key differ, the merge is not allowed. This results in an error.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The question is: are these rules a good default? Are there easy to reason about? I have no good answers.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha, sorry, we can start with the exiting top-level logic, right now the format is internal and as long as we do not have external image defintions we can always refactor. So +1 with starting from that (even though I think we should get something better soon(ish)).

ExcludePackages []string `yaml:"exclude_packages"`
} `yaml:"spec"`
}

// Allow overriding the embedded image type specs with an external directory
// This is of course absolutely misplaces, and should not be passed in as an
// environment variable, but it's good enough for a showcase.
// TODO: Make this more sane
var configDir fs.ReadDirFS
configDir = specs.Data
if overrideConfigDir, found := os.LookupEnv("OSBUILD_IMAGES_IMAGE_SPECS_DIR"); found {
configDir = os.DirFS(overrideConfigDir).(fs.ReadDirFS)
}

configFile, err := spec.FindBestConfig(configDir, t.arch.Distro().Name(), t.Arch().Name(), t.Name())
if err != nil {
return nil, nil, fmt.Errorf("failed to find image type spec: %w", err)
}

rawSpec, err := spec.MergeConfig(configDir, configFile)
if err != nil {
return nil, nil, fmt.Errorf("failed to merge image type spec: %w", err)
}

decoder := yaml.NewDecoder(bytes.NewReader(rawSpec))
decoder.KnownFields(true)

if err := decoder.Decode(&itSpec); err != nil {
return nil, nil, fmt.Errorf("failed to unmarshal image type spec: %w", err)
}

if len(itSpec.Spec.Packages) > 0 {
staticPackageSets[OSPkgsKey] = rpmmd.PackageSet{
Include: itSpec.Spec.Packages,
Exclude: itSpec.Spec.ExcludePackages,
}
}

// amend with repository information and collect payload repos
payloadRepos := make([]rpmmd.RepoConfig, 0)
for _, repo := range repos {
Expand Down
71 changes: 3 additions & 68 deletions pkg/distro/rhel/rhel10/ami.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -293,80 +293,15 @@ func defaultEc2ImageConfigX86_64() *distro.ImageConfig {
// PACKAGE SETS

func ec2PackageSet(t *rhel.ImageType) rpmmd.PackageSet {
ps := rpmmd.PackageSet{
Include: []string{
"@core",
"chrony",
"cloud-init",
"cloud-utils-growpart",
"dhcpcd",
"yum-utils",
"dracut-config-generic",
"grub2",
"langpacks-en",
"NetworkManager-cloud-setup",
"redhat-release",
"redhat-release-eula",
"rsync",
"tuned",
"tar",
},
Exclude: []string{
"aic94xx-firmware",
"alsa-firmware",
"alsa-tools-firmware",
"biosdevname",
"firewalld",
"iprutils",
"ivtv-firmware",
"iwl1000-firmware",
"iwl100-firmware",
"iwl105-firmware",
"iwl135-firmware",
"iwl2000-firmware",
"iwl2030-firmware",
"iwl3160-firmware",
"iwl3945-firmware",
"iwl4965-firmware",
"iwl5000-firmware",
"iwl5150-firmware",
"iwl6000-firmware",
"iwl6000g2a-firmware",
"iwl6000g2b-firmware",
"iwl6050-firmware",
"iwl7260-firmware",
"libertas-sd8686-firmware",
"libertas-sd8787-firmware",
"libertas-usb8388-firmware",
"plymouth",
// RHBZ#2064087
"dracut-config-rescue",
// RHBZ#2075815
"qemu-guest-agent",
},
}.Append(distroSpecificPackageSet(t))

return ps
return rpmmd.PackageSet{}
}

// rhel-ha-ec2 image package set
func rhelEc2HaPackageSet(t *rhel.ImageType) rpmmd.PackageSet {
ec2HaPackageSet := ec2PackageSet(t)
ec2HaPackageSet = ec2HaPackageSet.Append(rpmmd.PackageSet{
Include: []string{
"fence-agents-all",
"pacemaker",
"pcs",
},
})
return ec2HaPackageSet
return rpmmd.PackageSet{}
}

// rhel-sap-ec2 image package set
func rhelEc2SapPackageSet(t *rhel.ImageType) rpmmd.PackageSet {
return rpmmd.PackageSet{
Include: []string{
//"libcanberra-gtk2", // libcanberra-gtk2 is not available in RHEL-10
},
}.Append(ec2PackageSet(t)).Append(SapPackageSet(t))
return rpmmd.PackageSet{}
}
78 changes: 2 additions & 76 deletions pkg/distro/rhel/rhel10/azure.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,86 +88,12 @@ func mkAzureSapInternalImgType(rd *rhel.Distribution) *rhel.ImageType {

// Common Azure image package set
func azureCommonPackageSet(t *rhel.ImageType) rpmmd.PackageSet {
ps := rpmmd.PackageSet{
Include: []string{
"@Server",
"bzip2",
"cloud-init",
"cloud-utils-growpart",
"dracut-config-generic",
"efibootmgr",
"hyperv-daemons",
"kernel-core",
"kernel-modules",
"kernel",
"langpacks-en",
"lvm2",
"NetworkManager",
"NetworkManager-cloud-setup",
"nvme-cli",
"patch",
"rng-tools",
"selinux-policy-targeted",
"uuid",
"WALinuxAgent",
"yum-utils",
},
Exclude: []string{
"aic94xx-firmware",
"alsa-firmware",
"alsa-lib",
"alsa-sof-firmware",
"alsa-tools-firmware",
"biosdevname",
"bolt",
"buildah",
"cockpit-podman",
"containernetworking-plugins",
"dnf-plugin-spacewalk",
"dracut-config-rescue",
"glibc-all-langpacks",
"iprutils",
"ivtv-firmware",
"iwl100-firmware",
"iwl1000-firmware",
"iwl105-firmware",
"iwl135-firmware",
"iwl2000-firmware",
"iwl2030-firmware",
"iwl3160-firmware",
"iwl3945-firmware",
"iwl4965-firmware",
"iwl5000-firmware",
"iwl5150-firmware",
"iwl6000-firmware",
"iwl6000g2a-firmware",
"iwl6000g2b-firmware",
"iwl6050-firmware",
"iwl7260-firmware",
"libertas-sd8686-firmware",
"libertas-sd8787-firmware",
"libertas-usb8388-firmware",
"NetworkManager-config-server",
"plymouth",
"podman",
"python3-dnf-plugin-spacewalk",
"python3-hwdata",
"python3-rhnlib",
"rhn-check",
"rhn-client-tools",
"rhn-setup",
"rhnlib",
"rhnsd",
"usb_modeswitch",
},
}.Append(distroSpecificPackageSet(t))

return ps
return rpmmd.PackageSet{}
}

// Azure BYOS image package set
func azurePackageSet(t *rhel.ImageType) rpmmd.PackageSet {
return azureCommonPackageSet(t)
return rpmmd.PackageSet{}
}

// Azure SAP image package set
Expand Down
61 changes: 2 additions & 59 deletions pkg/distro/rhel/rhel10/bare_metal.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,7 @@ func mkTarImgType() *rhel.ImageType {
"root.tar.xz",
"application/x-tar",
map[string]rhel.PackageSetFunc{
rhel.OSPkgsKey: func(t *rhel.ImageType) rpmmd.PackageSet {
return rpmmd.PackageSet{
Include: []string{"policycoreutils", "selinux-policy-targeted"},
Exclude: []string{"rng-tools"},
}
},
rhel.OSPkgsKey: func(t *rhel.ImageType) rpmmd.PackageSet { return rpmmd.PackageSet{} },
},
rhel.TarImage,
[]string{"build"},
Expand Down Expand Up @@ -66,59 +61,7 @@ func mkImageInstallerImgType() *rhel.ImageType {
// PACKAGE SETS

func bareMetalPackageSet(t *rhel.ImageType) rpmmd.PackageSet {
ps := rpmmd.PackageSet{
Include: []string{
"@core",
"chrony",
"cockpit-system",
"cockpit-ws",
"dnf-utils",
"dosfstools",
"firewalld",
"iwl1000-firmware",
"iwl100-firmware",
"iwl105-firmware",
"iwl135-firmware",
"iwl2000-firmware",
"iwl2030-firmware",
"iwl3160-firmware",
"iwl5000-firmware",
"iwl5150-firmware",
"iwl6000g2a-firmware",
"iwl6000g2b-firmware",
"iwl6050-firmware",
"iwl7260-firmware",
"lvm2",
"net-tools",
"nfs-utils",
"oddjob",
"oddjob-mkhomedir",
"policycoreutils",
"psmisc",
"python3-jsonschema",
"qemu-guest-agent",
"redhat-release",
"redhat-release-eula",
"rsync",
"tar",
"tcpdump",
"tuned",
},
Exclude: []string{
"dracut-config-rescue",
},
}.Append(distroBuildPackageSet(t))

// Ensure to not pull in subscription-manager on non-RHEL distro
if t.IsRHEL() {
ps = ps.Append(rpmmd.PackageSet{
Include: []string{
"subscription-manager-cockpit",
},
})
}

return ps
return rpmmd.PackageSet{}
}

func installerPackageSet(t *rhel.ImageType) rpmmd.PackageSet {
Expand Down
Loading
Loading