Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[1.19] copier: add GetOptions.IgnoreUnreadable #3063

Closed
wants to merge 64 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
acc1a70
Cherry pick localhost fix and update CI configuration for release-1.1…
TomSweeneyRedHat Jan 14, 2021
0acbb37
Merge pull request #2910 from TomSweeneyRedHat/dev/tsweeney/picknew
openshift-merge-robot Jan 14, 2021
8462f42
Vendor in common 0.33.1
ashley-cui Jan 13, 2021
ae2b8d6
If destination does not exists, do not throw error
rhatdan Jan 15, 2021
7b83c4b
Bump to v1.19.2
rhatdan Jan 15, 2021
ae20f06
Merge pull request #2916 from rhatdan/1.19.2
openshift-merge-robot Jan 15, 2021
7323e12
Pick default OCI Runtime from containers.conf
rhatdan Jan 21, 2021
c868696
Use build-arg ENV val from local environment if set
umohnani8 Jan 22, 2021
2c6ecfe
Merge pull request #2932 from umohnani8/1.19
openshift-merge-robot Jan 25, 2021
4ed72e6
Merge pull request #2929 from rhatdan/release-1.19
openshift-merge-robot Jan 25, 2021
882be3c
local image lookup by digest
vrothberg Jan 25, 2021
4a532b6
Merge pull request #2934 from rhatdan/release-1.19
openshift-merge-robot Jan 26, 2021
e6f8b32
Bump golang.org/x/crypto to latest rel-1.19
TomSweeneyRedHat Jan 27, 2021
99cbd35
Merge pull request #2941 from containers/bump_crypto_rel1.19
openshift-merge-robot Jan 27, 2021
7b0b384
Rebuild layer if a change in ARG is detected
umohnani8 Jan 26, 2021
7c0a2bb
Vendor in containers/image v5.10.1
rhatdan Jan 28, 2021
8a8c080
Merge pull request #2945 from umohnani8/1.19
openshift-merge-robot Jan 28, 2021
ddedd08
[ci:docs] Fix man page for buildah push
rhatdan Jan 28, 2021
c40abcb
Merge pull request #2946 from rhatdan/release-1.19
openshift-merge-robot Jan 28, 2021
06e091b
Merge pull request #2947 from rhatdan/docs
openshift-merge-robot Jan 28, 2021
a7afe38
Bump to v1.19.3
TomSweeneyRedHat Jan 28, 2021
af31e45
Merge pull request #2949 from containers/bump_to_v.19.3
rhatdan Jan 29, 2021
7367d00
Fix build arg check
umohnani8 Feb 4, 2021
e0d3669
Merge pull request #2969 from umohnani8/1.19
rhatdan Feb 5, 2021
4602304
buildah manifest add localimage should work
rhatdan Feb 2, 2021
a2d229e
Finish plumbing for buildah bud --manifest
rhatdan Feb 5, 2021
1b43a59
bump containernetworking/cni library to v0.8.1 - fix for CVE-2021-20206
lsm5 Feb 5, 2021
36ed1f9
Merge pull request #2957 from rhatdan/release-1.19
rhatdan Feb 5, 2021
17521db
Merge pull request #2974 from lsm5/release-1.19-cni-bump
openshift-merge-robot Feb 5, 2021
93b9835
run: fix check for host pid namespace
giuseppe Feb 5, 2021
a96b716
Merge pull request #2975 from rhatdan/release-1.19
openshift-merge-robot Feb 6, 2021
b5df9da
--iidfile: print hash prefix
vrothberg Feb 8, 2021
679b71f
Merge pull request #2987 from vrothberg/1.19-iidfile
openshift-merge-robot Feb 8, 2021
175c3e8
Bump to v1.19.4
rhatdan Feb 6, 2021
76beccc
Merge pull request #2976 from rhatdan/bump-1.19.4
rhatdan Feb 8, 2021
3df9787
Vendor in latest containers/image and common
rhatdan Feb 9, 2021
0bc5bfd
Bump to v1.19.5
rhatdan Feb 9, 2021
1fb260f
Merge pull request #2991 from rhatdan/release-1.19
openshift-merge-robot Feb 10, 2021
d8e19a1
Fix caching layers with build args
terceiro Feb 10, 2021
db783f4
Merge pull request #2998 from umohnani8/1.19
openshift-merge-robot Feb 11, 2021
e862003
Stop overriding the location of the blob info cache
nalind Jan 12, 2021
9dd415b
Merge pull request #2999 from rhatdan/blob
openshift-merge-robot Feb 12, 2021
0a525bb
Fix bud capabilities test
cevich Jan 7, 2021
3e7918f
Fix config-flags-verification test on F33
cevich Jan 7, 2021
b55a038
Workaround for RHEL gating test failure
edsantiago Jan 20, 2021
e5384ed
Merge pull request #3014 from cevich/bz1914884_backport
openshift-merge-robot Feb 15, 2021
d37d51b
Don't fail copy to emptydir - v1.19
TomSweeneyRedHat Feb 17, 2021
016b90d
Merge pull request #3016 from containers/dev/tsweeney/emptydir_1.19
openshift-merge-robot Feb 17, 2021
04b7a5f
Bump c/containers/storage v1.24.6
TomSweeneyRedHat Feb 17, 2021
dcd385e
Merge pull request #3018 from containers/dev/tsweeney/store2_v1.24.6
openshift-merge-robot Feb 18, 2021
f938e05
Bulidah v1.19.6
TomSweeneyRedHat Feb 18, 2021
7aedb16
Merge pull request #3022 from containers/buildah_v1.19.6
rhatdan Feb 18, 2021
e5220cb
copier: return child process stderr if we can't JSON decode the response
nalind Feb 16, 2021
fdfd19c
copier: fix a renaming bug
nalind Feb 17, 2021
89348e4
copier: add an Eval function
nalind Mar 2, 2021
43caf54
copier.GetOptions: add NoDerefSymLinks
nalind Feb 17, 2021
05ad94d
ADD/COPY: create the destination directory first, chroot to it
nalind Feb 16, 2021
d51c437
Merge pull request #3052 from nalind/copier-mkdir-first-1.19
openshift-merge-robot Mar 3, 2021
bbbb912
Set upperdir permissions based on source
rhatdan Mar 3, 2021
0a1ca42
Bump to v1.19.7
rhatdan Mar 4, 2021
ca3c296
Merge pull request #3058 from rhatdan/release-1.19
openshift-merge-robot Mar 4, 2021
44a6801
copier: add GetOptions.IgnoreUnreadable
nalind Mar 3, 2021
8815f4d
v1.19.8
vrothberg Mar 5, 2021
544320e
bump to v1.19.9-dev
vrothberg Mar 5, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .cirrus.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ env:
#### Global variables used for all tasks
####
# Name of the ultimate destination branch for this CI run, PR or post-merge.
DEST_BRANCH: "master"
DEST_BRANCH: "release-1.19"
GOPATH: "/var/tmp/go"
GOSRC: "${GOPATH}/src/github.com/containers/buildah"
# Overrides default location (/tmp/cirrus) for repo clone
Expand Down
38 changes: 38 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,44 @@

# Changelog

## v1.19.6 (2021-02-18)
Bump c/containers/storage v1.24.6
Don't fail copy to emptydir
Workaround for RHEL gating test failure
Fix config-flags-verification test on F33
Fix bud capabilities test
Stop overriding the location of the blob info cache
Fix caching layers with build args

## v1.19.5 (2021-02-09)
Vendor in latest containers/image and common

## v1.19.4 (2021-02-06)
run: fix check for host pid namespace
bump containernetworking/cni library to v0.8.1 - fix for CVE-2021-20206
Finish plumbing for buildah bud --manifest
buildah manifest add localimage should work
Fix build arg check
--iidfile: print hash prefix

## v1.19.3 (2021-01-28)
[ci:docs] Fix man page for buildah push
Vendor in containers/image v5.10.1
Rebuild layer if a change in ARG is detected
Bump golang.org/x/crypto to latest rel-1.19
local image lookup by digest
Use build-arg ENV val from local environment if set
Pick default OCI Runtime from containers.conf

## v1.19.2 (2021-01-15)
If overlay mount point destination does not exists, do not throw error
Vendor in containers/common

## v1.19.1 (2021-01-14)
Cherry pick localhost fix and update CI configuration for release-1.19
use local image name for pull policy checks
Vendor in common 0.33.1

## v1.19.0 (2021-01-08)
Update vendor of containers/storage and containers/common
Buildah inspect should be able to inspect manifests
Expand Down
38 changes: 35 additions & 3 deletions add.go
Original file line number Diff line number Diff line change
Expand Up @@ -303,6 +303,13 @@ func (b *Builder) Add(destination string, extract bool, options AddAndCopyOption
renameTarget = filepath.Base(extractDirectory)
extractDirectory = filepath.Dir(extractDirectory)
}

// if the destination is a directory that doesn't yet exist, let's copy it.
newDestDirFound := false
if (len(destStats) == 1 || len(destStats[0].Globbed) == 0) && destMustBeDirectory && !destCanBeFile {
newDestDirFound = true
}

if len(destStats) == 1 && len(destStats[0].Globbed) == 1 && destStats[0].Results[destStats[0].Globbed[0]].IsRegular {
if destMustBeDirectory {
return errors.Errorf("destination %v already exists but is not a directory", destination)
Expand All @@ -317,13 +324,33 @@ func (b *Builder) Add(destination string, extract bool, options AddAndCopyOption
return errors.Wrapf(err, "error processing excludes list %v", options.Excludes)
}

// Copy each source in turn.
// Make sure that, if it's a symlink, we'll chroot to the target of the link;
// knowing that target requires that we resolve it within the chroot.
evalOptions := copier.EvalOptions{}
evaluated, err := copier.Eval(mountPoint, extractDirectory, evalOptions)
if err != nil {
return errors.Wrapf(err, "error checking on destination %v", extractDirectory)
}
extractDirectory = evaluated

// Set up ID maps.
var srcUIDMap, srcGIDMap []idtools.IDMap
if options.IDMappingOptions != nil {
srcUIDMap, srcGIDMap = convertRuntimeIDMaps(options.IDMappingOptions.UIDMap, options.IDMappingOptions.GIDMap)
}
destUIDMap, destGIDMap := convertRuntimeIDMaps(b.IDMappingOptions.UIDMap, b.IDMappingOptions.GIDMap)

// Create the target directory if it doesn't exist yet.
mkdirOptions := copier.MkdirOptions{
UIDMap: destUIDMap,
GIDMap: destGIDMap,
ChownNew: chownDirs,
}
if err := copier.Mkdir(mountPoint, extractDirectory, mkdirOptions); err != nil {
return errors.Wrapf(err, "error ensuring target directory exists")
}

// Copy each source in turn.
for _, src := range sources {
var multiErr *multierror.Error
var getErr, closeErr, renameErr, putErr error
Expand Down Expand Up @@ -356,7 +383,7 @@ func (b *Builder) Add(destination string, extract bool, options AddAndCopyOption
ChmodFiles: nil,
IgnoreDevices: rsystem.RunningInUserNS(),
}
putErr = copier.Put(mountPoint, extractDirectory, putOptions, io.TeeReader(pipeReader, hasher))
putErr = copier.Put(extractDirectory, extractDirectory, putOptions, io.TeeReader(pipeReader, hasher))
}
hashCloser.Close()
pipeReader.Close()
Expand Down Expand Up @@ -415,6 +442,11 @@ func (b *Builder) Add(destination string, extract bool, options AddAndCopyOption
if !globInfo.IsDir || !includeDirectoryAnyway(rel, pm) {
continue
}
} else {
// if the destination is a directory that doesn't yet exist, and is not excluded, let's copy it.
if newDestDirFound {
itemsCopied++
}
}
} else {
// Make sure we don't trigger a "copied nothing" error for an empty context
Expand Down Expand Up @@ -486,7 +518,7 @@ func (b *Builder) Add(destination string, extract bool, options AddAndCopyOption
ChmodFiles: nil,
IgnoreDevices: rsystem.RunningInUserNS(),
}
putErr = copier.Put(mountPoint, extractDirectory, putOptions, io.TeeReader(pipeReader, hasher))
putErr = copier.Put(extractDirectory, extractDirectory, putOptions, io.TeeReader(pipeReader, hasher))
}
hashCloser.Close()
pipeReader.Close()
Expand Down
2 changes: 1 addition & 1 deletion buildah.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ const (
Package = "buildah"
// Version for the Package. Bump version in contrib/rpm/buildah.spec
// too.
Version = "1.19.0"
Version = "1.19.9-dev"
// The value we use to identify what type of information, currently a
// serialized Builder structure, we are using as per-container state.
// This should only be changed when we make incompatible changes to
Expand Down
39 changes: 39 additions & 0 deletions changelog.txt
Original file line number Diff line number Diff line change
@@ -1,3 +1,42 @@

- Changelog for v1.19.6 (2021-02-18)
* Bump c/containers/storage v1.24.6
* Don't fail copy to emptydir
* Workaround for RHEL gating test failure
* Fix config-flags-verification test on F33
* Fix bud capabilities test
* Stop overriding the location of the blob info cache
* Fix caching layers with build args

- Changelog for v1.19.5 (2021-02-09)
* Vendor in latest containers/image and common

- Changelog for v1.19.4 (2021-02-06)
* run: fix check for host pid namespace
* bump containernetworking/cni library to v0.8.1 - fix for CVE-2021-20206
* Finish plumbing for buildah bud --manifest
* buildah manifest add localimage should work
* Fix build arg check
* --iidfile: print hash prefix

- Changelog for v1.19.3 (2021-01-28)
* [ci:docs] Fix man page for buildah push
* Vendor in containers/image v5.10.1
* Rebuild layer if a change in ARG is detected
* Bump golang.org/x/crypto to latest rel-1.19
* local image lookup by digest
* Use build-arg ENV val from local environment if set
* Pick default OCI Runtime from containers.conf

- Changelog for v1.19.2 (2021-01-15)
* If overlay mount point destination does not exists, do not throw error
* Vendor in containers/common

- Changelog for v1.19.1 (2021-01-14)
* Cherry pick localhost fix and update CI configuration for release-1.19
* use local image name for pull policy checks
* Vendor in common 0.33.1

- Changelog for v1.19.0 (2021-01-08)
* Update vendor of containers/storage and containers/common
* Buildah inspect should be able to inspect manifests
Expand Down
7 changes: 6 additions & 1 deletion cmd/buildah/bud.go
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,12 @@ func budCmd(c *cobra.Command, inputArgs []string, iopts budOptions) error {
if len(av) > 1 {
args[av[0]] = av[1]
} else {
delete(args, av[0])
// check if the env is set in the local environment and use that value if it is
if val, present := os.LookupEnv(av[0]); present {
args[av[0]] = val
} else {
delete(args, av[0])
}
}
}
}
Expand Down
10 changes: 9 additions & 1 deletion cmd/buildah/manifest.go
Original file line number Diff line number Diff line change
Expand Up @@ -307,7 +307,15 @@ func manifestAddCmd(c *cobra.Command, args []string, opts manifestAddOpts) error

digest, err := list.Add(getContext(), systemContext, ref, opts.all)
if err != nil {
return err
var storeErr error
// check if the local image exists
if ref, _, storeErr = util.FindImage(store, "", systemContext, imageSpec); storeErr != nil {
return err
}
digest, storeErr = list.Add(getContext(), systemContext, ref, opts.all)
if storeErr != nil {
return err
}
}

if opts.os != "" {
Expand Down
22 changes: 11 additions & 11 deletions commit.go
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,7 @@ func checkRegistrySourcesAllows(forWhat string, dest types.ImageReference) (inse
return false, nil
}

func (b *Builder) addManifest(ctx context.Context, manifestName string, imageSpec string) error {
func (b *Builder) addManifest(ctx context.Context, manifestName string, imageSpec string) (string, error) {
var create bool
systemContext := &types.SystemContext{}
var list manifests.List
Expand All @@ -235,38 +235,35 @@ func (b *Builder) addManifest(ctx context.Context, manifestName string, imageSpe
} else {
_, list, err = manifests.LoadFromImage(b.store, listImage.ID)
if err != nil {
return err
return "", err
}
}

names, err := util.ExpandNames([]string{manifestName}, "", systemContext, b.store)
if err != nil {
return errors.Wrapf(err, "error encountered while expanding image name %q", manifestName)
return "", errors.Wrapf(err, "error encountered while expanding image name %q", manifestName)
}

ref, err := alltransports.ParseImageName(imageSpec)
if err != nil {
if ref, err = alltransports.ParseImageName(util.DefaultTransport + imageSpec); err != nil {
// check if the local image exists
if ref, _, err = util.FindImage(b.store, "", systemContext, imageSpec); err != nil {
return err
return "", err
}
}
}

if _, err = list.Add(ctx, systemContext, ref, true); err != nil {
return err
return "", err
}
var imageID string
if create {
imageID, err = list.SaveToImage(b.store, "", names, manifest.DockerV2ListMediaType)
} else {
imageID, err = list.SaveToImage(b.store, listImage.ID, nil, "")
}
if err == nil {
fmt.Printf("%s\n", imageID)
}
return err
return imageID, err
}

// Commit writes the contents of the container, along with its updated
Expand Down Expand Up @@ -469,7 +466,7 @@ func (b *Builder) Commit(ctx context.Context, dest types.ImageReference, options
dest = dest2
}
if options.IIDFile != "" {
if err = ioutil.WriteFile(options.IIDFile, []byte(img.ID), 0644); err != nil {
if err = ioutil.WriteFile(options.IIDFile, []byte("sha256:"+img.ID), 0644); err != nil {
return imgID, nil, "", err
}
}
Expand All @@ -489,9 +486,12 @@ func (b *Builder) Commit(ctx context.Context, dest types.ImageReference, options
}

if options.Manifest != "" {
if err := b.addManifest(ctx, options.Manifest, imgID); err != nil {
manifestID, err := b.addManifest(ctx, options.Manifest, imgID)
if err != nil {
return imgID, nil, "", err
}
logrus.Debugf("added imgID %s to manifestID %s", imgID, manifestID)

}
return imgID, ref, manifestDigest, nil
}
Expand Down
3 changes: 0 additions & 3 deletions common.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,6 @@ func getSystemContext(store storage.Store, defaults *types.SystemContext, signat
sc.SignaturePolicyPath = signaturePolicyPath
}
if store != nil {
if sc.BlobInfoCacheDir == "" {
sc.BlobInfoCacheDir = filepath.Join(store.GraphRoot(), "cache")
}
if sc.SystemRegistriesConfPath == "" && unshare.IsRootless() {
userRegistriesFile := filepath.Join(store.GraphRoot(), "registries.conf")
if _, err := os.Stat(userRegistriesFile); err == nil {
Expand Down
43 changes: 42 additions & 1 deletion contrib/rpm/buildah.spec
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@

Name: buildah
# Bump version in buildah.go too
Version: 1.19.0
Version: 1.19.9-dev
Release: 1.git%{shortcommit}%{?dist}
Summary: A command line tool used to creating OCI Images
License: ASL 2.0
Expand Down Expand Up @@ -100,6 +100,47 @@ make DESTDIR=%{buildroot} PREFIX=%{_prefix} install install.completions
%{_datadir}/bash-completion/completions/*

%changelog
* Thu Mar 4, 2021 Dan Walsh <[email protected]> 1.19.7-1
- Set upperdir permissions based on source

* Thu Feb 18, 2021 Tom Sweeney <[email protected]> 1.19.6-1
- Bump c/containers/storage v1.24.6
- Don't fail copy to emptydir
- Workaround for RHEL gating test failure
- Fix config-flags-verification test on F33
- Fix bud capabilities test
- Stop overriding the location of the blob info cache
- Fix caching layers with build args

* Tue Feb 9, 2021 Dan Walsh <[email protected]> 1.19.5-1
- Vendor in latest containers/image and common

* Mon Feb 8, 2021 Dan Walsh <[email protected]> 1.19.4-1
- run: fix check for host pid namespace
- bump containernetworking/cni library to v0.8.1 - fix for CVE-2021-20206
- Finish plumbing for buildah bud --manifest
- buildah manifest add localimage should work
- Fix build arg check
- --iidfile: print hash prefix

* Thu Jan 28, 2021 Tom Sweeney <[email protected]> 1.19.3-1
- [ci:docs] Fix man page for buildah push
- Vendor in containers/image v5.10.1
- Rebuild layer if a change in ARG is detected
- Bump golang.org/x/crypto to latest rel-1.19
- local image lookup by digest
- Use build-arg ENV val from local environment if set
- Pick default OCI Runtime from containers.conf

* Fri Jan 15, 2021 Dan Walsh <[email protected]> 1.19.2-1
- If overlay mount point destination does not exists, do not throw error
- Vendor in containers/common

* Thu Jan 14, 2021 Dan Walsh <[email protected]> 1.19.1-1
- Cherry pick localhost fix and update CI configuration for release-1.19
- use local image name for pull policy checks
- Vendor in common 0.33.1

* Fri Jan 8, 2021 Tom Sweeney <[email protected]> 1.19.0-1
- Update vendor of containers/storage and containers/common
- Buildah inspect should be able to inspect manifests
Expand Down
Loading