podman build tags images in the local registry with a leading localhost/ and docker does not

[bowlofeggs]

Description

Does podman build have the goal of being a drop-in replacement for docker? If not, feel free to close this issue. If so, the issue is that podman build -t bodhi-ci/f27 tags the resulting image into the local engine with a leading localhost/ in the repository name, and docker does not do this. This results in a difference in how the image must be referenced later when running docker/podman run, because docker run… bodhi-ci/f27 will work, but podman run… bodhi-ci/f27 will not work (and will try to find that image in a variety of external registries).

Steps to reproduce the issue:

1. Build a Dockerfile with podman and docker, and use the -t flag.
2. For example, I've built my Bodhi CI images with -t bodhi-ci/f27 in the examples shown below.

Describe the results you received:

$ sudo docker images
REPOSITORY                          TAG                 IMAGE ID            CREATED             SIZE
bodhi-ci/f27                        latest              4668aacb1a2a        11 hours ago        1.29 GB
$ sudo podman images
REPOSITORY                          TAG      IMAGE ID       CREATED          SIZE
localhost/bodhi-ci/f27              latest   351ffa92bc74   2 minutes ago    1.34GB

Describe the results you expected:

$ sudo podman images
REPOSITORY                          TAG      IMAGE ID       CREATED          SIZE
bodhi-ci/f27              latest   351ffa92bc74   2 minutes ago    1.34GB

Output of rpm -q buildah or apt list buildah:

Buildah is not installed, but the text at https://github.com/containers/libpod/issues/new said to file issues with podman build here.

Output of buildah version:

See above.

Output of cat /etc/*release:

$ cat /etc/redhat-release 
Fedora release 30 (Rawhide)

Output of uname -a:

Linux host.example.com 4.19.0-0.rc3.git3.1.fc30.x86_64 #1 SMP Fri Sep 14 18:31:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

# This file is is the configuration file for all tools
# that use the containers/storage library.
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers.
size = ""

# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
#mount_program = "/usr/bin/fuse-overlayfs"

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

# mountopt specifies comma separated list of extra mount options
mountopt = "nodev"

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to UIDs/GIDs as they should appear outside of the container, and
# the length of the range of UIDs/GIDs.  Additional mapped sets can be listed
# and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and the a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,
# until all of the entries have been used for maps.
#
# remap-user = "storage"
# remap-group = "storage"

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper.
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem.
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base.
# device.
# mkfsarg = ""

# use_deferred_removal marks devicemapper block device for deferred removal.
# If the thinpool is in use when the driver attempts to remove it, the driver 
# tells the kernel to remove it as soon as possible. Note this does not free
# up the disk space, use deferred deletion to fully remove the thinpool.
# use_deferred_removal = "True"

# use_deferred_deletion marks thinpool device for deferred deletion.
# If the device is busy when the driver attempts to delete it, the driver
# will attempt to delete device every 30 seconds until successful.
# If the program using the driver exits, the driver will continue attempting
# to cleanup the next time the driver is used. Deferred deletion permanently
# deletes the device and all data stored in device will be lost.
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

# If specified, use OSTree to deduplicate files with the overlay backend
ostree_repo = ""

# Set to skip a PRIVATE bind mount on the storage home directory.  Only supported by
# certain container storage drivers
skip_mount_home = "false"

[TomSweeneyRedHat]

Thanks for the problem report @bowlofeggs . Thanks for the thorough problem report, can I bother you to reply with the output of podman version please?

podman/buildah build/bud should be a drop in replacement for docker and we'll take a look at this. However, I thought this had bubbled up before, @nalind or @rhatdan do you recall? I didn't find it in the issues, but may have missed it.

[bowlofeggs]

Sure thing!

$ podman --version
podman version 0.9.3-dev
$ rpm -q podman
podman-0.9.3-2.dev.gitc3a0874.fc30.x86_64

[rhatdan]

Well it is intended to be a drop in replacement for the most part. The localhost versus docker.io was intentional.

The registry component of an image should indicate where it came from. And we don't want to hard code the default location of docker.io.

Now if podman can not instantly run the image created by buildah or podman build, then that is a bug.

[rhatdan]

It seems like podman run is not looking into localhost if the name containers a repo and a name

@baude could you take a look.

podman build -t -t bodhi-ci/f27  ~
podman run -ti bodhi-ci/f27 /bin/sh

Fails.

podman build -t -t bodhi-ci  ~
podman run -ti bodhi-ci /bin/sh

Works.

Our image name processing is failing if the repo is specified.

[baude]

For me using master, neither work ... ill look at it sure.

[rhatdan]

I believe this is fixed in the latest release.

[adelton]

I'm hitting this issue with podman-0.11.2-15.dev.git75b19ca.fc30.x86_64 installed on Fedora 28 machine. Is the fix supposed to be in Fedora by now?

[rhatdan]

It should be in Master branch, and we are working on a podman-0.12.1 right now.

[jschewebbn]

I'm using podman 1.6.2 and still seeing this issue of images being prefixed with "localhost/". Should this still be the case? This makes compatibility with docker scripts difficult as the image names aren't the same.

[TomSweeneyRedHat]

I'm still seeing it on v1.8.0 too, but thought it had been fixed. I'm going to reopen this so it can be chased down.

[jschewebbn]

@TomSweeneyRedHat do you know of a work around that I can use until this is fixed? Is there a way to tell podman to not prefix the tag with "localhost/"?

[TomSweeneyRedHat]

I tried playing with it, but I didn't find a quick clean way to do so. However, when referencing the image, you can ignore the 'localhost/' part of it. For instance, in the above example you could do:

podman inspect bodhi-ci/f27

and that works, does that make it better for your use case @jschewebbn ?

[jschewebbn]

The problem that I have is that I'm creating images and then saving them to a docker archive file to transfer and the names of the images have "localhost/" prefixed and then when I try and load the images from the file on the other end with a script not expecting the "localhost/" the script fails.

[TomSweeneyRedHat]

@jschewebbn can you tell me the commands that you're using to save the docker archive file and then the command(s) the script is not happy with when trying to load it?

I tried playing with buildah push/pull to a docker-archive and I wasn't seeing the localhost in that output or the inspects that I did. These are the play commands that I used:

buildah push tom docker-archive:/tmp/tom:latest
buildah pull docker-archive:/tmp/tom

[jschewebbn]

This command creates the archive without error:

podman save -o file.tar container_name

Now in another environment docker is used to load the file. This fails because "container_name" doesn't exist, instead "localhost/container_name" exists.

docker load -i file.tar
docker tag container_name ${hostname}:5000/v2/container_name

I also want to be able to save multiple container images in a single archive file and podman doesn't support that right now per containers/podman#2669, so this is all really moot until I have another solution for that problem as well.

[jschewebbn]

The workflow that I've been doing with docker is this:

cd image1
docker build -t image1 .
cd ../image2
docker build -t image2 .
... # create all of the images

rm -f archive.tar
docker save -o archive.tar image1 image2 ...

Copy archive.tar to another host.

docker load -i archive.tar
docker pull registry:2
docker run ... registry:2

for image in image1 image2 ...; do
    docker tag ${image} ${host}:5000/v2/${image}
    docker push ${host}:5000/v2/${image}
done

Perhaps I'm doing some things incorrectly as my docker knowledge is pretty basic.

[sttyerase]

Fedora Rules>>podman images dbmidocker
REPOSITORY                  TAG   IMAGE ID       CREATED         SIZE
localhost/dbmidocker/hfb0   1.0   0378f49b278e   8 minutes ago   638 MB
Fedora Rules>>podman version
Version:            1.8.1
RemoteAPI Version:  1
Go Version:         go1.13.6
OS/Arch:            linux/amd64
Fedora Rules>>cat /etc/redhat-release
Fedora release 31 (Thirty One)
Fedora Rules>>

Same issue here. This seems to create issues with pushing the image to Dockerhub.

[rhatdan]

@vrothberg PTAL, I think this is around the saving with multiple images?

[vrothberg]

@vrothberg PTAL, I think this is around the saving with multiple images?

It's a different issue although it affects save and load as well. Podman and Buildah is prefexing short images (or not fully-qualified images without a domain/registry) with "localhost/". That's inconsistent with Docker's behaviour but consistent among Podman and Buildah.

Personally, I don't like the "localhost/" because the tools don't what I want them to but I also see wisdom explicitly enforcing fully-qualified image names.

I am closing the issue as changing the behaviour will very likely break users who already rely in the "localhost/" prefixes. However, in some cases it might cause issues migrating from Docker to Podman/Buildah.

[jschewebbn]

I take this closing of the ticket to mean that podman does not want to be compatible with docker. This will make it very difficult for those of us advocating people move from docker to podman as they now have an incompatibility that there is no work around for. If podman were to have a command line option to suppress this "localhost/" prefix that would be sufficient. Then I could tell people that podman save works just like docker, except you need this extra flag and it works.

[vrothberg]

@jschewebbn, thanks for continuing the conversation!

Did you encounter a migration issue regarding this behaviour?

Podman will treat "foo" and "localhost/foo" as the same image, so scripts that operated on Docker can safely migrate over to Podman as "foo" will implicitly resolve to "localhost/foo". However, podman images will list "localhost/foo" which is the only incompatibility I know.

[jschewebbn]

I don't know what a migration issue is. I've been following this issue as this is the problem that I'm having. In my specific use case I would like to create images with podman and then load them with docker. See #1034 (comment) for the details.

[rhatdan]

So you want podman save of foo to save it as docker.io/library/foo?

[jschewebbn]

I want podman save foo to create a file that has the same manifest.json and repositories files that docker save foo has.

For example when I execute docker save foo I see this in manifest.json:

[
  {
    "Config": "cd998886946a19daf9b43f35476427b9e800999f343ca463eae89f4570c10cc9.json",
    "RepoTags": [
      "foo:latest"
    ],
    "Layers": [
...
    ]
  }
]

When I executed podman save foo I see this in manifest.json:

[
  {
    "Config": "cd998886946a19daf9b43f35476427b9e800999f343ca463eae89f4570c10cc9.json",
    "RepoTags": [
      "localhost/foo:latest"
    ],
    "Layers": [
...
    ]
  }
]

The key here is the prefix on the "RepoTags" property.

The repositories file has a similar issue:
docker shows

{
  "foo": {
    "latest": "cdd290f400b43594611786fe015d8f78205421db0bd8afbbf3a12d55fffdd1ed"
  }
}

podman shows

{
  "localhost/frc":{
      "latest":"92b78c85f2b4b16c8ea4f6235af775f09d227062270fdd84bd7052c19d850528"
    }
}

[dajtxx]

I'm trying to move from docker to podman, and tried to bring up a docker-compose stack by pointing docker-compose to the podman unix socket.

I have to change the image names of my locally built images in my docker-compose.yml file to have the localhost/ prefix or it doesn't work. The postgres and rabbitmq images are fine as-is.

So this is another case where podman is not a drop-in replacement for docker.

Perhaps working with docker-compose is outside the scope of the podman project. I was just looking for the simplest possible way to get my project moved over while I learn about podman.

---

EDIT: I just found the page https://www.redhat.com/sysadmin/container-image-short-names and learned about image name aliases. This solves my problem at the expense of having to edit /etc/containers/registries.conf.

[rhatdan]

Please open a new issue on this. The bottom line is we recommend you use long names, rather then rely on Docker.io always being the default.

[dajtxx]

The images from docker.io worked fine, it was the locally built ones that podman puts localhost/ in front of that are the 'problem'.

It looks like this was a design decision by the podman team, so it's not a bug. I put my comment in because google brought me to this issue and I thought it might be useful for others to see another option for getting around that when google brings them here.

[rhatdan]

Right locally built images do not default to any registry. Perhaps a modification to this would be to allow users to specify a default registry, to eliminate this problem.

[rhatdan]

If you want us to hard code docker.io prefix, then no it is not.

[nitind]

Must there be a prefix at all?

[rhatdan]

I don't believe we can support ""/imagename

[robrich]

+1 for removing the localhost/ prefix. How does this pain me? Kind doesn't assume that localhost/myimage:v0.1 and myimage:v0.1 are the same. So now I need to modify all my k8s yaml files to add the localhost/ prefix.

[rhatdan]

What are you running your k8s yaml files with? podman kube play?

[adelton]

I for one have changed all my CI's and habits to explicitly say localhost/ whenever I mean an image not coming from remote registry, like locally built. It adds the prefix for docker operations, and it does not break podman. It is kind of nice to be explicit.

[sttyerase]

Explicit is nice in one sense. It makes intentions clear to others. That's why I like the POSIX --someexplicitoptionname convention over -s. However, if you want Podman to be interchangeable with Docker, it's a show stopper. If you want to force individuals, groups and companies to choose between Docker and Podman, then keep the localhost prefix.

This issue is so old, I don't know why it came up now. I long ago abandoned it and forgot I had a watch on it.

[robrich]

What are you running your k8s yaml files with? podman kube play?

I'm using Podman to build images, Podman Desktop to get the images to the kind cluster, and kubectl to start the containers given a yaml file.

[rhatdan]

Docker defaulted the locally built image to docker.io, hard coding their business use case into their tooling. Podman will not do the same.

Images should be pushed to a container registry, at which point they will be named with the registry. localhost/image will become docker.io/image if pushed to docker.io or quay.io/image if pushed to quay.io.

[jschewebbn]

Actually the locally built docker images don't have any prefix associated with them. With podman they have "localhost/". The workflow that is a problem is exporting the locally created images and sharing them as they have different names when created in podman than when created in docker.

[vrothberg]

Docker will always normalize short names to docker.io. So an image foo will be normalized to docker.io/library/foo:latest. Podman decided to normalize short names (when tagging!) to localhost/foo:latest. This behavior is so old that changing it would break existing workloads.

I can imagine adding an opt-in mechanism to avoid the localhost/ prefix. This may impact other functionality though and hence requires some more thought. But let's not decide that in a 5 years old issue that has been closed years ago. If you have a strong desire for such an opt-in switch, I want to encourage you to open a fresh issue.

[rhatdan]

If you have an example of a docker-compose script that blows up in this manner please open a new issue.

Podman should be able to compensate and find an image based on shortname if built locally. BTW Is this happening on a MAC or Windows Box?

[jschewebbn]

This is on a Linux box and my particular problem isn't that podman isn't able to compensate, it's that the images exported from podman are not the same as the images exported from docker and therefore when imported into docker there are problems because the names have the "localhost/" prefix. See #1034 (comment) for a detailed explanation of the issue that makes podman images incompatible with docker and therefore makes it harder for docker developers to adopt podman.

My fix is to run #1034 (comment) after saving images from podman before passing to someone that is using them with docker.

[realsircodesalot]

What about this as a possible solution? Instead of removing the localhost/ tag prefix when referencing images podman searches for tags prefixed with localhost/ first?

So if my DockerFile is referencing bodhi-ci/f27 it will search for an image tagged localhost/bodhi-ci/f27?

It could be setting possibly to tell podman to use it's own localhost/ prefixed images first before attempting to look for it at repository X's URL?

It makes sense to me why the repo name tag prefix is there - that provides clarity as to the origin of the image.

---

I came across this issue when attempting to use C# (.NET) and MS SQL (dotnet-mssql). During the build process it creates a temporary image tag and expects to be able to use it w/o the localhost/ prefix resulting in an error.

I'm looking for a way around it by possibly manually building the images that devcontainer command would and modifying the docker-compose.yml file, but if podman was aware of it's localhost/ tagged images, it would make life a little easier/more compatible.

[rhatdan]

Please open new issues or discussions.