macos: podman cannot be opened because the developer cannot be verified

[ssbarnea]

When I try to execute podman from cli I get:

“podman” cannot be opened because the developer cannot be verified.
macOS cannot verify that this app is free from malware.
Homebrew Cask downloaded this file on 30 August 2019 from github.com.

The only options are "Move to Bin" and "Cancel", and neither of them seem to provide useful.

We need to assure that podman can be installed (brew) and executed on macos, without extra tricks.

Please do not suggest disabling system security. Last time I did this was years ago. In 2019 nobody should do that. Brew installed software compiles locally and does not need signing but pre-compiled casks must use valid Apple backed signing, for good reasons.

This still seems to be valid for the latest version of podman cask as brew cask outdated does return nothing:

$ brew cask info podman                                                                                                                                                    podman: 1.6.0-28-gdac7889d
https://github.com/containers/libpod/
/usr/local/Caskroom/podman/1.6.0-28-gdac7889d (2 files, 47.4MB)
From: https://github.com/Homebrew/homebrew-cask/blob/master/Casks/podman.rb
==> Name
podman
==> Artifacts
podman-v1.6.0-28-gdac7889d/podman (Binary)

[rhatdan]

@ssbarnea Any chance you can help us get this right?

[ssbarnea]

@rhatdan sorry not right now. I am overloaded with other tasks, funny one of them being building the rpms for centos and rhel 8. Be sure that if I find the time I will propose somethign.

[github-actions]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[rhatdan]

I am keeping this open, since MAC support is a key feature we need to fully support.

[ssbarnea]

@rhatdan Happy new year! Let's see if we can have a talk (ping me on irc: zbr) around podman on non-Linux hosts and its directions. MacOS is key for podman success and I personally do not see the current varlink implementation accessible (easy to use). My hope is that we can make it work as easy as the docker cli regardless if the "server" is on localhost or remove. In fact user should not even care about that aspect.

[afbjorklund]

I think that if you install podman from source ("Formula" in brew-speak) it would work better than when installing from binary ("Cask" in brew-speak). Of course, that implies building libpod on a Mac...

[afbjorklund]

Minikube has the same problem too, and I guess every other non-Apple developer in the world ? Even if the Mac itself is available, clearing the redtape for signing stuff takes ages (even for the .deb and .rpm!)

[afbjorklund]

Here is such a formula, if some Mac users wants to test and add it to the Homebrew repository:

class Podman < Formula
  desc "Pod Manager"
  homepage "https://podman.io/"
  url "https://github.com/containers/libpod.git",
	:tag      => "v1.6.4",
	:revision => "5cc92849f7fc9dd734ca2fd8f3ae8830b9a7eb26"
  sha256 "6e59821320b435543bc7554e73faa66d5956e4ad3f7e7f4ea03bebd6726758e9"
  head "https://github.com/containers/libpod.git"

  depends_on "go" => :build

  BUILDTAGS_CROSS="containers_image_openpgp exclude_graphdriver_btrfs exclude_graphdriver_devicemapper exclude_graphdriver_overlay"

  def install
    ENV["CGO_ENABLED"] = "0"
    # Use BUILDTAGS_CROSS to avoid any system dependencies
    ENV["BUILDTAGS"] = "#{BUILDTAGS_CROSS} varlink"
    # See https://github.com/containers/libpod/issues/4787
    ENV["CIRRUS_TAG"] = "v#{version}"
    system "make", "podman-remote"
    # Pretend we are the real deal, and not just a client
    bin.install "bin/podman-remote" => "podman"
  end

  def caveats
    <<~EOS
      You should set PODMAN_VARLINK_BRIDGE to your remote Linux system:
         export PODMAN_VARLINK_BRIDGE="ssh root@host varlink bridge"
    EOS
  end

  test do
    assert_match version.to_s, shell_output("#{bin}/podman --version").chomp
  end
end

It seems to be working on Linux, tested OK with eval $(podman-machine env --varlink)

$ brew install podman
==> Cloning https://github.com/containers/libpod.git
Updating /home/anders/.cache/Homebrew/podman--git
==> Checking out tag v1.6.4
HEAD is now at 5cc9284... Bump to v1.6.4
HEAD is now at 5cc9284 Bump to v1.6.4
==> make podman-remote
==> Caveats
You should set PODMAN_VARLINK_BRIDGE to your remote Linux system:
   export PODMAN_VARLINK_BRIDGE="ssh root@host varlink bridge"
==> Summary
🍺  /home/anders/.linuxbrew/Cellar/podman/1.6.4: 6 files, 49.2MB, built in 50 seconds
$ podman version
Client:
Version:            1.6.4
RemoteAPI Version:  1
Go Version:         go1.13.5
Git Commit:         5cc92849f7fc9dd734ca2fd8f3ae8830b9a7eb26
Built:              Sun Jan  5 10:01:24 2020
OS/Arch:            linux/amd64

Service:
Error: could not get runtime: dial unix /run/podman/io.podman: connect: permission denied

The last error (about not being able to find the socket file) is "expected", see #4364

It is very similar to another error, as reported by another popular container service:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

But unlike Docker for Mac, Podman does not (yet?) have a Linux VM bundled with it.

[afbjorklund]

@ssbarnea Please let me know if podman-machine can be a interim solution before "Podman for Mac" or whatever integrated solution (like HyperKit and LinuxKit) that you guys come up with in the end...

[guymguym]

After running a new downloaded binary that fails on unidentified developer, open your "System Preferences" -> "Security & Privacy" and you will see close to the bottom a message saying that process was blocked and a button "Allow Anyway". Click the button and run the binary again, and then you will be able to approve it.

Looks something like this -

[hickeycm]

This occurs because macOS Gatekeeper automatically applies the com.apple.quarantine extended attribute on the binary because the application is not notarized. See the Apple support page Safely open apps on your Mac for more information.

You can see that this is the case by viewing the extended attributes of the binary.

$ type podman
podman is /usr/local/bin/podman
$ xattr /usr/local/bin/podman
com.apple.quarantine

To resolve the issue, you can manually delete the quarantine attribute and then verify that you can now run podman:

$ xattr -d com.apple.quarantine /usr/local/bin/podman
$ podman --help
manage pods and images

Usage:
  podman [flags]
  podman [command]
...

To permanently fix the issue, the app should be notarized before distribution. See Apple's information on notarizing macOS software before distribution. There is an extended HN discussion here with additional commentary.

[rhatdan]

@lsm5 Any chance you could see about "notarizing" our application?

[cevich]

@rhatdan I chatted with @lsm5 about this. We need to tread carefully here so we don't give anyone wrong impressions about what is trusted and what is not.

pre-compiled casks must use valid Apple backed signing, for good reasons.

Same for every other OS as well. Any builds we do in public-side automation are technically susceptible to tampering, even if only for a short while before detection. However, those signatures carry a much longer validity period. Therefor, if we automatically place a "stamp-of-approval" on every build done on the public-side, we could unknowingly violate trust at some point, and not know about it.

What needs to happen to enable signing and trust of published bits, is for that to occur along with builds, within a more locked-down and secure environment. The packages can then be pushed or uploaded somewhere on the public side for safe consumption by users.

This is not an impossible task, nor one for which we lack compute resources. It's a man-hours-required issue IMHO. Somebody has to build/maintain/monitor the build and signing environment indefinitely.

Make sense?

[ssbarnea]

If you already know how to do it manually and we do have a macos host that can be used as signer, I could try to find a safe way to sign. In the end the key is to secure the key, not what is signed. Probably is ok to sign only merged code and not open PRs, or at least not with the same certificate.

I wonder if we can find a way to make “brew install” or “brew cask install” to take care of the signing issue. The key here is to find a way to install podman cli that is unattended.

[cevich]

Probably is ok to sign only merged code and not open PRs, or at least not with the same certificate.

In both cases, we're executing w/in a public cloud using in-tree automation logic. Since it's possible to modify the logic w/in a PR, there's the possibility for malicious things to enter unnoticed. In other words, a PR could simply modify the build + upload logic to name + sign the file as if it was a release.

I'm not comfortable trusting this public-side automation machinery to hold something with as much potential impact as a signing key. I consider using the public-side releases at all, a "at your own risk" endeavor. Having to disable related security measures in order to install, is desirable behavior.

OTOH, I do appreciate the situation from a user's perspective. As the project community grows, my duct-tape build/release automation is certainly showing holes. Signing/certifying is one of them. Needing that process to involve actual MacOS and Windows VMs further complicates matters (on several levels) since they're not open platforms.

I believe to move this issue forward, what we need is:

1. People with the necessary MacOS and Windows scripting skills necessary to write/maintain private automation that can build podman, package it, sign/certify, then upload the artifacts.
2. Necessary licences and/or VM images for MacOS and Windows.
3. Dedicated/Separate private cloud(s) where all this can happen safely as part of the libpod release process.
4. Funding for all of the above.

note: I am not infallible either, so if somebody knows a different way to do this safely, please explain.

[cevich]

I think I see a way we could help improve at least the "when and where" of automating this stuff better. Cirrus-CI recently fixed a problem that was causing post-tag-push testing to not work. We don't make good use of it now, so it mostly fails. For example: https://cirrus-ci.com/build/5826216389246976

However, since it's impossible for anyone to trigger these runs without upstream-push rights, this is a good prime-mover to begin some kind of trusted build-sign-release process. What I'm thinking is, we teach Cirrus-CI to tickle a webhook on a private repo. on gitlab. Any automation setup there would also be private, and so a much safer place to store signing certificates and private keys.

If we/I could get that working, then the only piece that's missing is finding funding for a knowledgeable people/person and a cloud to run MacOS and Windows VMs in.

[cevich]

Update: Github-Actions supports both Windows and MacOS container-based automation (in addition to Linux). Doing the build/certify/signing of packages there solves all the trust, license, and cloud issues I identified. We can encode secrets (private keys) within github such that they are not exposed in cirrus or public clouds. Conveniently, triggering is branch centric, meaning the workflow/process logic itself cannot be modified by incoming code (unless implemented badly on the branch). Also conveniently, the automation and triggering components of this are closely related to another area I'm currently researching, regarding automation metrics.

[aranw]

@cevich I was wondering whether there are any updates on this. The macOS version installed via homebrew is at 1.6.0 and I believe some of the features I'm interested in using are only available in 1.7 or 1.8

If there is no progress made on this is there install instructions somewhere for building podman locally to get the latest version?

[afbjorklund]

I added a brew "formula" (source) above , if someone is interested in getting in included in Homebrew instead of the "cask" (binary)... ? But gave up on the download links or releases, and made my own.

https://github.com/boot2podman/machine#podman-remote

[cevich]

Update: Still working on this, multiple PRs in flight. Currently ironing out details of the human release process translation into automation.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[ssbarnea]

I hope this is not really stale. Having podman in macos is critical for it to become mainstream.

[cevich]

nope, not stale. I'm working on a new CI/CD setup which includes the build-release step at the end: #6784 Though it's still likely to take a while, as I'm not getting much help on the mac/windows scripting side. I may have to learn that stuff myself 😢

[afbjorklund]

I don’t think installing the podman-remote client was the problem here...

To become “mainstream” it needs to provide an integrated solution (including Linux) and I don’t think that anyone is currently working on that ? I guess that CRC is closest to a ready-to-run package, even if it does a whole lot more than “just” podman. Maybe parts of it could be reused for a “light” version

EDIT: This was crc-org/crc#961, which was added in crc-org/crc@8198b5d and disabled in crc-org/crc@4597b26 (so it is currently not implemented)

[cevich]

Correct, the problem here is we need to run a build on a MacOS VM and (somehow) sign/certify the package. However, we also need a place/method to publish it, which is how this is related to what's happening in #6784. That PR will add a 'release' hook at the end of the Cirrus-run. That will be the automation trigger for producing, signing, and publishing a official release packages (on github). There will also be a hook into the regular PR-workflow, so that we know the process works (minus the signing and publishing part).

[afbjorklund]

What I meant was that I don't actually think that it is the client that is the problem (as long as ignoring signing), it is the server.

[cevich]

I'm still confused. I thought this issue was about package-signing, in which case we need to sign all the packages that would be installed. Last I checked, only the client is able to be installed on Mac and windows.

[afbjorklund]

I'm still confused. I thought this issue was about package-signing, in which case we need to sign all the packages that would be installed. Last I checked, only the client is able to be installed on Mac and windows.

You are right. Just mentioned that if it is impossible to accomplish signing, then it is possible to install from source (using "brew install") or to download a binary using the terminal. The signing blocks the straight-forward approach of using the browser...

The server side isn't actually installed on mac or windows, it is running in a virtual machine with linux. More or less "hidden". But one could argue that it should still be installed on the same computer, if wanting to use podman without an external host ?

Sorry for the noise... Back to signing the client binary.

(the server might be punted over to CRC in the future)

[ashley-cui]

@ssbarnea try

brew cask uninstall podman 
brew install podman

Podman is now available through brew, not brew cask, and should not have the signing issue now.

[cevich]

Update (on the signed-installer effort): This has been pushed down one notch on my TODO list for now. I'm currently focusing on erasing some automation technical-debt. Once that's done, I'll get back onto this effort. For end-users, debugging/playing, the solution @ashley-cui invented will hopefully be adequate. (thanks Ashley)

[ssbarnea]

Super! For the first time I was able to install podman and run podman --version on MacOS without any security prompts.

Sadly the --connection-string format seems to be something undocumented, doing a google search also gives no answers, even trying to search this on official docs site brings no hope.

[TomSweeneyRedHat]

So close! @ashley-cui @jwhonce ideas on how to get @ssbarnea over the finish line?

[ssbarnea]

Based on what I read the other day I expect that I will not be able to connect to my centos-8 devbox as it is running podman-1.7 but I should be able to connect to my fedora box which is runnind 2.0.5, once I reverse engineer the connection string format, or that is the --url ?.

What is the podman equivalent of export DOCKER_HOST=ssh://[email protected]?

[ashley-cui]

@ssbarnea Take a look at the docs for podman system connection. We should be publishing a blog soon on how to set everything up, but the basic gist of it is:

Copy ssh pub key from mac to Linux box
On your Linux box: systemctl --user enable podman.socket
On Mac: podman system connection add *name* --identity path/to/ssh_key

Then you should be able use podman commands as normal

[afbjorklund]

@ashley-cui : the formula seems to be failing in linuxbrew, but I guess that was expected (hardcodes podman-remote-darwin)

I was trying to test it, like before #4511 (comment)

[afbjorklund]

What is the podman equivalent of export DOCKER_HOST=ssh://[email protected]?

export CONTAINER_HOST=ssh://[email protected]/run/podman/podman.sock

EDIT: The /var/run is deprecated by systemd, I think I was thinking of /var/run/docker.sock :-)

lrwxrwxrwx 1 root root 4 dec  7  2018 /var/run -> /run

[ssbarnea]

time="2020-08-29T15:37:30+01:00" level=info msg="podman filtering at log level debug"
time="2020-08-29T15:37:30+01:00" level=debug msg="Called ps.PersistentPreRunE(podman --log-level debug ps)"
time="2020-08-29T15:37:30+01:00" level=debug msg="Found SSH_AUTH_SOCK "/private/tmp/com.apple.launchd.d5nY2UOxQ1/Listeners", ssh-agent signer enabled"
Error: Get "http://d/v1.0.0/libpod../../../_ping": ssh: rejected: connect failed (open failed)

The error is not clear, does not give any hints regarding the nature of the failure.

After running systemctl enable --now podman.socket on the remote host the problem resolved and podman seems now to be able to communicate.

In order to avoid a huge amount of confusions among early adopters, I think we need a clear error message, one that would mention that /var/run/podman/podman.sock was not found or not readable and also to suggest running systemctl enable --now podman.socket on remote host to enable the service.

I am really happy that now I can really use podman from macos podman run hello-world suceeded... well the message was "Hello from Docker!" which was not really true but it works just fine.

Thanks to all involved in this!!!

[afbjorklund]

In order to avoid a huge amount of confusions among early adopters, I think we need a clear error message, one that would mention that /var/run/podman/podman.sock was not found or not readable and also to suggest running systemctl enable --now podman.socket on remote host to enable the service.

We talked a bit about it in #6809, when discussing how to replace the "sudo varlink" method currently used...
Some systems don't allow connecting as root (either at all, or simply by not providing a ssh key by default)

So wanted a way to separate between connecting to root socket and connecting as rootless and user socket.
Currently that is being accomplished explicitly by using different socket paths, which I suppose "works"

export CONTAINER_HOST=ssh://[email protected]/run/podman/podman.sock
export CONTAINER_HOST=ssh://[email protected]/run/user/1000/podman/podman.sock

But this has less and less to do with signing the binary, so probably not appropriate for this particular issue.

[ssbarnea]

All nice until I tried my first build and discovered that no stdout/stderr is displayed at all. So I raised #7498

Apparently when building in remote mode, we loose any ability to see buildah output.

[afbjorklund]

All nice until I tried my first build and discovered that no stdout/stderr is displayed at all.

Yeah, but that bug seems to be same on Linux (with podman-remote v2). I can't build at all either... ☹️

[rhatdan]

@ashley-cui Is this fixed now?

[ssbarnea]

We should close this ticket as I considered it resolved, or unable to reproduce anymore, mainly because we now compile it with brew instead of using a cask.

Still, I just raised another issue #7602

I see someone added do-not-close label, so I will let them make a decision on that.