Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Podman version 3.0.2 failed to pull localhost images, everytime trying to pull from registry #11181

Closed
sudhirtomar opened this issue Aug 10, 2021 · 7 comments
Labels
locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@sudhirtomar
Copy link

sudhirtomar commented Aug 10, 2021

I am trying to use podman existing images but it's failing, either it's failing to load the image from localhost or it's going to pull the image from registry -

podman --version
podman version 3.0.2-dev

OS - RHEL 8

podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/splunk/scs v1.12.0 ac68a53e5a7e 17 months ago 495 MB
localhost/sc4slocal offlineimage ac68a53e5a7e 17 months ago 495 MB


"podman run localhost/sc4slocal:offlineimage" OR "podman run sc4slocal:offlineimage"
syslog-ng checking config
sc4s version=v1.12.0
syslog-ng starting
Aug 10 11:10:27 7a370d484433 syslog-ng[1]: syslog-ng starting up; version='3.25.1'
Aug 10 11:10:27 7a370d484433 syslog-ng-config: sc4s version=v1.12.0

This is running fine


But Whenever I tried to run from /lib/systemd/system/sc4s.service It will start creating problems -
Podman + systemd = https://splunk-connect-for-syslog.readthedocs.io/en/latest/gettingstarted/podman-systemd-general/
Read-First = https://splunk-connect-for-syslog.readthedocs.io/en/latest/gettingstarted/#offline-container-installation

Configuration -

[Unit]
Description=SC4S Container
Wants=NetworkManager.service network-online.target
After=NetworkManager.service network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Environment="SC4S_IMAGE=sc4slocal:offlineimage"

### Required mount point for syslog-ng persist data (including disk buffer)
Environment="SC4S_PERSIST_MOUNT=splunk-sc4s-var:/var/lib/syslog-ng"

### Optional mount point for local overrides and configurations; see notes in docs
Environment="SC4S_LOCAL_MOUNT=/opt/sc4s/local:/etc/syslog-ng/conf.d/local:z"

### Optional mount point for local disk archive (EWMM output) files
Environment="SC4S_ARCHIVE_MOUNT=/opt/sc4s/archive:/var/lib/syslog-ng/archive:z"

### Map location of TLS custom TLS
Environment="SC4S_TLS_MOUNT=/opt/sc4s/tls:/etc/syslog-ng/tls:z"

TimeoutStartSec=0

###ExecStartPre=/usr/bin/podman pull $SC4S_IMAGE --(commented as suggested in Read First )
ExecStartPre=/usr/bin/bash -c "/usr/bin/systemctl set-environment SC4SHOST=$(hostname -s)"

ExecStart=/usr/bin/podman run \
        -e "SC4S_CONTAINER_HOST=${SC4SHOST}" \
        -v "$SC4S_PERSIST_MOUNT" \
        -v "$SC4S_LOCAL_MOUNT" \
        -v "$SC4S_ARCHIVE_MOUNT" \
        -v "$SC4S_TLS_MOUNT" \
        --env-file=/opt/sc4s/env_file \
        --network host \
        --name SC4S \
        --rm $SC4S_IMAGE

run the configuration -

sudo systemctl daemon-reload 
sudo systemctl enable sc4s
sudo systemctl start sc4s

Issue while running above command-

If I set in configuration -

[Service]
Environment="SC4S_IMAGE=sc4slocal:offlineimage"

systemd[1]: Started SC4S Container.
podman[3042091]: Trying to pull registry.redhat.io/sc4slocal:offlineimage...
podman[3042091]: Get "https://registry.redhat.io/v2/": dial tcp 184.87.7.162:443: i/o timeout
podman[3042091]: Trying to pull docker.io/library/sc4slocal:offlineimage...
podman[3042091]: Get "https://registry-1.docker.io/v2/": dial tcp 107.23.149.57:443: i/o timeout
podman[3042091]: Error: 3 errors occurred while pulling:
podman[3042091]: * Error initializing source docker://registry.access.redhat.com/sc4slocal:offlineimage: error pinging docker registry registry.acce>
podman[3042091]: * Error initializing source docker://registry.redhat.io/sc4slocal:offlineimage: error pinging docker registry registry.redhat.io: G>
podman[3042091]: * Error initializing source docker://sc4slocal:offlineimage: error pinging docker registry registry-1.docker.io: Get "https://regis>
systemd[1]: sc4s.service: Main process exited, code=exited, status=125/n/a
systemd[1]: sc4s.service: Failed with result 'exit-code'.

If I set in configuration -

[Service]
Environment="SC4S_IMAGE=localhost/sc4slocal:offlineimage"

systemd[1]: Started SC4S Container.
podman[3054347]: Trying to pull localhost/sc4slocal:offlineimage...
podman[3054347]: time="2021-08-10T12:37:16Z" level=warning msg="failed, retrying in 1s ... (1/3). Error: Error initializing source docker://localhost>
podman[3054347]: time="2021-08-10T12:37:17Z" level=warning msg="failed, retrying in 2s ... (2/3). Error: Error initializing source docker://localhost>
podman[3054347]: time="2021-08-10T12:37:19Z" level=warning msg="failed, retrying in 4s ... (3/3). Error: Error initializing source docker://localhost>
podman[3054347]: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
podman[3054347]: Error: Error initializing source docker://localhost/sc4slocal:offlineimage: error pinging docker registry localhost: Get "https://lo>
systemd[1]: sc4s.service: Main process exited, code=exited, status=125/n/a

Can anyone please help why this is behaving like this, have disabled the PULL from configuration but still it's going to pull.
Earlier I was using tag - latest not I have replaced with offlineimage but still same issue.

#7699
#9111
https://stackoverflow.com/questions/63970144/podman-play-does-not-pull-localhost-image-get-https-localhost-v2-dial-tcp

@mheon
Copy link
Member

mheon commented Aug 10, 2021

When you are using podman run, you seem to be referring to the image as localhost/sc4slocal, but when you are using your systemd unit file you are only using sc4slocal. Does it work if you use the full localhost/sc4slocal path in the systemd unit?

Also, I note that this looks like a custom-crafted systemd unit - we strongly recommend you use podman generate systemd to produce unit files, to avoid a number of pitfalls around the use of PID files.

@sudhirtomar
Copy link
Author

sudhirtomar commented Aug 10, 2021

@mheon
When I use full localhost/sc4slocal path it will give error like below

[Service]
Environment="SC4S_IMAGE=localhost/sc4slocal:offlineimage"

systemd[1]: Started SC4S Container.
podman[3054347]: Trying to pull localhost/sc4slocal:offlineimage...
podman[3054347]: time="2021-08-10T12:37:16Z" level=warning msg="failed, retrying in 1s ... (1/3). Error: Error initializing source docker://localhost>
podman[3054347]: time="2021-08-10T12:37:17Z" level=warning msg="failed, retrying in 2s ... (2/3). Error: Error initializing source docker://localhost>
podman[3054347]: time="2021-08-10T12:37:19Z" level=warning msg="failed, retrying in 4s ... (3/3). Error: Error initializing source docker://localhost>
podman[3054347]: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
podman[3054347]: Error: Error initializing source docker://localhost/sc4slocal:offlineimage: error pinging docker registry localhost: Get "https://lo>
systemd[1]: sc4s.service: Main process exited, code=exited, status=125/n/a

I will give a hit with podman generate systemd to produce unit files, but the same file configuration is running on my another machine.

@sudhirtomar
Copy link
Author

@mheon

With help of podman generate systemd create the new unit file.

podman generate systemd --name splunksyslog
# container-splunksyslog.service
# autogenerated by Podman 3.0.2-dev
# Tue Aug 10 15:16:34 GMT 2021

[Unit]
Description=Podman container-splunksyslog.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStart=/usr/bin/podman start splunksyslog
ExecStop=/usr/bin/podman stop -t 10 splunksyslog
ExecStopPost=/usr/bin/podman stop -t 10 splunksyslog
PIDFile=/run/containers/storage/overlay-containers/b3ce3cce9e50a9ecc429bws46ba23defrccc7536d883ertgbf43c34f1cbc0e3a5c5/userdata/conmon.pid
Type=forking

[Install]
WantedBy=multi-user.target default.target

With above configuration podman started successfully with container-splunksyslog.service

sudo systemctl start sc4s
systemctl status sc4s.service
● sc4s.service - Podman container-splunksyslog.service
   Loaded: loaded (/usr/lib/systemd/system/sc4s.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-08-10 17:13:25 GMT; 4s ago
     Docs: man:podman-generate-systemd(1)
  Process: 3180906 ExecStopPost=/usr/bin/podman stop -t 10 splunksyslog (code=exited, status=0/SUCCESS)
  Process: 3180763 ExecStop=/usr/bin/podman stop -t 10 splunksyslog (code=exited, status=0/SUCCESS)
  Process: 3190532 ExecStart=/usr/bin/podman start splunksyslog (code=exited, status=0/SUCCESS)
 Main PID: 3190628 (conmon)
    Tasks: 2 (limit: 75125)
   Memory: 1.9M
   CGroup: /system.slice/sc4s.service
           └─3190628 /usr/bin/conmon --api-version 1 -c b3ce3cce9e50a9ecc429b6346ba23defrccc7536d883ertgbf43c34f1cbc0e3a5c5
systemd[1]: Starting Podman container-splunksyslog.service...
podman[3190532]: splunksyslog
systemd[1]: Started Podman container-splunksyslog.service.

But when I am checking the podman logs, I am seeing below errors.

podman logs splunksyslog
syslog-ng checking config
sc4s version=v1.12.0
syslog-ng starting
Aug 10 15:49:21 b3ce4dda9e50 syslog-ng[1]: syslog-ng starting up; version='3.25.1'
Aug 10 15:49:21 b3ce4dda9e50 syslog-ng-config: sc4s version=v1.12.0
Aug 10 15:49:21 b3ce4dda9e50 syslog-ng[1]: curl: error sending HTTP request; url='/services/collector/event', error='URL using bad/illegal format or missing URL', worker_index='0', driver='d_hec_internal#0', location='/opt/syslog-ng/etc/conf.d/destinations/splunk_hec_internal.conf:2:5'
Aug 10 15:49:21 b3ce4dda9e50 syslog-ng[1]: curl: error sending HTTP request; url='/services/collector/event', error='URL using bad/illegal format or missing URL', worker_index='1', driver='d_hec_internal#0', location='/opt/syslog-ng/etc/conf.d/destinations/splunk_hec_internal.conf:2:5'
Aug 10 15:49:21 b3ce4dda9e50 syslog-ng[1]: Server disconnected while preparing messages for sending, trying again; driver='d_hec_internal#0', location='/opt/syslog-ng/etc/conf.d/destinations/splunk_hec_internal.conf:2:5', worker_index='0', time_reopen='10', batch_size='1'
Aug 10 15:49:21 b3ce4dda9e50 syslog-ng[1]: Server disconnected while preparing messages for sending, trying again; driver='d_hec_internal#0', location='/opt/syslog-ng/etc/conf.d/destinations/splunk_hec_internal.conf:2:5', worker_index='1', time_reopen='10', batch_size='1'

I am unable to identify, why this issue is occurring.

Also If I am trying to run with

ExecStart=/usr/bin/podman start splunksyslog\
        -e "SC4S_CONTAINER_HOST=${SC4SHOST}" \
        -v "$SC4S_PERSIST_MOUNT" \
        -v "$SC4S_LOCAL_MOUNT" \
        -v "$SC4S_ARCHIVE_MOUNT" \
        -v "$SC4S_TLS_MOUNT" \
        --env-file=/opt/sc4s/env_file \
        --network host \
        --name SC4S \
        --rm $SC4S_IMAGE

then service is not starting.

I have most of my HEC configuration in /opt/sc4s/env_file, look like the new container not referring to the config file.

@edsantiago edsantiago changed the title Podman version 3.0.2 failed to pull loaclhost images, everytime trying to pull from registry Podman version 3.0.2 failed to pull localhost images, everytime trying to pull from registry Aug 10, 2021
@sudhirtomar
Copy link
Author

Podman issue resolved, hence closing this issue.

@sudhirtomar
Copy link
Author

Podman issue resolved, hence closing this issue.

@bayeslearner
Copy link

This issue is there in 3.4.2.

@mheon
Copy link
Member

mheon commented Dec 6, 2021

Please open a new issue

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 21, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

No branches or pull requests

3 participants