Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Podman Init wont Resolve DNS Windows WSL #14495

Closed
Siythrun opened this issue Jun 6, 2022 · 33 comments
Closed

Podman Init wont Resolve DNS Windows WSL #14495

Siythrun opened this issue Jun 6, 2022 · 33 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. machine stale-issue windows issue/bug on Windows

Comments

@Siythrun
Copy link

Siythrun commented Jun 6, 2022

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Podman machine unable to resolve DNS

Steps to reproduce the issue:

  1. podman machine init

Describe the results you received:

When trying to install packages for Fedora 35 the WSL VM is unable to resolve the repo mirrors

Errors during downloading metadata for repository 'fedora':
  - Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org]
Error: Failed to download metadata for repo 'fedora': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org]
Error: package upgrade on guest OS failed: exit status 1

Describe the results you expected:

I would expect it to use the hosts DNS server settings to resolve

Additional information you deem important (e.g. issue happens only occasionally):

This might be related to an issue with WSL microsoft/WSL#3438
It is important not for me to be able to access the Internet on any WSL instance I always need to config the resolve.conf to have the appropriate DNS nameservers

Output of podman version:

Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v4.1.0/libpod/_ping": dial unix /wsl$/Fedora34/run/podman/podman.sock: connect: A socket operation encountered a dead network.

Output of podman info --debug:

Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v4.1.0/libpod/_ping": dial unix /wsl$/Fedora34/run/podman/podman.sock: connect: A socket operation encountered a dead network.
P

Package info (e.g. output of rpm -q podman or apt list podman):

N/A

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
The ability to update the resolve conf as part of the init would be appreciated or to run init wait for it to fail fix the VM manually and then continue the setup

@openshift-ci openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Jun 6, 2022
@mheon
Copy link
Member

mheon commented Jun 6, 2022

@baude @ashley-cui PTAL

@nobodyman1
Copy link

nobodyman1 commented Jun 7, 2022

I tested version 4.0.3, both versions have same issue.

@nobodyman1
Copy link

Please check the following command:
wsl -d podman-machine-default cat /etc/resolv.conf

The output is something like this:

# This file was automatically generated by WSL. To stop automatic generation of this file, add the following entry to /etc/wsl.conf:
# [network]
# generateResolvConf = false
nameserver 192.168.96.1

If I start the machine with wsl -d podman-machine-default and change /etc/resolv.conf to nameserver 8.8.8.8 the dns resolution is working as expected. (Tried with dnf upgrade -y.)

But I don´t know how the make this setting permanent.

@Siythrun
Copy link
Author

Siythrun commented Jun 9, 2022

I actualy have to do this alot and wrote a script to auto config our Dev Laptops to provision new instances of WSL Fedora35 for general dev work here is a powershell snippet for making it persists however it requires killing the WSL instance

if (Test-Path \\wsl$\podman-machine-default\){
    Write-Host -BackgroundColor Gray -ForegroundColor Black "Patching podman-machine-default Network"
    wsl -d Fedora35 rm -f \\wsl$\podman-machine-default\etc\wsl.conf \\wsl$\podman-machine-default\etc\resolv.conf
    wsl -d Fedora35 echo -e "[network]\ngenerateResolvConf = false" | tee -a \\wsl$\podman-machine-default\etc\wsl.conf
    ((Get-Content "\\wsl$\podman-machine-default\etc\wsl.conf") -join "`n") + "`n" | Set-Content -NoNewline "\\wsl$\podman-machine-default\etc\wsl.conf"
    wsl -t podman-machine-default
    wsl -d podman-machine-default unlink /etc/resolv.conf
    $DNSList=Get-DnsClientServerAddress -AddressFamily IPv4 | Select-Object -ExpandPropert ServerAddresses
    foreach ($DNS in $DNSList) {
        wsl -d podman-machine-default echo -e "nameserver $DNS" | tee -a \\wsl$\podman-machine-default\etc\resolv.conf
    }
    wsl -d podman-machine-defaultecho -e "search lan example.com" | tee -a \\wsl$\podman-machine-default\etc\resolv.conf
    ((Get-Content "\\wsl$\podman-machine-default\etc\resolv.conf") -join "`n") + "`n" | Set-Content -NoNewline "\\wsl$\podman-machine-default\etc\resolv.conf"
}else{
    Write-Host -ForegroundColor Red "Cant access \\wsl$\podman-machine-default\"
    exit;
}

It checks to see if the WSL instance exists then patches the wsl.conf to stop the auto regen of the resolve.conf next adding all the nameservers from windows to the list and adds a search lan if needed

But due to it killing the WSL instance the machine startup has not progressed and will not start it would be better if there was some way we could configure the resolve.conf before the init command trys to update fedora or if there was some way to make an existing WSL instance into a podman machine

@nobodyman1
Copy link

I´m not able to use your example code.
wsl -d podman-machine-default unlink /etc/resolv.conf is working (although I think it´s not needed), but
wsl -d podman-machine-default echo -e "nameserver 8.8.8.8" | tee -a \\wsl$\podman-machine-default\etc\resolv.conf doesn´t work!
And take a look at wsl -d podman-machine-defaultecho -e "search lan example.com" ... this can´t work.
Did you post the correct code?

@chungyueh-chiu
Copy link

chungyueh-chiu commented Jun 14, 2022

Had the same issue with podman 4.1.0 on windows.

I first ran podman machine init.

> podman machine init
Extracting compressed file
Installing packages (this will take awhile)...
Failed to set locale, defaulting to C.UTF-8
Fedora 35 - x86_64                                                                                                                                                               0.0  B/s |   0  B     02:00
Errors during downloading metadata for repository 'fedora':
  - Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org]
Error: Failed to download metadata for repo 'fedora': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org]
Error: package upgrade on guest OS failed: exit status 1

Then I wsl into podman-machine-default and modified the /etc/resolv.conf to make it possible to download from "mirrors.fedoraproject.org".
It still failed when I ran "podman machine start"

I followed here to change the /etc/resolv.conf and set it immutable.
Due to chattr: command not found, I first ran yum install e2fsprogs, and it also downloaded the fedora files automatically.

I exit podman-machine-default and ran podman machine start from Powershell, and it showed like this.

> podman machine start
Starting machine "podman-machine-default"
/bin/bash: line 1: /root/bootstrap: No such file or directory
Error: WSL bootstrap script failed: exit status 127

@JerryMWeeks
Copy link

@b03602036 I arrived at /root/bootstrap etc. without having to do as much!
But I had to manually generate the ssh files
ssh-keygen -f podman-machine-default
Copy the priv key file to .ssh
wsl -d podman-machine-default
in the Linux machine
mkdir .ssh touch .ssh/known_hosts vi .ssh/known_hosts
Paste the pub key into the file and :wq
exit the container
run podman machine start

@JerryMWeeks
Copy link

I got suspicious with WSL and Windows Firewall (depending on your Windows build)
Admin PowerShell and run
New-NetFirewallRule -DisplayName "WSL" -Direction Inbound -InterfaceAlias "vEthernet (WSL)" -Action Allow New-NetFirewallRule -DisplayName "WSL" -Direction Outbound -InterfaceAlias "vEthernet (WSL)" -Action Allow
Then start type windows firewall and run
choose advanced advanced settings
click on windows defender firewall properties

image

Click on Protected network settings

image

Untick vEthernet WSL

image

Didn't make any difference though! :(

Found it suggested somewhere when you search for WSL 2 Windows Firewall blocked

@JerryMWeeks
Copy link

JerryMWeeks commented Jun 16, 2022

Is this where the error comes from?
Inside the Fedora WSL machine run the following:

sudo dnf -y install podman
Fedora 35 - x86_64                                                                      0.0  B/s |   0  B     02:00
Errors during downloading metadata for repository 'fedora':
  - Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org]
Error: Failed to download metadata for repo 'fedora': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org]

Rather than adding a new comment adding this I found in the remote_client.md

Setting up SSH

Remote Podman uses SSH to communicate between the client and server. The remote client works considerably smoother using SSH keys. To set up your ssh connection, you need to generate an ssh key pair from your client machine. NOTE: in some instances, using a rsa key will cause connection issues, be sure to create an ed25519 key.

ssh-keygen -t ed25519

Your public key by default should be in your home directory under ~/.ssh/id_ed25519.pub. You then need to copy the contents of id_ed25519.pub and append it into ~/.ssh/authorized_keys on the Linux server. You can automate this using ssh-copy-id.

And this - found in keys.go:
var sshCommand = []string{"ssh-keygen", "-N", "", "-t", "ed25519", "-f"}
The resultant command - for manually building the same is (press enter without password twice when prompted)
ssh-keygen -t ed25519 -f id_ed25519
which is a pain if you happen to use GitLab - you'll already have that name taken!

####Next up if clearing the ssh /tls for getting Fedora podman
sudo dnf -y install podman
in my case failed
tested using
curl https://ostree.fedoraproject.org/mirrorlist curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.se/docs/sslcerts.html

#Okay you are going to enjoy this!

Next up is to get the cert from ostree.fedoraproject.org:443
Open windows command and run (assumed that you have openssl installed if you are doing wsl)
echo -n | openssl s_client -connect ostree.fedoraproject.org:443 | openssl x509 > fedoramirrors.crt
that will create you a file fedoramirrors.crt in the directory where you ran the command
Open the file in Notepad++
If you haven't got it, create a new file in c:\users<yourusername>.wslconfig
Edit the file adding the following entries, adjust as needed to match your machine size
The all important line is the network entry - as it prevents wsl from rewriting the dns file in the distro.

  • I cannot use the "hash" symbol in the code below so replace the word "hash" with #
    (and markup is failing so screenshot as well)

`"hash" Settings apply across all Linux distros running on WSL 2
[wsl2]

"hash" Limits VM memory to use no more than 20 GB, this can be set as whole numbers using GB or MB
memory=20GB

"hash" Sets the VM to use 6 virtual processors
processors=6

"hash" Sets amount of swap storage space to 12GB, default is 25% of available RAM (48GB)
swap=12GB

[network]
generateResolvConf = false`

image

Run wsl -d podman-machine-default
to get to the Podman machine (Fedora)
In the WSL Machine run
sudo vi /etc/resolv.conf
and add
nameserver 8.8.8.8

Esc :wq to save and exit vi
Save the file - it will looks something like this
image

Now run
touch /etc/pki/ca-trust/source/anchors/fedoramirrors.crt
followed by
sudo vi /etc/pki/ca-trust/source/anchors/fedoramirrors.crt
Then copy and paste the contents of the crt file open in Notepad++ (right click should paste)
Take note the paste trims off -----BEG at the start of the cert, so type if back in. ESC and :wq to write the file
Now run
sudo update-ca-trust to load the new cert
Now test using
curl https://ostree.fedoraproject.org/mirrorlist which should now run without error
If it does, you can go get podman! Run
sudo dnf -y install podman
a few moments later:
a bunch of completions will roll by

`Verifying : qrencode-libs-4.1.1-1.fc35.x86_64 54/62
Verifying : shadow-utils-subid-2:4.9-9.fc35.x86_64 55/62
Verifying : systemd-249.12-3.fc35.x86_64 56/62
Verifying : systemd-libs-249.12-3.fc35.x86_64 57/62
Verifying : systemd-networkd-249.12-3.fc35.x86_64 58/62
Verifying : systemd-pam-249.12-3.fc35.x86_64 59/62
Verifying : systemd-resolved-249.12-3.fc35.x86_64 60/62
Verifying : util-linux-2.37.4-1.fc35.x86_64 61/62
Verifying : util-linux-core-2.37.4-1.fc35.x86_64 62/62

Installed:
aardvark-dns-1.0.3-1.fc35.x86_64 acl-2.3.1-2.fc35.x86_64 catatonit-0.1.7-1.fc35.x86_64
conmon-2:2.1.0-2.fc35.x86_64 containernetworking-plugins-1.1.0-1.fc35.x86_64 containers-common-4:1-45.fc35.noarch
criu-3.17-2.fc35.x86_64 criu-libs-3.17-2.fc35.x86_64`

and more
Now to find if it works! phew.
which it doesn't where is this file?
podman machine start Starting machine "podman-machine-default" /bin/bash: line 1: /root/bootstrap: No such file or directory Error: WSL bootstrap script failed: exit status 127

@JerryMWeeks
Copy link

JerryMWeeks commented Jun 16, 2022

By the way @Siythrun the entry
[network]
generateResolvConf = false
in .wslconfig
followed the the editing of the sudo vi /etc/resolv.conf
fixes this issue, but not the bootstrap etc.
I am wondering if a fresh podman init with the .wslconfig settings set, will now work.
my test machine is complaining "file already in use" so this is for tomorrow now.

@nobodyman1
Copy link

nobodyman1 commented Jun 16, 2022

Remark:
I installed the newest version v4.1.1.
podman machine init (and DNS resolution) is working!

Update:
I have another system where I installed the newest version.
But on this system the error persists!

@JerryMWeeks
Copy link

JerryMWeeks commented Jun 17, 2022

I just reset my dev machine, reinstalled WSL, followed by podman 4.1.1 and then ran init -
still getting this irritating message!
podman machine init Downloading VM image: fedora-35-x86_64.tar.xz: done Extracting compressed file Importing operating system into WSL (this may take 5+ minutes on a new WSL install)... The process cannot access the file because it is being used by another process. Error: WSL import of guest OS failed: exit status 4294967295

Is there any approach I can use to debug the install and find this locked file?
I have switched off followed by reboot 3 times before getting this (before doing anything else)
Antivirus perhaps?

@Luap99 Luap99 added machine windows issue/bug on Windows labels Jun 17, 2022
@Luap99
Copy link
Member

Luap99 commented Jun 17, 2022

@n1hility PTAL

@nobodyman1
Copy link

I just reset my dev machine, reinstalled WSL, followed by podman 4.1.1 and then ran init - still getting this irritating message! podman machine init Downloading VM image: fedora-35-x86_64.tar.xz: done Extracting compressed file Importing operating system into WSL (this may take 5+ minutes on a new WSL install)... The process cannot access the file because it is being used by another process. Error: WSL import of guest OS failed: exit status 4294967295

Is there any approach I can use to debug the install and find this locked file? I have switched off followed by reboot 3 times before getting this (before doing anything else) Antivirus perhaps?

You will find the downloaded files in %USERPROFILE%\.local\share\containers\podman\machine\wsl.
I think you should check wsl -l -v if the state of the machine is Stopped.

@JerryMWeeks
Copy link

Thanks @nobodyman1 - actually between each init I have been running
wsl --unregister podman-machine-default
followed by
deleting the sub-folder podman and all contents under
%USERPROFILE%.local\share\containers , %USERPROFILE%.config\share\containers ,and the file: %USERPROFILE%\appdata\roaming\containers\containers.conf
Without deleting these podman machine init fails on machine exists or similar.
Oddly my work laptop I can repeatedly delete, experiment, recreate etc. (but dns lookup to mirrors.fedoraproject.org always fails). But on the dev machine I always get the file locked message - I am now suspecting it is because I am RDP connected, via a double login (first local followed by domain logins).

@n1hility
Copy link
Member

n1hility commented Jun 17, 2022

Other related WSL Issues: microsoft/WSL#4285 microsoft/WSL#8365

We can provide a way for you to perform manual configuration actions through init, but if you need to overwrite resolv.conf there is likely a WSL networking problem at play. Are any of you who are experiencing this problem using a VPN or special firewall?

A common cause for this sort of thing is that your windows defender policy is dropping packets.

Could you guys check the following

  1. cat /etc/resolv.conf (before patching it)
  2. Output of ifconfig (you will need to dns install net-tools first - after patching resolv.conf)
  3. Output of netstat -rn
  4. On a Windows powershell prompt: Output of ipconfig

What you should see is that these are consistent. The value of resolv.conf matches the gateway address which also matches the windows WSL vEthernet interface IP.

This is just a rudimentary check that the file is generated correctly and still in sync with the active network.

Next try

  1. wsl --shutdown
  2. wsl -d podman-machine-default
  3. curl www.google.com # Does DNS work yet?

Then reboot the system completely and try the same, does it work then?

Assuming there is no special firewalls or a VPN, the next thing to look at is your Windows Defender Firewall policy.

A simple first test is to try completely fully disabling windows defender off (not just the interface). Go to Control Panel\System and Security\Windows Defender Firewall. Turn off for everything. If you do so does DNS start working?

If so you can turn it back on and try to figure out why:

After you start a wsl command, If you look at the inbound rules, using the following powershell command:

Get-NetFirewallRule -Name HNS* |
 Format-Table -Wrap -Autosize -Property Profile, Direction,
 @{Name='Intf';e={($PSItem | Get-NetFirewallInterfaceFilter).InterfaceAlias}}, 
 @{Name='LPrt';e={($PSItem | Get-NetFirewallPortFilter).LocalPort}},
 @{Name='RPrt';e={($PSItem | Get-NetFirewallPortFilter).RemotePort}},
 @{Name='Prot';e={($PSItem | Get-NetFirewallPortFilter).Protocol}},
 @{Name='RAdd';e={($PSItem | Get-NetFirewallAddressFilter).RemoteAddress}},
 @{Name='LAdd';e={($PSItem | Get-NetFirewallAddressFilter).LocalAddress}},
 @{Name='Prog';e={($PSItem | Get-NetFirewallApplicationFilter).Program}},
 Name

You should see something like this (when WSL is running)

Profile Direction Intf            LPrt RPrt Prot RAdd LAdd Prog                              Name
------- --------- ----            ---- ---- ---- ---- ---- ----                              ----
    Any   Inbound Any             53   Any  TCP  Any  Any  %SystemRoot%\system32\svchost.exe HNS Container Networking -
                                                                                             ICS DNS (TCP-In) - E152AFC2-7
                                                                                             93D-4750-A09F-8123F6404192 -
                                                                                             0
    Any   Inbound vEthernet (WSL) 53   Any  UDP  Any  Any  Any                               HNS Container Networking -
                                                                                             DNS (UDP-In) - E152AFC2-793D-
                                                                                             4750-A09F-8123F6404192 - 0

The interface for the UDP rule should match the name in your ipconfig output. If instead you have a UID that does not exist, then that would be a problem.

If that or anything else with the rules looks off, you could create your own DNS rules just apply them to the any interface.

If everything looks correct you could still confirm its dropping the packets, you can enable logging by launching windows defender firewall -> advanced settings -> right clicking on the (Windows Defender Firewall with) icon above the Rules icons, and choose properties, then pick the public tab, then logging, enable log dropped packets, and specify a file location for the log. It rotates so depending on how much dropped traffic you have it might fly by. Then you attempt a dns query and if its logged then you confirm thats the source of the problem.

If it is dropping then you can try the same solution, create a custom rule to get it working.

If you exhaust the Windows Defender route then it likely is some other networking issue, a VPN client, a firewall in an AV tool, or some other network interface that might be mucking with the routing.

@n1hility
Copy link
Member

and more Now to find if it works! phew. which it doesn't where is this file? podman machine start Starting machine "podman-machine-default" /bin/bash: line 1: /root/bootstrap: No such file or directory Error: WSL bootstrap script failed: exit status 127

Podman machine does a number of customization steps, so thats why you are receiving this error. Until we have a simpler override mechanism, what might work as a temporary hack, is you could modify the rootfs that we start from:

https://github.com/fedora-cloud/docker-brew-fedora/blob/35/x86_64/fedora-35-x86_64.tar.xz

If you add your custom resolv.conf and a /etc/wsl.conf to the tarball with

[network]
generateResolvConf = false

then you pass that the new tar.xz to

podman machine init --image-path custom.tar.xz

I'm not completely sure if those files will survive, but worth a try.

@JerryMWeeks
Copy link

JerryMWeeks commented Jun 20, 2022

Other related WSL Issues: microsoft/WSL#4285 microsoft/WSL#8365

...
If you exhaust the Windows Defender route then it likely is some other networking issue, a VPN client, a firewall in an AV tool, or some other network interface that might be mucking with the routing.

thanks @n1hility
I was looking at this issue #14593 - it just happens that I have to use zScaler and another VPN type of app. These are disabled when I run the init, but policy prevents me from fully exiting them (prior to init I do an ipconfig /flushdns and restart PowerShell in an attempt to avoid hanging routes)

@JerryMWeeks
Copy link

JerryMWeeks commented Jun 20, 2022

and more Now to find if it works! phew. which it doesn't where is this file? podman machine start Starting machine "podman-machine-default" /bin/bash: line 1: /root/bootstrap: No such file or directory Error: WSL bootstrap script failed: exit status 127

Podman machine does a number of customization steps, so thats why you are receiving this error. Until we have a simpler override mechanism, what might work as a temporary hack, is you could modify the rootfs that we start from:

https://github.com/fedora-cloud/docker-brew-fedora/blob/35/x86_64/fedora-35-x86_64.tar.xz

If you add your custom resolv.conf and a /etc/wsl.conf to the tarball with

[network]
generateResolvConf = false

then you pass that the new tar.xz to

podman machine init --image-path custom.tar.xz

I'm not completely sure if those files will survive, but worth a try.

I patched the tar.xz with wsl.conf, but then get this

podman machine init --image-path C:\Users\myname\fedora.tar.xz Error: open C:\Users\myname\.local\share\containers\podman\machine\wsl: is a directory

So I tried intercepting the init by pasting the tar (with same name) into
C:\Users\myname.local\share\containers\podman\machine\wsl
as soon as extract completed - but I suspect there is a CRC check or similar as it then responded:

podman machine init Downloading VM image: fedora-35-x86_64.tar.xz: done Extracting compressed file
Paste of patched file named podman-machine-default_fedora-35-x86_64.tar
Importing operating system into WSL (this may take 5+ minutes on a new WSL install)... Installing packages (this will take awhile)... Error: package upgrade on guest OS failed: exit status 1

Next was to create a fast node server, and serve the file that way, which gave the same failure.
I added the wsl.conf by running wsl Ubuntu extracting the Fedora image, creating the file, then compressing again like this
tar fcz fedora.tar.gz -C / home/fed/
I think that the approach has broken the image as I cannot even access it using
wsl -d podman-machine-default

podman machine init --image-path http://localhost:9000/fedora.tar.xz Downloading VM image: fedora.tar.xz: done Extracting compressed file Importing operating system into WSL (this may take 5+ minutes on a new WSL install)... Installing packages (this will take awhile)... Error: package upgrade on guest OS failed: exit status 1

about 1 hour later
latest attempt - I managed to append the wsl.conf file into the existing archive, this worked better for init, but failed at the resolving of mirror again:
`podman machine init --image-path http://localhost:9000/fedora-35-x86_64.tar.xz
Downloading VM image: fedora-35-x86_64.tar.xz: done
Extracting compressed file
Importing operating system into WSL (this may take 5+ minutes on a new WSL install)...
Installing packages (this will take awhile)...
Failed to set locale, defaulting to C.UTF-8
Fedora 35 - x86_64 0.0 B/s | 0 B 02:00
Errors during downloading metadata for repository 'fedora':

Could you guys check the following

cat /etc/resolv.conf (before patching it)
cat /etc/resolv.conf Generated by NetworkManager nameserver 192.168.122.1
Output of ifconfig (you will need to dns install net-tools first - after patching resolv.conf)
patched resolv.conf with namserver 8.8.8.8 then run:
yum install net-tools
success
ifconfig

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.189.115  netmask 255.255.240.0  broadcast 172.17.191.255
        inet6 fe80::215:5dff:fedb:2911  prefixlen 64  scopeid 0x20<link>
        ether 00:15:5d:db:29:11  txqueuelen 1000  (Ethernet)
        RX packets 93478  bytes 140812597 (134.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 15353  bytes 1066005 (1.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Output of netstat -rn
netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 172.17.176.1 0.0.0.0 UG 0 0 0 eth0 172.17.176.0 0.0.0.0 255.255.240.0 U 0 0 0 eth0

On a Windows powershell prompt: Output of ipconfig
ipconfig

Windows IP Configuration
Unknown adapter Appgate SDP:
   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
Ethernet adapter Ethernet 2:
   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
Ethernet adapter Ethernet 4:
   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
Ethernet adapter vEthernet (WSL):
   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::94e:53a4:ab7b:9f77%76
   IPv4 Address. . . . . . . . . . . : 172.17.176.1
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . :
Unknown adapter OpenVPN TAP-Windows6:
   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
Wireless LAN adapter Local Area Connection* 1:
   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
Wireless LAN adapter Local Area Connection* 2:
   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
Ethernet adapter Ethernet 3:
   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
Wireless LAN adapter Wi-Fi:
   Connection-specific DNS Suffix  . : Lynx9000
   Link-local IPv6 Address . . . . . : fe80::2811:2e09:b5a6:b56c%6
   IPv4 Address. . . . . . . . . . . : 192.168.38.109
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.38.1
Ethernet adapter Bluetooth Network Connection:
   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
Tunnel adapter Teredo Tunneling Pseudo-Interface:
   Connection-specific DNS Suffix  . :
   IPv6 Address. . . . . . . . . . . : 2001:0:284a:364:4d6:2c72:aae4:674f
   Link-local IPv6 Address . . . . . : fe80::4d6:2c72:aae4:674f%14
   Default Gateway . . . . . . . . . : ::

@JerryMWeeks
Copy link

JerryMWeeks commented Jun 20, 2022

I found an alternative way - for those urgently needing to get Podman to run.

  1. Run podman machine init - let it run until it fails
  2. Run wsl --shutdown (to make sure the failed podman install has stopped)
  3. Go to Azure or AWS, spin up a VM - Win 10 or whatever your os is. (Azure offers a free trial - so you can do this without cost!) Update: Make sure the login on the VM is the same as your login on local machine (the ssh cert path inside the Podman default machine will fail otherwise)
  4. Login to your new VM
  5. Run PowerShell admin
  6. Run WSL --install (the latest way to install wsl)
  7. Reboot to complete the install - on reboot Ubuntu should complete loading, you might need to wait for that to complete
  8. Download and install podman
  9. run podman machine init
  10. allow setup to complete
  11. run podman machine start
  12. observe if start is successful
  13. if yes, run podman machine stop
  14. 7zip up c:\users<remote login name>.local and .config and .ssh (you might need to enable viewing hidden / system files)
  15. Copy the .7z file (or zip if you used zip)
  16. Paste in your local laptop at a suitable location (c:\temp\podman) And unzip.
  17. Copy containers.conf from the VM c:\users<your remote login>\AppData\roaming\containers
  18. paste that in the same suitable location (c:\temp\podman)
  19. Edit the json file podman-machine-default.json in the unzipped .config\containers\podman\machine\wsl updating the path in 3 places to match your local machine login / user name (unless you set up your VM in Azure to have the same login name)
  20. e.g. line 9: "C:\Users\remote-vm-login-name\.ssh\podman-machine-default becomes "C:\Users\local-login-name\.ssh\podman-machine-default
  21. Copy .config to c:\users<your local login> replacing / overwriting existing files
  22. Copy .local to c:\users<your local login> replacing / overwriting existing files
  23. Copy .ssh contents to c:\users<your local login>.ssh
  24. copy the edited / saved podman-machine-default.json to c:\users<your local login>\AppData\roaming\containers replacing / overwriting existing file
  25. run podman machine stop (shouldn't do anything - might error)
  26. run podman machine start

and if you get as lucky as me ...

podman machine start
Starting machine "podman-machine-default"

This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command:

       podman machine set --rootful

API forwarding for Docker API clients is not available due to the following startup failures.
       CreateFile \\.\pipe\docker_engine: All pipe instances are busy.

Podman clients are still able to connect.
Machine "podman-machine-default" started successfully

Now to get on with the next step ;-)

This should in principle always work as the podman "disk" is a vhd and the container concept is portable - but I am not responsible if you break something!

@Bindernews
Copy link

Bindernews commented Jun 21, 2022

Here's my workaround, it requires a separate WSL instance (I used ubuntu) and you'll need to run the commands as root so the owner bits are properly saved.

  1. Download the fedora image as mentioned in @JerryMWeeks post - https://github.com/fedora-cloud/docker-brew-fedora/blob/35/x86_64/fedora-35-x86_64.tar.xz
  2. Open WSL instance as root and cd to directory where you downloaded the fedora tar file
  3. mkdir /fed && tar xfJ fedora-35-x86_64.tar.xz -C /fed - Extract the tar
  4. cd /fed
  5. echo -e "nameserver 1.1.1.1\n" >./etc/resolv.conf - Use cloudflare DNS (replace IP as required)
  6. echo -e "[network]\ngenerateResolvConf = false\n" >./etc/wsl.conf - Tell WSL not to mess with our DNS
  7. tar cfJ /fedora.tar.xz * - Rebuild the tar
  8. mv /fedora.tar.xz <windows_folder> - Put it where podman can get at it
  9. (In powershell/cmd) podman machine init --image-path fedora.tar.xz - init podman

And here's a scripty version of it. May contain typos.

#!/bin/bash
# In powershell: cd to where TAR_IN is located, run 'wsl -d <distro> -u root --', then run this script
TAR_IN="fedora-35-x86_64.tar.xz"
TAR_OUT="fedora.tar.xz"
mkdir /fed && tar xfJ "$TAR_IN" -C /fed
pushd /fed
echo -e "nameserver 1.1.1.1\n" >./etc/resolv.conf
echo -e "[network]\ngenerateResolvConf = false\n" >./etc/wsl.conf
tar cvfJ "/$TAR_OUT" *
popd
mv "/$TAR_OUT" .

@davdr
Copy link

davdr commented Jun 23, 2022

I patched the tar.xz with wsl.conf, but then get this

podman machine init --image-path C:\Users\myname\fedora.tar.xz
Error: open C:\Users\myname.local\share\containers\podman\machine\wsl: is a directory

For other people trying this: turns out that this works with --image-path \Users\... but not --image-path C:\Users\....

@nobodyman1
Copy link

nobodyman1 commented Jun 24, 2022

It doesn´t work for me. If I use --image-path \Users\... podman still downloads fedora-35-x86_64.tar.xz to the wsl directory.
This behaviour seems to be normal! #14388
And I think the --image-path is a relative path to %USERPROFILE%\.local\share\containers\podman\machine\wsl.

@davdr
Copy link

davdr commented Jun 24, 2022

Indeed: the --image-path is the path to the file you want podman machine init to use as image for the WSL distro it creates. This is if you want to use your own custom image instead of the default Fedora 35 image that's dowloaded over http, as JerryMWeeks tried to do.

@nobodyman1
Copy link

nobodyman1 commented Jun 24, 2022

But how does it work? There is no documentation available!
Where should I save the template? --image-path C:\Users\... and --image-path \Users\... are not working!
Did you try it?

@davdr
Copy link

davdr commented Jun 24, 2022

@nobodyman1 it's documented here: https://docs.podman.io/en/latest/markdown/podman-machine-init.1.html and yes, I've already used it in another context

@nobodyman1
Copy link

nobodyman1 commented Jun 24, 2022

Thanks a lot. Now it works with --image-path \Users\... and --image-path /Users/.. (but not if a drive letter is specified).
The reason was simple: I had an additional parameter in my commandline which was wrong!
I already looked at the documentation but it´s not up to date ("Podman on macOS"). There might be some additional hints for Windows users.

@majidaldo
Copy link

Here's my workaround, it requires a separate WSL instance (I used ubuntu) and you'll need to run the commands as root so the owner bits are properly saved.

  1. Download the fedora image as mentioned in @JerryMWeeks post - https://github.com/fedora-cloud/docker-brew-fedora/blob/35/x86_64/fedora-35-x86_64.tar.xz
  2. Open WSL instance as root and cd to directory where you downloaded the fedora tar file
  3. mkdir /fed && tar xfJ fedora-35-x86_64.tar.xz -C /fed - Extract the tar
  4. cd /fed
  5. echo -e "nameserver 1.1.1.1\n" >./etc/resolv.conf - Use cloudflare DNS (replace IP as required)
  6. echo -e "[network]\ngenerateResolvConf = false\n" >./etc/wsl.conf - Tell WSL not to mess with our DNS
  7. tar cfJ /fedora.tar.xz * - Rebuild the tar
  8. mv /fedora.tar.xz <windows_folder> - Put it where podman can get at it
  9. (In powershell/cmd) podman machine init --image-path fedora.tar.xz - init podman

And here's a scripty version of it. May contain typos.

#!/bin/bash
# In powershell: cd to where TAR_IN is located, run 'wsl -d <distro> -u root --', then run this script
TAR_IN="fedora-35-x86_64.tar.xz"
TAR_OUT="fedora.tar.xz"
mkdir /fed && tar xfJ "$TAR_IN" -C /fed
pushd /fed
echo -e "nameserver 1.1.1.1\n" >./etc/resolv.conf
echo -e "[network]\ngenerateResolvConf = false\n" >./etc/wsl.conf
tar cvfJ "/$TAR_OUT" *
popd
mv "/$TAR_OUT" .

need the echo -e "[network]\ngenerateResolvConf = false\n" >./etc/wsl.conf in the script.

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@n1hility
Copy link
Member

podman 4.2 (just released) now fetches a prebuilt image with no package downloads. If you run into a routing or firewall issue that prevents wsl from functioning, machine init should still succeed. You can use podman machine ssh or the wsl prompt to provide custom dns settings after the init process completes.

@rhatdan
Copy link
Member

rhatdan commented Aug 15, 2022

@n1hility Can we close this issue?

@n1hility
Copy link
Member

@rhatdan Yes I think so. There is a follow-up feature request we could potentially do that allowed a user to provide a custom dns server on init vs manually editing wsl.conf and resolv.conf. This issue has gone on pretty long though, so probably makes sense to split that into another issue.

@majidaldo
Copy link

@rhatdan Yes I think so. There is a follow-up feature request we could potentially do that allowed a user to provide a custom dns server on init vs manually editing wsl.conf and resolv.conf. This issue has gone on pretty long though, so probably makes sense to split that into another issue.

please

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Aug 24, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 24, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. machine stale-issue windows issue/bug on Windows
Projects
None yet
Development

No branches or pull requests