DietPi-Software | WireGuard + Pi-hole

[MichaIng]

To allow DNS request though WireGuard, Pi-hole needs to be configured to allow DNS requests from all interfaces (option 1 on settings > DNS).

Somehow on raw Raspbian this is not required, with otherwise identical WireGuard + Pi-hole settings, which is actually strange since wg0 != eth0/wlan0.

Thanks to @johnnyt83 for the testing session to point this out.

• Test on VM
  • Verified "Listen on all interfaces" being required.
• Test with dhcpcd taking care of interfaces (instead of /etc/network/interfaces)
  • Doesn't make a difference on DietPi, "Listen on all interfaces" still required.
  • Connection from inside or outside local network does not make a difference, as expected.
• Add result to online docs or configure this automatically when WireGuard + Pi-hole installs are detected. EDIT: Added to WireGuard + OpenVPN + Pi-hole online docs for now
  • Setting can be set in our install code: pihole -a -i all
  • Does pihole -a -i wg0 work as well? Nope!
  • Better is pihole -a -i local to not allow any requests from www. Although the VPNs task here is exactly to allow requests without opening port 53 to www, so it doesn't matter that much 😉.

[MichaIng]

@johnnyt83
What is also strange in your case, that when DNS requests are limited to eth0, the app you tested was still able to load ads. I just tested and in my case domains simply fail to load, which makes sense since DNS requests fail (in browser). If you find time you could verify that blocked domains fail in browser or load to blocking page being shown.

Only idea is that the app is able to use an alternative DNS server, if the WireGuard configured one fails, which is not optimal, but I guess possible. But the DNS requests should still be tunnelled, which is definitely forced by the WireGuard app when configure so (0.0.0.0/0, ::0).

---

Another question:

• I remember on your Raspbian that dhcpcd was running with some command arguments/options.
• On my test, with default install, is runs plain without any arguments.
• If you find time, could you check htop about possible arguments of dhcpcd?

---

Next I remember you had the Pi-hole DHCP server enabled, while having a router that already serves as DHCP server.

• I played around with this and it does not make any difference, nor harm, as long as the IP ranges do no overlap at least.
• However just as a suggestion: Better just have one DHCP server in the network. If you router can do that, disable the one from Pi-hole to reduce the risk of network issues and load from the Pi.
• Ah or was the DHCP server actually in use? Can be checked by the DHCP lease table. Perhaps this makes a difference, e.g. that Pi-hole by default allow requests from from the local network that it handles DHCP for. Possibly the switch to DietPi SDcard, even with identical settings, did not make all DHCP clients automatically re-lease their IPs, so Pi-hole does not get known with them. Possible requires one with more network knowledge, as I am also tapping in the blue about this, what can have an effect on this topic and what can be ruled out.

[MichaIng]

Further research that actually prove what the default behaviour should be (as is on DietPi):

• https://discourse.pi-hole.net/t/listen-on-all-interfaces-using-pihole-with-wireguard-or-other-vpn/19484
• https://discourse.pi-hole.net/t/wireguard-blocking-dns-requests-from-other-machines/19481

Still no idea why on Raspbian this is not required. Perhaps some RPi network setup that I did not yet find, loops requests from wg0 in a way that Pi-hole/RPi receives them as coming from either localhost or eth0. The masquerade iptables forwarding of course has this affect (basically NAT) for all outdoing requests (leaving eth0), but for requests to WireGuard server device itself this should not be the case and would be an overhead.

And now I have an idea:

• Perhaps hostname and/or hosts file entries have an effect on how/where incoming requests are routed/forwarded...
• @johnnyt83 next task, if you find time to check on Raspbian: cat /etc/hostname; cat /etc/hosts, you can make content anonym by replacing the actual IP/hostname with <RPi-IP> <RPi-hostname> or such.

[Joulinar]

Hi Guys,

I'm using PiHole + WireGuard as well. To have DNS working on my mobile devices while using WireGuard, I switched on Listen on all interfaces within PiHole settings. Usually it works fine but I noticed some strange behaviour after restarting my RPi. Once restarted, PiHole seems not working correctly as I'm not able to open webpages on my local computer because DNS resolution is not working.

nslookup google.de
DNS request timed out.
    timeout was 2 seconds.
Server:  UnKnown
Address:  192.168.0.11

DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.

Within PiHole log, I see request from localhost only. No other request from other devices. It seems PiHole is not accepting request from interface eth0 as well as wg0. To get it back working, I need to open PiHole Admin webpage via local IP address http://192.168.0.11/admin. Immediately after opening the webpage, I can see DNS queries within the log and DNS queries getting answered again.

nslookup google.de
Server:  wpad.lan
Address:  192.168.0.11

Non-authoritative answer:
Name:    google.de
Addresses:  2a00:1450:4001:824::2003
          172.217.23.163

I don’t know why but a trigger is needed to get PiHole listing on all interfaces after reboot ☹

[MichaIng]

@Joulinar
Many thanks for reporting. Can you please try the following:

• Reboot
• Try to resolve any domain on the RPi itself, e.g. wget --spider dietpi.com
• If that works, try it from other clients that use this Pi-hole.
• If that does not work, could you try to ping the RPi and retry to use Pi-hole/DNS from clients.
• Probably the network adapter is in some kind of idle/energy savings more which does not wake on DNS requests but on TCP/80 requests and probably pings only.
• You could also install some tool to monitor the requests arriving the RPi, e.g. apt install iptraf-ng && iptraf-ng then go to "all interfaces", try DNS resolving from some client and monitor if those port 53 requests from the client are listed or not.
• If the packets/requests arrive the RPi, but then for sure Pi-hole is simply ignoring them, until website is opened, then I guess it is a Pi-hole but the we should report: https://github.com/pi-hole/pi-hole/issues

[Joulinar]

@MichaIng I did as you suggested. I rebooted my RPi and I could resolve DNS queries on the Pi itself but not from other clients in my LAN. I executed the following steps:

• Reboot RPi > done
• resolve domain from Win client > not working
• SSH to RPi > working
• resolve domain on the RPi itself > working
• resolve domain from Win client > not working
• ping to RPi > working
• resolve domain from Win client > not working
• telnet on port 80 from Win Client to RPi > working
• resolve domain from Win client > not working
• opening PiHole Admin Web Interface from Win Client > working
• resolve domain from Win client > working

To check the interface I installed iptraf-ng and it seems packages are arriving on port 53 but get ignored by PiHole.

Traffic from Win Client to PRi after reboot

x UDP (59 bytes) from 192.168.0.134:61604 to 192.168.0.11:53 on eth0
x UDP (72 bytes) from 192.168.0.134:52033 to 192.168.0.11:53 on eth0 
x UDP (72 bytes) from 192.168.0.134:52033 to 192.168.0.11:53 on eth0
x UDP (72 bytes) from 192.168.0.134:52033 to 192.168.0.11:53 on eth0
x UDP (72 bytes) from 192.168.0.134:52033 to 192.168.0.11:53 on eth0
x UDP (59 bytes) from 192.168.0.134:61604 to 192.168.0.11:53 on eth0
x UDP (72 bytes) from 192.168.0.134:52033 to 192.168.0.11:53 on eth0

Traffic on the RPi itself

x UDP (54 bytes) from 127.0.0.1:48323 to 127.0.0.1:53 on lo 
x UDP (54 bytes) from 127.0.0.1:48323 to 127.0.0.1:53 on lo  
x UDP (54 bytes) from 192.168.0.11:14761 to 1.0.0.1:53 on eth0
x UDP (70 bytes) from 1.0.0.1:53 to 192.168.0.11:14761 on eth0

And finally after accessing PiHole Admin Interface

x UDP (75 bytes) from 192.168.0.134:54769 to 192.168.0.11:53 on eth0  
x UDP (71 bytes) from 192.168.0.134:54772 to 192.168.0.11:53 on eth0 
x UDP (93 bytes) from 192.168.0.11:53 to 192.168.0.134:54772 on eth0 
x UDP (58 bytes) from 192.168.0.134:54773 to 192.168.0.11:53 on eth0
x UDP (58 bytes) from 192.168.0.11:55843 to 1.0.0.1:53 on eth0
x UDP (133 bytes) from 1.0.0.1:53 to 192.168.0.11:55843 on eth0
x UDP (133 bytes) from 192.168.0.11:53 to 192.168.0.134:54773 on eth0
x UDP (58 bytes) from 192.168.0.134:54774 to 192.168.0.11:53 on eth0 
x UDP (58 bytes) from 192.168.0.11:23044 to 1.0.0.1:53 on eth0
x UDP (133 bytes) from 1.0.0.1:53 to 192.168.0.11:23044 on eth0
x UDP (58 bytes) from 192.168.0.11:53 to 192.168.0.134:54774 on eth0
x UDP (54 bytes) from 192.168.0.134:54775 to 192.168.0.11:53 on eth0   

What I noticed as well, the RPi is using 127.0.0.1 to resolve his own DNS requests. Means the lo interface and not eth0 interface. Maybe that's why the PRi is able resolve DNS queries but other clients not.

Seems like an issue on PiHole as it's not picking up request on port 53 on all interfaces.

I opened a ticket with PiHole pi-hole/pi-hole#2898

[MichaIng]

@Joulinar
Many thanks for doing this test. So indeed this sound like something that we need to forward to Pi-hole devs. EDIT: Ah you did, many thanks 👍!

What I noticed as well, the RPi is using 127.0.0.1 to resolve his own DNS requests.

That is expected, otherwise Pi-hole ads blocking would not be done for the RPi itself. Pi-hole is an own DNS provider (the pihole-FTL service, which is a dnsmasq wrapper). You configure upstream DNS servers in the Pi-hole admin panel. So all DNS requests should be send to pihole-FTL, listening on 127.0.0.1 and eth0 port 53 requests. Pi-hole then does it's magic by filtering bad ads requests and answering the DNS request with 0.0.0.0 or 127.0.0.1/local IP, based on whether blocking page is enabled or not. All requests that pass the filters/blocklists, are forwarded to the configured upstream DNS and the resolved IP forwarded back to the requesting client.

[AzureMarker]

If the packets/requests arrive the RPi, but then for sure Pi-hole is simply ignoring them, until website is opened, then I guess it is a Pi-hole but the we should report: https://github.com/pi-hole/pi-hole/issues

Based @Joulinar's last comment, this is not the case. The device does not receive any requests until something is triggered by opening the web interface. Once that trigger occurs, the requests are able to go through. Pi-hole is not ignoring any requests, the requests are being stopped by some network function.

The only way I can see this trigger happening is if the system is misconfigured in such a way as to trigger this line:
https://github.com/pi-hole/pi-hole/blob/1a741f696eec2ee148fc59bc0dc98e862475e6ef/pihole#L293

The web interface gets the status by calling pihole status web. If the Pi-hole is missing a critical status config line, pihole enable will be called. That command may trigger the DNS server to be restarted. This restart may fix the networking issue.

However, the misconfiguration would have been fixed by calling pihole enable. If it isn't, then either the enable call never succeeds or the file system resets itself after a reboot.

[MichaIng]

@Mcat12 @Joulinar
Maybe I misinterpreted Traffic from Win Client to PRi after reboot and Traffic on the RPi itself. Were the UDP (XY bytes) from 192.168.0.134:<port> to 192.168.0.11:53 only visible on the client or on the RPi as well?

• If they are shown on the client only, but e.g. iptraf-ng on the RPi does not list these requests, then of course it is a network issue an not related to Pi-hole.
• If they are visible on the RPi client, but Pi-hole is not reacting to them, then it most likely is a Pi-hole issue.

If indeed the port 53 requests do not reach the Pi, then I am wondering why port 80 requests (web interface) do? The only difference I can imagine is UDP vs TCP.

However, the misconfiguration would have been fixed by calling pihole enable. If it isn't, then either the enable call never succeeds or the file system resets itself after a reboot.

If the file system resets itself, then this should be visible by other issues. However easy to test. Run and paste the following once when DNS resolving does not work (yet), once after accessing the web UI, once after another reboot:

cat /etc/pihole/setupVars.conf

That covers all possible relation with config and pihole enable being even called and boot persistent.

---

And just another hint, since this topic initially was about using Pi-hole through a WireGuard VPN connection, as of the request interfaces and IPs, this is all outside of any VPN client, so not related to WireGuard.

[Joulinar]

@Mcat12 @MichaIng

just to avoid any confusion, The whole capturing was done on my RPi.

x UDP (59 bytes) from 192.168.0.134:61604 to 192.168.0.11:53 on eth0
x UDP (72 bytes) from 192.168.0.134:52033 to 192.168.0.11:53 on eth0 
x UDP (72 bytes) from 192.168.0.134:52033 to 192.168.0.11:53 on eth0
x UDP (72 bytes) from 192.168.0.134:52033 to 192.168.0.11:53 on eth0
x UDP (72 bytes) from 192.168.0.134:52033 to 192.168.0.11:53 on eth0
x UDP (59 bytes) from 192.168.0.134:61604 to 192.168.0.11:53 on eth0
x UDP (72 bytes) from 192.168.0.134:52033 to 192.168.0.11:53 on eth0

For me the picture looks like, my client 192.168.0.134 is sending packages to my RPi 192.168.0.11 on Port 53 on eth0. Means packages are received on the RPi and nothing is happening afterwards.

I did some more investigation on the topic. If I'm switching PiHole configuration from Listen on all interfaces to Listen only on interface eth0 and do a reboot, everything is working fine. However, if I switch back from Listen only on interface eth0 to Listen on all interfaces, the issue is back again.

Interesting point: If I'm running pihole status on my RPi after reboot, it seems fixing the issue as well.

[MichaIng]

@Joulinar

just to avoid any confusion, The whole capturing was done on my RPi.

Okay, so I got it right 👍. But just to sort it out, could you paste cat /etc/pihole/setupVars.conf after you did a reboot?

I did some more investigation on the topic. If I'm switching PiHole configuration from Listen on all interfaces to Listen only on interface eth0 and do a reboot, everything is working fine.

That is actually an interesting point and the way where Pi-hole indeed can/does ignore requests.

@Mcat12

Pi-hole is not ignoring any requests, the requests are being stopped by some network function.

As of above, Pi-hole has a mechanism to ignore requests based on conditions. And thanks to @Joulinar testing we are coming closer to which condition seems to fail until a web access is done. Probably Pi-hole is not identifying the client as a local network one? pihole enable and/or the DNS service restart can of course still be the fix.

I guess it can be hard to replicate since most likely it depends on the particular network setup, but I will run some test here as well with different network conditions (hosts entries, DHCP and things like that). We can do that more targeted with some details about how Pi-hole/dnsmasq does this check.

From the code: https://github.com/pi-hole/pi-hole/blob/development/advanced/Scripts/webpage.sh#L200
dnsmasq man page: https://manpages.debian.org/buster/dnsmasq-base-lua/dnsmasq.8.en.html

--local-service
Accept DNS queries only from hosts whose address is on a local subnet, ie a subnet for which an interface exists on the server. This option only has effect if there are no --interface, --except-interface, --listen-address or --auth-server options. It is intended to be set as a default on installation, to allow unconfigured installations to be useful but also safe from being used for DNS amplification attacks.

What I see now, is that there is one DNS query related dnsmasq setting left: https://github.com/pi-hole/pi-hole/blob/development/advanced/Scripts/webpage.sh#L191-L193

• interface and local-service are removed, but except-interface=nonexisting is left, if present once.
• dnsmasq man does not give a hint, but either it does not allow requests from interfaces that are not present/accessible/configured on the server, or it excludes the interface "nonexisting", which cannot exist if one did not hack the interface names, thus the setting does not exclude anything.
• BUT, even if as above, the setting does not explicitly exclude any existing interface, according to the man page it unsets local-service: "only has effect if there are no ... --except-interface ... options"
  • If so, then indeed all interfaces would stay allowed, which would not cause the issue this topic is about, but would not be desired as well for security reason.
• AND if "nonexisting" indeed blocks all interfaces that are not visible/known/accessible/whatever to the server/dnsmasq, then I could imaging it might have an effect, if pihole-FTL starts before the interface has been configured. At least the sysvinit service is not forced to start after or even require network to be configured first: https://github.com/pi-hole/pi-hole/blob/master/advanced/Templates/pihole-FTL.service#L4-L5
• Ah but we (DietPi) actually replace the syslog requirement with network: https://github.com/MichaIng/DietPi/blob/dev/dietpi/dietpi-software#L8861-L8862
  • This is due to DietPi by default comes with systemd-journald as syslog facility only, which seems to not allow pihole-FTL to start. However didn't test for a while, actually current systemd generators should translate this a way to include systemd-journald as well.

---

Testing schedule

Assure that any conflicting dnsmasq setting is removed and re-add local-service:

for i in /etc/dnsmasq{,.d/*}.conf; do
sed -Ei '/^(local-service|interface=|except-interface=|listen-address=|auth-server=)/d' $i
done
echo 'local-service' >> /etc/dnsmasq.d/01-pihole.conf
reboot
# Try to resolve host from client

Start pihole-FTL manually after reboot

update-rc.d -f pihole-FTL remove
systemctl disable pihole-FTL
reboot
systemctl start pihole-FTL
# Try to resolve host from client
# Re-enable pihole-FTL autostart
systemctl enable pihole-FTL

Allow DNS pubic queries

pihole -a -i all
reboot
# Try to resolve host from client

Test public DNS requests when switching "all" to "local" (not related to this issue)

pihole -a -i local
# Try to resolve host from remote client after opening/forwarding port 53

[Joulinar]

@MichaIng below the output of cat /etc/pihole/setupVars.conf after RPi reboot.

root@DietPi:~# cat /etc/pihole/setupVars.conf
WEBPASSWORD=360xxx
PIHOLE_INTERFACE=eth0
IPV4_ADDRESS=192.168.0.11/24
IPV6_ADDRESS=2003:xxx
QUERY_LOGGING=true
INSTALL_WEB_SERVER=false
INSTALL_WEB_INTERFACE=true
LIGHTTPD_ENABLED=false
BLOCKING_ENABLED=true
DHCP_ACTIVE=true
DHCP_START=192.168.0.20
DHCP_END=192.168.0.149
DHCP_ROUTER=192.168.0.1
DHCP_LEASETIME=168
PIHOLE_DOMAIN=lan
DHCP_IPv6=false
DHCP_rapid_commit=false
DNSMASQ_LISTENING=local
PIHOLE_DNS_1=8.8.8.8
PIHOLE_DNS_2=8.8.4.4
PIHOLE_DNS_3=2001:4860:4860:0:0:0:0:8888
PIHOLE_DNS_4=2001:4860:4860:0:0:0:0:8844
PIHOLE_DNS_5=1.1.1.1
PIHOLE_DNS_6=1.0.0.1
PIHOLE_DNS_7=2606:4700:4700::1111
PIHOLE_DNS_8=2606:4700:4700::1001
DNS_FQDN_REQUIRED=true
DNS_BOGUS_PRIV=true
DNSSEC=false
CONDITIONAL_FORWARDING=false
root@DietPi:~#

but it did not changed, once PiHole is working again. Pls let me know if I should do some further testing.

[MichaIng]

@Joulinar
Thanks, that proves that it is not due to settings/file reset, those are as expected. So pihole enable should not run on web access, since BLOCKING_ENABLED=true is present. Whatever triggers DNS request handling on web access, it must be something different.

Further testing can be done as of "Testing schedule" above.

[Joulinar]

@MichaIng

I did the tests as described above. Below you can find the results:

Test 1

• No effect
• DNS couldn’t be resolved on client

Test 2

• Reboot takes quite long (just a feeling, as it’s headless system)
• Starting the service manual seems working
• Switching back to “auto start”, is reintroducing the issue

Test 3

• Seems fixing the issue
• DNS could be resolved on client after reboot without manual intervention

Test 4

• Setting the value and did a reboot
• DNS couldn’t be resolved on client

For me looks like the issue is with the PiHole setting Listen on all interfaces

[MichaIng]

@Joulinar
Many thanks for running the tests.

Test 4

Okay so it seems that the remaining except-interface=nonexisting dnsmasq setting is not causing local-service to be ignored. However it should be removed regardless when switching settings, to have things cleaned and being failsafe.
EDIT: Just tested and I've overseen something as except-interface=nonexisting DOES get removed when running pihole -a -i local. So everything fine about that. As well not other conflicting settings are present/active by default.

For me looks like the issue is with the PiHole setting Listen on all interfaces

Jep, in combination with most likely too early service start on boot.
I just checked in my case:

journalctl
...
Sep 10 14:19:52 VM-Buster systemd[1]: Started ifup for eth0.
Sep 10 14:19:52 VM-Buster systemd[1]: Started Raise network interfaces.
Sep 10 14:19:52 VM-Buster systemd[1]: Reached target Network.
Sep 10 14:19:52 VM-Buster systemd[1]: Starting DietPi-Boot...
Sep 10 14:19:52 VM-Buster systemd[1]: Starting Permit User Sessions...
Sep 10 14:19:52 VM-Buster systemd[1]: Reached target Network is Online.
Sep 10 14:19:52 VM-Buster systemd[1]: Starting LSB: pihole-FTL daemon...
Sep 10 14:19:52 VM-Buster systemd[1]: Started Permit User Sessions.
Sep 10 14:19:52 VM-Buster pihole-FTL[371]: Not running
Sep 10 14:19:52 VM-Buster sh[306]: eth0=eth0
Sep 10 14:19:53 VM-Buster su[467]: (to pihole) root on none
Sep 10 14:19:53 VM-Buster su[467]: pam_unix(su:session): session opened for user pihole by (uid=0)
Sep 10 14:19:53 VM-Buster pihole-FTL[371]: FTL started!

• pihole-FTL starts after eth0 being initialised and client DNS resolution works well with pihole -a -i local.

[Joulinar]

@MichaIng

Seems eth0 is started way after PiHole on my RPi

Sep 10 21:11:20 DietPi systemd[1]: Reached target Basic System.
Sep 10 21:11:20 DietPi systemd[1]: Starting dhcpcd on all interfaces...
Sep 10 21:11:20 DietPi systemd[1]: Starting WPA supplicant...
Sep 10 21:11:20 DietPi systemd[1]: Starting Disable WiFi if country not set...
Sep 10 21:11:20 DietPi systemd[1]: Starting DietPi-RAMlog...
Sep 10 21:11:20 DietPi dhcpcd[279]: Not running dhcpcd because /etc/network/interfaces
Sep 10 21:11:20 DietPi dhcpcd[279]: defines some interfaces that will use a
Sep 10 21:11:20 DietPi dhcpcd[279]: DHCP client or static address
Sep 10 21:11:20 DietPi systemd[1]: Starting LSB: pihole-FTL daemon...
Sep 10 21:11:20 DietPi systemd[1]: Starting DietPi-RAMdisk...
Sep 10 21:11:20 DietPi systemd[1]: Started Clean PHP session files every 30 mins.
Sep 10 21:11:20 DietPi systemd[1]: Reached target Timers.
Sep 10 21:11:20 DietPi systemd[1]: dhcpcd.service: Control process exited, code=exited status=6
Sep 10 21:11:20 DietPi systemd[1]: Failed to start dhcpcd on all interfaces.
Sep 10 21:11:20 DietPi systemd[1]: dhcpcd.service: Unit entered failed state.
Sep 10 21:11:20 DietPi systemd[1]: dhcpcd.service: Failed with result 'exit-code'.
Sep 10 21:11:20 DietPi systemd[1]: Started Disable WiFi if country not set.
Sep 10 21:11:20 DietPi pihole-FTL[285]: Not running
Sep 10 21:11:21 DietPi wpa_supplicant[280]: dbus: Could not acquire the system bus: org.freedesktop.DBus.Error.FileNotFound - Failed to connect
to socket /var/run/dbus/system_bus_socket: No such file or directory
Sep 10 21:11:21 DietPi wpa_supplicant[280]: Failed to initialize wpa_supplicant
Sep 10 21:11:21 DietPi systemd[1]: wpa_supplicant.service: Main process exited, code=exited, status=255/n/a
Sep 10 21:11:21 DietPi systemd[1]: Failed to start WPA supplicant.
Sep 10 21:11:21 DietPi systemd[1]: wpa_supplicant.service: Unit entered failed state.
Sep 10 21:11:21 DietPi systemd[1]: wpa_supplicant.service: Failed with result 'exit-code'.
Sep 10 21:11:21 DietPi systemd[1]: Started DietPi-RAMlog.
Sep 10 21:11:21 DietPi systemd[1]: Started DietPi-RAMdisk.
Sep 10 21:11:21 DietPi systemd[1]: Starting DietPi-PreBoot...
Sep 10 21:11:21 DietPi su[358]: Successful su for pihole by root
Sep 10 21:11:21 DietPi su[358]: + ??? root:pihole
Sep 10 21:11:21 DietPi su[358]: pam_unix(su:session): session opened for user pihole by (uid=0)
Sep 10 21:11:22 DietPi pihole-FTL[285]: FTL started!
Sep 10 21:11:22 DietPi su[358]: pam_unix(su:session): session closed for user pihole
Sep 10 21:11:22 DietPi systemd[1]: Started LSB: pihole-FTL daemon.
Sep 10 21:11:22 DietPi haveged[276]: haveged: ver: 1.9.1; arch: generic; vend: ; build: (gcc 6.3.0 CTV); collect: 128K
Sep 10 21:11:22 DietPi haveged[276]: haveged: cpu: (VC); data: 16K (D); inst: 16K (D); idx: 10/40; sz: 15688/72592
Sep 10 21:11:22 DietPi haveged[276]: haveged: tot tests(BA8): A:1/1 B:1/1 continuous tests(B):  last entropy estimate 7.99854
Sep 10 21:11:22 DietPi haveged[276]: haveged: fills: 0, generated: 0
Sep 10 21:11:22 DietPi kernel: random: crng init done
Sep 10 21:11:22 DietPi kernel: random: 7 urandom warning(s) missed due to ratelimiting
Sep 10 21:11:22 DietPi systemd[1]: Started DietPi-PreBoot.
Sep 10 21:11:22 DietPi systemd[1]: Reached target Network (Pre).
Sep 10 21:11:22 DietPi systemd[1]: Started ifup for eth0.
Sep 10 21:11:22 DietPi systemd[1]: Starting Raise network interfaces...
Sep 10 21:11:22 DietPi kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Sep 10 21:11:22 DietPi kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Sep 10 21:11:23 DietPi sh[504]: eth0=eth0
Sep 10 21:11:23 DietPi systemd[1]: Started Raise network interfaces.
Sep 10 21:11:23 DietPi systemd[1]: Reached target Network.
Sep 10 21:11:23 DietPi systemd[1]: Starting Permit User Sessions...
Sep 10 21:11:23 DietPi systemd[1]: Starting DietPi-Boot...
Sep 10 21:11:23 DietPi systemd[1]: Reached target Network is Online.
Sep 10 21:11:23 DietPi systemd[1]: Starting /etc/rc.local Compatibility...
Sep 10 21:11:23 DietPi systemd[1]: Starting WireGuard via wg-quick(8) for wg0...
Sep 10 21:11:23 DietPi systemd[1]: Started Permit User Sessions.
Sep 10 21:11:23 DietPi systemd[1]: Started /etc/rc.local Compatibility.
Sep 10 21:11:23 DietPi wg-quick[638]: [#] ip link add wg0 type wireguard

But the log looks similar if I switch to pihole -a -i all. And there everything is working fine.

[MichaIng]

@Joulinar
Okay my assumption is then indeed that the underlying dnsmasq identifies "local" clients based on the hosts configured interfaces+IP. Some trigger when accessing the web UI then updates this.

I am just wondering why this is the case since our installer explicitely adds $network to the pihole-FTL sysvinit services start requirement and replaces $syslog with this since it caused issues at least in the past when rsyslog was not installed.

Probably dhcpcd or wpa_supplicant services cause the networking service to be out of order.

To things you can test:

• systemctl disable wpa_supplicant since the networking service includes calling it to configure WiFi. This service is never required on DietPi.
• systemctl disable dhcpcd since this required /etc/network/interfaces to be configured different to have dhcpcd taking control instead of networking.service. This change is btw already implemented with DietPi v6.26 for new installs and as patch during update as well.

Then reboot and see if the order is different.

[Joulinar]

@MichaIng first of all, many thanks for still looking into this. But unfortunetaly it dosn't change anything :/

Anyway I'm now using pihole -a -i all as this is working fine for me after rebooting my RPi

[MichaIng]

@Joulinar
Can you please paste:

systemctl cat pihole-FTL

---

I just checked, at least on Debian Buster, the $network dependency is interpreted/translated by the systemd generator into

After=network-online.target
Wants=network-online.target

In your case this does not seem to be the case, or some dependency/ordering failed and messed it up.
I enabled dhcpcd on my Buster system:

Sep 13 12:02:39 VM-Buster systemd[1]: Starting DHCP Client Daemon...
Sep 13 12:02:39 VM-Buster systemd[1]: Starting DietPi-RAMdisk...
Sep 13 12:02:39 VM-Buster systemd[1]: Started Clean PHP session files every 30 mins.
Sep 13 12:02:39 VM-Buster systemd[1]: Started Daily Cleanup of Temporary Directories.
Sep 13 12:02:39 VM-Buster systemd[1]: Reached target Timers.
Sep 13 12:02:39 VM-Buster systemd[1]: Starting DietPi-RAMlog...
Sep 13 12:02:39 VM-Buster systemd[1]: Found device Virtio network device.
Sep 13 12:02:39 VM-Buster kernel: 8021q: 802.1Q VLAN Support v1.8
Sep 13 12:02:39 VM-Buster systemd[1]: Started DietPi-RAMlog.
Sep 13 12:02:39 VM-Buster dhcpcd[204]: no interfaces have a carrier
Sep 13 12:02:39 VM-Buster dhcpcd[204]: no interfaces have a carrier
Sep 13 12:02:39 VM-Buster dhcpcd[204]: forked to background, child pid 225
Sep 13 12:02:39 VM-Buster dhcpcd[204]: forked to background, child pid 225
Sep 13 12:02:39 VM-Buster systemd[1]: Started DHCP Client Daemon.
Sep 13 12:02:39 VM-Buster dhcpcd[225]: eth0: waiting for carrier
Sep 13 12:02:39 VM-Buster kernel: 8021q: adding VLAN 0 to HW filter on device eth0
Sep 13 12:02:39 VM-Buster systemd[1]: Started DietPi-RAMdisk.
Sep 13 12:02:39 VM-Buster systemd[1]: Starting DietPi-PreBoot...
Sep 13 12:02:40 VM-Buster dhcpcd[225]: eth0: carrier acquired
Sep 13 12:02:40 VM-Buster dhcpcd[225]: DUID XXXX
Sep 13 12:02:40 VM-Buster dhcpcd[225]: eth0: IAID 27:3d:19:0f
Sep 13 12:02:40 VM-Buster dhcpcd[225]: eth0: adding address XXXX
Sep 13 12:02:40 VM-Buster dhcpcd[225]: ipv6_addaddr1: Permission denied
Sep 13 12:02:40 VM-Buster dhcpcd[225]: eth0: probing address 192.168.178.30/24
Sep 13 12:02:40 VM-Buster haveged[202]: haveged: ver: 1.9.1; arch: x86; vend: GenuineIntel; build: (gcc 8.3.0 ITV); collect: 128K
Sep 13 12:02:40 VM-Buster haveged[202]: haveged: cpu: (L4 VC); data: 32K (L4 V); inst: 32K (L4 V); idx: 24/40; sz: 31410/52825
Sep 13 12:02:40 VM-Buster haveged[202]: haveged: tot tests(BA8): A:1/1 B:1/1 continuous tests(B):  last entropy estimate 8.00097
Sep 13 12:02:40 VM-Buster haveged[202]: haveged: fills: 0, generated: 0
Sep 13 12:02:40 VM-Buster kernel: random: crng init done
Sep 13 12:02:40 VM-Buster kernel: random: 7 urandom warning(s) missed due to ratelimiting
Sep 13 12:02:40 VM-Buster systemd[1]: Started DietPi-PreBoot.
Sep 13 12:02:40 VM-Buster systemd[1]: Reached target Network (Pre).
Sep 13 12:02:40 VM-Buster systemd[1]: Started ifup for eth0.
Sep 13 12:02:40 VM-Buster systemd[1]: Starting Raise network interfaces...
Sep 13 12:02:40 VM-Buster systemd[1]: Started Raise network interfaces.
Sep 13 12:02:40 VM-Buster systemd[1]: Reached target Network.
Sep 13 12:02:40 VM-Buster systemd[1]: Starting Permit User Sessions...
Sep 13 12:02:40 VM-Buster systemd[1]: Starting DietPi-Boot...
Sep 13 12:02:40 VM-Buster systemd[1]: Reached target Network is Online.
Sep 13 12:02:40 VM-Buster systemd[1]: Starting LSB: pihole-FTL daemon...
Sep 13 12:02:40 VM-Buster systemd[1]: Started Permit User Sessions.
Sep 13 12:02:40 VM-Buster pihole-FTL[379]: Not running
Sep 13 12:02:40 VM-Buster sh[314]: eth0=eth0
Sep 13 12:02:41 VM-Buster su[473]: (to pihole) root on none
Sep 13 12:02:41 VM-Buster su[473]: pam_unix(su:session): session opened for user pihole by (uid=0)
Sep 13 12:02:41 VM-Buster pihole-FTL[379]: FTL started!
Sep 13 12:02:41 VM-Buster su[473]: pam_unix(su:session): session closed for user pihole
Sep 13 12:02:41 VM-Buster systemd[1]: Started LSB: pihole-FTL daemon.
Sep 13 12:02:41 VM-Buster systemd[1]: Starting Network Time Synchronization...
Sep 13 12:02:41 VM-Buster dhcpcd[225]: eth0: soliciting an IPv6 router

• So pihole-FTL still respects the "Reached target Network is Online." target.
• However what I am really shocked about is that dhcpcd does not respect any other network handler anymore, but just starts and doubles/overwrites what has been configured already or will be for reason configured at a later time. The systemd unit has been crippled to:

root@VM-Buster:~# systemctl cat dhcpcd
# /lib/systemd/system/dhcpcd.service
[Unit]
Description=DHCP Client Daemon
Wants=network.target
Documentation=man:dhcpcd(8)

[Service]
Type=forking
ExecStart=/usr/sbin/dhcpcd
Restart=always

[Install]
WantedBy=multi-user.target

• It does not even set itself Before=network-online.target or after network.target + network-pre.target. What the hack went wrong there?
• On Stretch, there was a sysvinit service only, which did careful tests for other networking services and interface configurations and exited in case. This too simple systemd unit has fatal consequences that "can" break networks when dhcpcd5 is installed as some dependency only, e.g. with Pi-hole...
• Will add some PR: https://salsa.debian.org/smlx-guest/dhcpcd5/blob/debian/sid/debian/dhcpcd5.dhcpcd.service
• Related bug report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820069
  • The old sysvinit service checked /etc/network/interfaces for either "static" or "dhcp" options and in both cases exits (see @Joulinar output). I just checked on my Stretch system, where it checks for "dhcp" only but not for "static" anymore, which is already bad. And on Buster no such check is done which is why dhcpcd simply configures everything when enabled, which is done by default when installed... Not very integrative...

[Joulinar]

@MichaIng I See you put a lot of investigation on it. Not sure if it's still needed but below the output of systemctl cat pihole-FTL

root@DietPi:~# systemctl cat pihole-FTL                                                                                                         # /run/systemd/generator.late/pihole-FTL.service
# Automatically generated by systemd-sysv-generator

[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/pihole-FTL
Description=LSB: pihole-FTL daemon
Before=multi-user.target
Before=multi-user.target
Before=multi-user.target
Before=graphical.target
After=remote-fs.target

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
SuccessExitStatus=5 6
ExecStart=/etc/init.d/pihole-FTL start
ExecStop=/etc/init.d/pihole-FTL stop
ExecReload=/etc/init.d/pihole-FTL reload
root@DietPi:~#

[MichaIng]

@Joulinar
Okay, I was able to replicate:

• pihole -up of course updates the sysvinit service as well, thus reverts our manual adjustment to have network as startup dependency. It does not make much sense to apply custom changes like that in our installer when updates revert them anyway 😅.
• After forcing an update/reconfiguration/checkout, the generated systemd unit matches what you have pasted. After reboot, pihole-FTL.service starts before network has been set up, at least it can.
• And now I can even see the logs where Pi-hole is ignoring a non-local DNS request:

Client

root@VM-Stretch:~# wget --spider dietpi.com
Spider mode enabled. Check if remote file exists.
--2019-09-14 16:36:16--  http://dietpi.com/
Resolving dietpi.com (dietpi.com)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘dietpi.com’

Pi-hole server

Sep 14 16:36:16 dnsmasq[301]: Ignoring query from non-local network

Client after accessing web UI, even without login:

root@VM-Stretch:~# wget --spider dietpi.com
Spider mode enabled. Check if remote file exists.
--2019-09-14 16:38:40--  http://dietpi.com/
Resolving dietpi.com (dietpi.com)... 104.27.179.199, 104.27.178.199, 2606:4700:30::681b:b3c7, ...
Connecting to dietpi.com (dietpi.com)|104.27.179.199|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://dietpi.com/ [following]
Spider mode enabled. Check if remote file exists.
--2019-09-14 16:38:40--  https://dietpi.com/
Connecting to dietpi.com (dietpi.com)|104.27.179.199|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

@Mcat12
I will open a related issue on your side, I think all required testing and replication is done. Since I guess you cannot or wont do changes to the underlying dnsmasq, pihole-FTL needs to start after network interfaces have been set up.

Issue up: pi-hole/pi-hole#2924