Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DietPi Software | PiHole: "Queries over time" show only graph 1 hour #839

Closed
k-plan opened this issue Mar 26, 2017 · 21 comments
Closed

DietPi Software | PiHole: "Queries over time" show only graph 1 hour #839

k-plan opened this issue Mar 26, 2017 · 21 comments
Labels
Milestone

Comments

@k-plan
Copy link
Contributor

k-plan commented Mar 26, 2017

170326-0001

Tested devices:

  • RPi Zero W
  • NanoPi Neo
  • RPi3
  • OrangePi One

DietPi Version:

  • v1.46
  • v1.47

Installed software:

  • SSH Server: Dropbear
  • File Server: None
  • Log System: DietPi-Ramlog Fix for dumb or unset TERM #2
  • Webserver Preference: Lighttpd
  • User Data Location: SD/EMMC | /mnt/dietpi_userdata

Install PiHole via dietpi-software

PiHole Version:

~# pihole -v
::: Pi-hole version is v2.13.2 (Latest version is v2.13.2)
::: Web-Admin version is v2.5.2 (Latest version is v2.5.2)

PiHole logfile:

root@NanoPi-Neo:~# cat /var/log/pihole.log
Mar 26 16:17:01 dnsmasq[9486]: gestartet, Version 2.72, Cachegröße 10000
Mar 26 16:17:01 dnsmasq[9486]: Übersetzungsoptionen: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect
Mar 26 16:17:01 dnsmasq[9486]: Warnung: Ignoriere "resolv-file", weil "no-resolv" aktiv ist
Mar 26 16:17:01 dnsmasq[9486]: Benutze Namensserver 192.168.1.1#53
Mar 26 16:17:01 dnsmasq[9486]: Benutze Namensserver 192.168.1.55#53
Mar 26 16:17:01 dnsmasq[9486]: /etc/hosts gelesen - 6 Adressen
Mar 26 16:17:01 dnsmasq[9486]: /etc/pihole/local.list gelesen - 4 Adressen
Mar 26 16:17:10 dnsmasq[9486]: /etc/pihole/gravity.list gelesen - 347296 Adressen
Mar 26 16:21:41 dnsmasq[9486]: query[A] github.com from 192.168.1.102
Mar 26 16:21:41 dnsmasq[9486]: forwarded github.com to 192.168.1.1
Mar 26 16:21:41 dnsmasq[9486]: forwarded github.com to 192.168.1.55
Mar 26 16:21:41 dnsmasq[9486]: query[AAAA] github.com from 192.168.1.102
Mar 26 16:21:41 dnsmasq[9486]: forwarded github.com to 192.168.1.1
Mar 26 16:21:41 dnsmasq[9486]: forwarded github.com to 192.168.1.55
Mar 26 16:21:41 dnsmasq[9486]: reply github.com is 192.30.253.113
Mar 26 16:21:41 dnsmasq[9486]: reply github.com is 192.30.253.112
.....

Fiddle around a lot, e.g. fresh install pihole, different devices, starting with new images, install by hand or dietpi.txt, Ramlog 1 or 2 or no, but with no success. 😢

I think it is a problem in dietpi-logclear with dnsmasq:

https://github.com/Fourdee/DietPi/blob/master/dietpi/dietpi-logclear#L108

because every hour at xx:17:01 dnsmasq log start new again.

Wondering, why no other DietPi user notice graphing issue in the PiHole web interface ... ❓


BTW: my main productive system, with a old and outdated PiHole installation, it works like a charm ...

170326-0002

@Fourdee
Copy link
Collaborator

Fourdee commented Mar 26, 2017

@k-plan

Thanks for the report 👍

There is an exclusion for PiHole's log file clearing with DietPi-Ramlog. It should allow for 24 hour logs. Let me run some tests here and check source.

edit:

Log System: DietPi-Ramlog #2

Most likely this, Mode 2 probably moving the log contents, before 24h check is done.


Notes:

  • Option 1: exclude PiHole logs from backup in Mode 2
  • Option 2: 24h timer before backup of logs and clear in Mode 2. Chance of loosing <24h of PiHole logs if power failure.

@Fourdee Fourdee added this to the v148 milestone Mar 26, 2017
@k-plan
Copy link
Contributor Author

k-plan commented Mar 26, 2017

@Fourdee

yes, I noticed this, but can't understand or debug, what your script is doing ... sorry for my disability. 😃

On the outdated PiHole installation, with Ramlog 2, graphing will works like a charm.
Noticed as well, that here is no "preamble" in the log-file and it will always start at 00:00:xx :

root@Odroid-C2:~# cat /var/log/pihole.log
Mar 26 00:00:35 dnsmasq[32401]: query[AAAA] apple-mobile.query.yahooapis.com from 192.168.1.112
Mar 26 00:00:35 dnsmasq[32401]: forwarded apple-mobile.query.yahooapis.com to 192.168.1.1
Mar 26 00:00:35 dnsmasq[32401]: forwarded apple-mobile.query.yahooapis.com to 192.168.1.3
Mar 26 00:00:35 dnsmasq[32401]: query[A] apple-mobile.query.yahooapis.com from 192.168.1.112
Mar 26 00:00:35 dnsmasq[32401]: forwarded apple-mobile.query.yahooapis.com to 192.168.1.1
Mar 26 00:00:35 dnsmasq[32401]: query[AAAA] guzzoni.apple.com from 192.168.1.112
Mar 26 00:00:35 dnsmasq[32401]: forwarded guzzoni.apple.com to 192.168.1.1
Mar 26 00:00:35 dnsmasq[32401]: query[A] guzzoni.apple.com from 192.168.1.112
Mar 26 00:00:35 dnsmasq[32401]: forwarded guzzo
.....

@Fourdee
Copy link
Collaborator

Fourdee commented Mar 26, 2017

@k-plan

On the outdated PiHole installation, with Ramlog 2, graphing will works like a charm.

Yep, checked code, looks fine. Shouldn't trigger a clear unless previous day.

Noticed as well, that here is no "preamble" in the log-file and it will always start at 00:00:xx

Thanks, i'll check 👍

@k-plan
Copy link
Contributor Author

k-plan commented Mar 26, 2017

Yep, checked code, looks fine. Shouldn't trigger a clear unless previous day.

The first screenshot is taken from this device (NanoPi v146)

root@NanoPi-Neo:~# uptime
 21:04:54 up 7 days,  8:57,  1 user,  load average: 0,01, 0,04, 0,08

But it's the same, as well on RPi Zero W or OPi One (latest image version) updated to v147,
running Pi-hole v2.13.2 with Admin-LTE v2.5.2

@Fourdee
Copy link
Collaborator

Fourdee commented Mar 27, 2017

Tests with Ramlog mode 2:

  • 🈯️ Confirmed worked on DietPi v142 system. Possible log format has changed in later version of PiHole, which will break scrape. Will test.
  • 🈯️ Fresh install of v147 system
    image

@Fourdee
Copy link
Collaborator

Fourdee commented Mar 27, 2017

@k-plan

Really strange, I cant seem to replicate this on RAMlog mode 1/2. PiHole logs are only cleared if day != today, or 5MB limit.

Just wondering if your PiHole log file is hitting 5MB> in size? in which case, it will be cleared to free up RAMlog (20MB tmpfs max):
https://github.com/Fourdee/DietPi/blob/master/dietpi/dietpi-logclear#L128-L136

Other thing could be locale related, causing a failed scrape. Running en_GB.UTF-8? Others should be fine, as long as en_GB.UTF-8 is still installed.

@k-plan
Copy link
Contributor Author

k-plan commented Mar 27, 2017

@Fourdee

Just wondering if your PiHole log file is hitting 5MB> in size?

No, it's only my personal test device for my single computer.
Rest of the hole network use the old PiHole installation on a other device.

root@NanoPi-Neo:~# ls -lah /var/log/pihole.log
-rwxrwxr-x 1 dnsmasq www-data 102K Mär 27 16:13 /var/log/pihole.log

Other thing could be locale related, causing a failed scrape.

root@NanoPi-Neo:~# locale -a
C
C.UTF-8
de_DE.utf8
en_GB.utf8
POSIX
root@NanoPi-Neo:~# locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

But anyway ... shall we start form the scratch? (hope it is the last time, I do this ...)
Which device? NanoPi Neo (v.1), OPi One or RPi Zero (W)?
Last image version? Update to last DietPi?
RAM-Log 2? May I change locale to de_DE?
Install by hand or full automated?

@k-plan
Copy link
Contributor Author

k-plan commented Mar 27, 2017

@Fourdee

OPi One installation:

  • image: DietPi_v145_NanoPiNEO-armv7-(Jessie).img

Software:

  • RAM Log 2
  • lighttp
  • mc
  • PiHole
root@OPi-One:~# locale -a
C
C.UTF-8
de_DE.utf8
en_GB.utf8
POSIX


root@OPi-One:~# locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=
  • first PiHole installation on OPi-One failed/stop (before dialog for asking to install Admin-LTE), show in terminal Fail in /DietPi/dietpi/dietpi-software line 7xxx (if i remember correct),
    follow by a automated reboot, PiHole was not installed (tested: pihohle -v - not found)
    but PiHole is shown in dietpi-software as installed!

So choose for uninstall and after this run install via dietpi-software again.

After automated reboot, PiHole works.

Setup Pihole:
cp /etc/pihole/adlists.default /etc/pihole/adlists.list
nano /etc/pihole/adlists.list
pihole -g

Tested and work:

root@OPi-One:~# cat /var/log/pihole.log
Mar 27 17:30:16 dnsmasq[1011]: started, version 2.72 cachesize 10000
Mar 27 17:30:16 dnsmasq[1011]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntr$
Mar 27 17:30:16 dnsmasq[1011]: warning: ignoring resolv-file flag because no-resolv is set
Mar 27 17:30:16 dnsmasq[1011]: using nameserver 192.168.1.1#53
Mar 27 17:30:16 dnsmasq[1011]: using nameserver 192.168.1.55#53
Mar 27 17:30:16 dnsmasq[1011]: read /etc/hosts - 2 addresses
Mar 27 17:30:16 dnsmasq[1011]: read /etc/pihole/local.list - 4 addresses
Mar 27 17:30:36 dnsmasq[1011]: read /etc/pihole/gravity.list - 204894 addresses
Mar 27 17:31:09 dnsmasq[1011]: query[A] api.github.com from 127.0.0.1
Mar 27 17:31:09 dnsmasq[1011]: forwarded api.github.com to 192.168.1.1
Mar 27 17:31:09 dnsmasq[1011]: forwarded api.github.com to 192.168.1.55
Mar 27 17:31:09 dnsmasq[1011]: query[AAAA] api.github.com from 127.0.0.1
Mar 27 17:31:09 dnsmasq[1011]: forwarded api.github.com to 192.168.1.1
Mar 27 17:31:09 dnsmasq[1011]: forwarded api.github.com to 192.168.1.55
Mar 27 17:31:09 dnsmasq[1011]: reply api.github.com is 192.30.253.116
Mar 27 17:31:09 dnsmasq[1011]: reply api.github.com is 192.30.253.117
Mar 27 17:31:10 dnsmasq[1011]: query[A] api.github.com from 127.0.0.1
Mar 27 17:31:10 dnsmasq[1011]: cached api.github.com is 192.30.253.117
Mar 27 17:31:10 dnsmasq[1011]: cached api.github.com is 192.30.253.116
Mar 27 17:31:10 dnsmasq[1011]: query[AAAA] api.github.com from 127.0.0.1
....

Graphing in "Queries over time" start working after 10 min.

After 47 min:

170327-0001

root@OPi-One:~# cat /var/log/pihole.log
Mar 27 18:17:02 dnsmasq[5818]: started, version 2.72 cachesize 10000
Mar 27 18:17:02 dnsmasq[5818]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntr$
Mar 27 18:17:02 dnsmasq[5818]: warning: ignoring resolv-file flag because no-resolv is set
Mar 27 18:17:02 dnsmasq[5818]: using nameserver 192.168.1.1#53
Mar 27 18:17:02 dnsmasq[5818]: using nameserver 192.168.1.55#53
Mar 27 18:17:02 dnsmasq[5818]: read /etc/hosts - 2 addresses
Mar 27 18:17:02 dnsmasq[5818]: read /etc/pihole/local.list - 4 addresses
Mar 27 18:17:10 dnsmasq[5818]: read /etc/pihole/gravity.list - 343244 addresses
Mar 27 18:18:01 dnsmasq[5818]: query[A] raw.githubusercontent.com from 192.168.1.50
Mar 27 18:18:01 dnsmasq[5818]: forwarded raw.githubusercontent.com to 192.168.1.1
Mar 27 18:18:01 dnsmasq[5818]: forwarded raw.githubusercontent.com to 192.168.1.55
Mar 27 18:18:01 dnsmasq[5818]: query[AAAA] raw.githubusercontent.com from 192.168.1.50
Mar 27 18:18:01 dnsmasq[5818]: forwarded raw.githubusercontent.com to 192.168.1.1
Mar 27 18:18:01 dnsmasq[5818]: forwarded raw.githubusercontent.com to 192.168.1.55
Mar 27 18:18:01 dnsmasq[5818]: reply raw.githubusercontent.com is <CNAME>
Mar 27 18:18:01 dnsmasq[5818]: reply github.map.fastly.net is 151.101.112.133
Mar 27 18:18:01 dnsmasq[5818]: reply raw.githubusercontent.com is <CNAME>
Mar 27 18:18:01 dnsmasq[5818]: reply github.map.fastly.net is NODATA-IPv6
Mar 27 18:18:30 dnsmasq[5818]: query[A] api.github.com from 192.168.1.102
Mar 27 18:18:30 dnsmasq[5818]: forwarded api.github.com to 192.168.1.1
Mar 27 18:18:30 dnsmasq[5818]: forwarded api.github.com to 192.168.1.55
Mar 27 18:18:30 dnsmasq[5818]: query[AAAA] api.github.com from 192.168.1.102
Mar 27 18:18:30 dnsmasq[5818]: forwarded api.github.com to 192.168.1.55
Mar 27 18:18:30 dnsmasq[5818]: reply api.github.com is 192.30.253.116
Mar 27 18:18:30 dnsmasq[5818]: reply api.github.com is 192.30.253.117
Mar 27 18:19:34 dnsmasq[5818]: query[A] api.github.com from 192.168.1.102
Mar 27 18:19:34 dnsmasq[5818]: forwarded api.github.com to 192.168.1.1
Mar 27 18:19:34 dnsmasq[5818]: forwarded api.github.com to 192.168.1.55
Mar 27 18:19:34 dnsmasq[5818]: query[AAAA] api.github.com from 192.168.1.102
Mar 27 18:19:34 dnsmasq[5818]: forwarded api.github.com to 192.168.1.55
Mar 27 18:19:34 dnsmasq[5818]: reply api.github.com is 192.30.253.117
Mar 27 18:19:34 dnsmasq[5818]: reply api.github.com is 192.30.253.116
....

👎

Now installation on RPi-Zero-W is running ....

@k-plan
Copy link
Contributor Author

k-plan commented Mar 27, 2017

@Fourdee

RPi-Zero-W installation:

  • image: DietPi_v145_RPi-armv6-(Jessie).img

Software:

  • RAM Log 2
  • lighttp
  • mc
  • PiHole
root@RPi-Zero-W:~# locale -a
C
C.UTF-8
de_DE.utf8
en_GB.utf8
POSIX

root@RPi-Zero-W:~# locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=
  • PiHole installation on RPi-Zero-W works without a problem.

After automated reboot, PiHole works.

Setup Pihole:
cp /etc/pihole/adlists.default /etc/pihole/adlists.list
nano /etc/pihole/adlists.list
pihole -g

170327-0003

root@RPi-Zero-W:~# cat /var/log/pihole.log
Mar 27 18:20:15 dnsmasq[946]: gestartet, Version 2.72, Cachegröße 10000
Mar 27 18:20:15 dnsmasq[946]: Übersetzungsoptionen: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntra$
Mar 27 18:20:15 dnsmasq[946]: Warnung: Ignoriere "resolv-file", weil "no-resolv" aktiv ist
Mar 27 18:20:15 dnsmasq[946]: Benutze Namensserver 192.168.1.55#53
Mar 27 18:20:15 dnsmasq[946]: Benutze Namensserver 192.168.1.66#53
Mar 27 18:20:15 dnsmasq[946]: /etc/hosts gelesen - 2 Adressen
Mar 27 18:20:15 dnsmasq[946]: /etc/pihole/local.list gelesen - 4 Adressen
Mar 27 18:20:29 dnsmasq[946]: /etc/pihole/gravity.list gelesen - 204894 Adressen
Mar 27 18:28:27 dnsmasq[946]: query[PTR] lb._dns-sd._udp.LAN from 192.168.1.102
Mar 27 18:28:27 dnsmasq[946]: forwarded lb._dns-sd._udp.LAN to 192.168.1.55
Mar 27 18:28:27 dnsmasq[946]: forwarded lb._dns-sd._udp.LAN to 192.168.1.66
Mar 27 18:28:27 dnsmasq[946]: query[PTR] lb._dns-sd._udp.0.10.16.172.in-addr.arpa from 192.168.1.102
Mar 27 18:28:27 dnsmasq[946]: forwarded lb._dns-sd._udp.0.10.16.172.in-addr.arpa to 192.168.1.55
Mar 27 18:28:27 dnsmasq[946]: forwarded lb._dns-sd._udp.0.10.16.172.in-addr.arpa to 192.168.1.66
Mar 27 18:28:27 dnsmasq[946]: query[PTR] b._dns-sd._udp.LAN from 192.168.1.102
Mar 27 18:28:27 dnsmasq[946]: forwarded b._dns-sd._udp.LAN to 192.168.1.55
Mar 27 18:28:27 dnsmasq[946]: forwarded b._dns-sd._udp.LAN to 192.168.1.66
Mar 27 18:28:27 dnsmasq[946]: query[PTR] db._dns-sd._udp.LAN from 192.168.1.102
Mar 27 18:28:27 dnsmasq[946]: forwarded db._dns-sd._udp.LAN to 192.168.1.55
.......

  • After 57 min:

170327-0004

root@RPi-Zero-W:~# cat /var/log/pihole.log
Mar 27 19:17:02 dnsmasq[4170]: gestartet, Version 2.72, Cachegröße 10000
Mar 27 19:17:02 dnsmasq[4170]: Übersetzungsoptionen: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntr$
Mar 27 19:17:02 dnsmasq[4170]: Warnung: Ignoriere "resolv-file", weil "no-resolv" aktiv ist
Mar 27 19:17:02 dnsmasq[4170]: Benutze Namensserver 192.168.1.55#53
Mar 27 19:17:02 dnsmasq[4170]: Benutze Namensserver 192.168.1.66#53
Mar 27 19:17:02 dnsmasq[4170]: /etc/hosts gelesen - 2 Adressen
Mar 27 19:17:02 dnsmasq[4170]: /etc/pihole/local.list gelesen - 4 Adressen
Mar 27 19:17:24 dnsmasq[4170]: /etc/pihole/gravity.list gelesen - 343244 Adressen
Mar 27 19:17:29 dnsmasq[4170]: query[A] gitter.im from 192.168.1.102
Mar 27 19:17:29 dnsmasq[4170]: forwarded gitter.im to 192.168.1.55
Mar 27 19:17:29 dnsmasq[4170]: forwarded gitter.im to 192.168.1.66
Mar 27 19:17:29 dnsmasq[4170]: query[AAAA] gitter.im from 192.168.1.102
Mar 27 19:17:29 dnsmasq[4170]: forwarded gitter.im to 192.168.1.55
Mar 27 19:17:29 dnsmasq[4170]: forwarded gitter.im to 192.168.1.66
Mar 27 19:17:29 dnsmasq[4170]: reply gitter.im is 54.209.199.250
Mar 27 19:17:29 dnsmasq[4170]: reply gitter.im is 54.174.166.63
Mar 27 19:17:39 dnsmasq[4170]: query[A] github.com from 192.168.1.102
Mar 27 19:17:39 dnsmasq[4170]: forwarded github.com to 192.168.1.55
Mar 27 19:17:39 dnsmasq[4170]: query[AAAA] github.com from 192.168.1.102
Mar 27 19:17:39 dnsmasq[4170]: forwarded github.com to 192.168.1.55
Mar 27 19:17:39 dnsmasq[4170]: reply github.com is 192.30.253.112
Mar 27 19:17:39 dnsmasq[4170]: reply github.com is 192.30.253.113
Mar 27 19:17:41 dnsmasq[4170]: query[A] live.github.com from 192.168.1.102
Mar 27 19:17:41 dnsmasq[4170]: forwarded live.github.com to 192.168.1.55
Mar 27 19:17:41 dnsmasq[4170]: query[AAAA] live.github.com from 192.168.1.102
Mar 27 19:17:41 dnsmasq[4170]: forwarded live.github.com to 192.168.1.55
Mar 27 19:17:41 dnsmasq[4170]: reply live.github.com is 192.30.253.124
Mar 27 19:17:41 dnsmasq[4170]: reply live.github.com is 192.30.253.125
Mar 27 19:18:41 dnsmasq[4170]: query[A] api.github.com from 192.168.1.102
Mar 27 19:18:41 dnsmasq[4170]: forwarded api.github.com to 192.168.1.55
Mar 27 19:18:41 dnsmasq[4170]: forwarded api.github.com to 192.168.1.66
Mar 27 19:18:41 dnsmasq[4170]: query[AAAA] api.github.com from 192.168.1.102
Mar 27 19:18:41 dnsmasq[4170]: forwarded api.github.com to 192.168.1.55
Mar 27 19:18:41 dnsmasq[4170]: reply api.github.com is 192.30.253.117
Mar 27 19:18:41 dnsmasq[4170]: reply api.github.com is 192.30.253.116
......

Do you noticed "Every Hour at 17 Minutes " on ALL tested devices❓

170327-0005

update: 20:17:00

root@RPi-Zero-W:~# cat /var/log/pihole.log
Mar 27 20:17:02 dnsmasq[8208]: gestartet, Version 2.72, Cachegröße 10000
Mar 27 20:17:02 dnsmasq[8208]: Übersetzungsoptionen: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntr$
Mar 27 20:17:02 dnsmasq[8208]: Warnung: Ignoriere "resolv-file", weil "no-resolv" aktiv ist
Mar 27 20:17:02 dnsmasq[8208]: Benutze Namensserver 192.168.1.55#53
Mar 27 20:17:02 dnsmasq[8208]: Benutze Namensserver 192.168.1.66#53
Mar 27 20:17:02 dnsmasq[8208]: /etc/hosts gelesen - 2 Adressen
Mar 27 20:17:02 dnsmasq[8208]: /etc/pihole/local.list gelesen - 4 Adressen
Mar 27 20:17:25 dnsmasq[8208]: /etc/pihole/gravity.list gelesen - 343244 Adressen
Mar 27 20:21:24 dnsmasq[8208]: query[A] github.com from 192.168.1.102
Mar 27 20:21:24 dnsmasq[8208]: forwarded github.com to 192.168.1.55
Mar 27 20:21:24 dnsmasq[8208]: forwarded github.com to 192.168.1.66
Mar 27 20:21:24 dnsmasq[8208]: query[AAAA] github.com from 192.168.1.102
Mar 27 20:21:24 dnsmasq[8208]: forwarded github.com to 192.168.1.55
Mar 27 20:21:24 dnsmasq[8208]: forwarded github.com to 192.168.1.66
Mar 27 20:21:24 dnsmasq[8208]: reply github.com is 192.30.253.113
Mar 27 20:21:24 dnsmasq[8208]: reply github.com is 192.30.253.112
Mar 27 20:21:28 dnsmasq[8208]: query[A] avatars3.githubusercontent.com from 192.168.1.102
Mar 27 20:21:28 dnsmasq[8208]: forwarded avatars3.githubusercontent.com to 192.168.1.66
Mar 27 20:21:28 dnsmasq[8208]: query[AAAA] avatars3.githubusercontent.com from 192.168.1.102
Mar 27 20:21:28 dnsmasq[8208]: forwarded avatars3.githubusercontent.com to 192.168.1.66
Mar 27 20:21:28 dnsmasq[8208]: reply avatars3.githubusercontent.com is <CNAME>
Mar 27 20:21:28 dnsmasq[8208]: reply github.map.fastly.net is 151.101.112.133
Mar 27 20:21:28 dnsmasq[8208]: reply avatars3.githubusercontent.com is <CNAME>
Mar 27 20:21:28 dnsmasq[8208]: reply github.map.fastly.net is NODATA-IPv6
Mar 27 20:21:41 dnsmasq[8208]: query[A] avatars2.githubusercontent.com from 192.168.1.102
Mar 27 20:21:41 dnsmasq[8208]: forwarded avatars2.githubusercontent.com to 192.168.1.66
Mar 27 20:21:41 dnsmasq[8208]: query[AAAA] avatars2.githubusercontent.com from 192.168.1.102
Mar 27 20:21:41 dnsmasq[8208]: forwarded avatars2.githubusercontent.com to 192.168.1.66
Mar 27 20:21:41 dnsmasq[8208]: reply avatars2.githubusercontent.com is <CNAME>
Mar 27 20:21:41 dnsmasq[8208]: reply github.map.fastly.net is 151.101.112.133
Mar 27 20:21:41 dnsmasq[8208]: reply avatars2.githubusercontent.com is <CNAME>
Mar 27 20:21:41 dnsmasq[8208]: reply github.map.fastly.net is NODATA-IPv6
....

Yes, /etc/cron.hourly/ works ....

@k-plan
Copy link
Contributor Author

k-plan commented Mar 27, 2017

for testing:

  • unselect locale de_DE on same RPi Zero W and reboot
root@RPi-Zero-W:~# locale -a
C
C.UTF-8
en_GB.utf8
POSIX

root@RPi-Zero-W:~# locale
LANG=en_GB.UTF-8
LANGUAGE=
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=

170327-0006

Really strange, I cant seem to replicate this on RAMlog mode 1/2. PiHole logs are only cleared if day != today, or 5MB limit.

No, choose, add and switch locale=de_DE to default and you can see the magic ... 😃


BTW: Hope, with "Pi-hole FTL engine" all become better. 😃

Coming Soon To A Pi-hole Near You: FTL

Pi-hole FTL engine

@Fourdee
Copy link
Collaborator

Fourdee commented Mar 28, 2017

@k-plan

No, choose, add and switch locale=de_DE to default and you can see the magic ...

Thanks, i'll check 👍

FTL

Looks like a change from php to C: https://github.com/pi-hole/FTL/blob/master/main.c should make for much reduced query times, espically with use of multithreading: https://github.com/pi-hole/FTL/blob/master/main.c#L43-L49

If IO on log file has any impact on query time, it "should" run faster on DietPi (RAMlog) 😉

@Fourdee
Copy link
Collaborator

Fourdee commented Mar 28, 2017

@k-plan

https://github.com/pi-hole/FTL#pi-hole-ftl

This project is copyright under the latest version of the EUPL.

Yep, and Sonic is copyright of SEGA lmao

@k-plan
Copy link
Contributor Author

k-plan commented Mar 28, 2017

Only change default locale=en_GB didn't help. Must remove all other language support.


If IO on log file has any impact on query time, it "should" run faster on DietPi (RAMlog) 😉

Flat file support will decease. Running database in RAMlog? 😉

FTL will help remediate the immediate need for a faster interface. The second component on the roadmap would be using a database instead of trying to query the entire text of the log file in it’s raw format.

@Fourdee
Copy link
Collaborator

Fourdee commented Mar 29, 2017

@k-plan

Got it:

  • en_GB default:
root@DietPi:~# LANG=en_GB.UTF-8 date +'%b %e'
Mar 29
  • de_DE default with en_GB installed:
root@DietPi:~# LANG=en_GB.UTF-8 date +'%b %e'
Mär 29

This will cause scrape to miss those entries, removing them.

I'll try a fix.

  • 🈯️
root@DietPi:~# LC_ALL=en_GB.UTF-8 date +'%b %e'
Mar 29

Fourdee referenced this issue Mar 29, 2017
+ PiHole log clear fix, non en_GB default:
https://github.com/Fourdee/DietPi/issues/839
@Fourdee
Copy link
Collaborator

Fourdee commented Mar 29, 2017

@k-plan

This commit should do it:
Fourdee@3a6c6a3

@k-plan
Copy link
Contributor Author

k-plan commented Mar 29, 2017

@Fourdee

Test is running on one device.
We have to wait a little bit and see what's happens.

Thanks.

@k-plan
Copy link
Contributor Author

k-plan commented Mar 30, 2017

Testing:

  • switch locale language back to my default de_DE:

170330-0001

  • apply your commit by hand:
root@OPi-One:~# cat /DietPi/dietpi/dietpi-logclear | grep 'local month_day_today'
                                        ## local month_day_today=$(LANG=en_GB.UTF-8 date +'%b %e')
                                        local month_day_today=$(LC_ALL=en_GB.UTF-8 date +'%b %e')

  • reboot the device

  • de_DE default with en_GB installed:

root@OPi-One:~# date
Do 30. Mär 13:00:27 CEST 2017

root@OPi-One:~# LANG=en_GB.UTF-8 date +'%b %e'
Mar 30

root@OPi-One:~# LC_ALL=en_GB.UTF-8; date +'%b %e'
Mär 30

root@OPi-One:~# cat /var/log/pihole.log
Mar 30 10:58:42 dnsmasq[1061]: started, version 2.72 cachesize 10000
Mar 30 10:58:42 dnsmasq[1061]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntr$
Mar 30 10:58:42 dnsmasq[1061]: warning: ignoring resolv-file flag because no-resolv is set
Mar 30 10:58:42 dnsmasq[1061]: using nameserver 192.168.1.56#53
Mar 30 10:58:42 dnsmasq[1061]: read /etc/hosts - 2 addresses
Mar 30 10:58:42 dnsmasq[1061]: read /etc/pihole/local.list - 4 addresses
Mar 30 10:59:03 dnsmasq[1061]: read /etc/pihole/gravity.list - 343242 addresses
Mar 30 11:26:38 dnsmasq[1061]: query[PTR] lb._dns-sd._udp.LAN from 192.168.1.102
Mar 30 11:26:38 dnsmasq[1061]: forwarded lb._dns-sd._udp.LAN to 192.168.1.56
Mar 30 11:26:38 dnsmasq[1061]: query[PTR] lb._dns-sd._udp.0.1.168.192.in-addr.arpa from 192.168.1.102
Mar 30 11:26:38 dnsmasq[1061]: forwarded lb._dns-sd._udp.0.1.168.192.in-addr.arpa to 192.168.1.56
Mar 30 11:26:38 dnsmasq[1061]: query[PTR] 102.1.168.192.in-addr.arpa from 192.168.1.102
Mar 30 11:26:38 dnsmasq[1061]: config 192.168.1.102 is NXDOMAIN
....
root@OPi-One:~# ls -lah /var/log/pihole.log
-rwxrwxr-x 1 dnsmasq www-data 101K Mär 30 13:06 /var/log/pihole.log
  • AdminLTE:

170330-0003


  • checkout "testing branch" on a other device:
root@NanoPi-Neo:~# cat /DietPi/dietpi/.version
148

root@NanoPi-Neo:~# date
Do 30. Mär 13:07:46 CEST 2017

root@NanoPi-Neo:~# LANG=en_GB.UTF-8 date +'%b %e'
Mar 30

root@NanoPi-Neo:~# LC_ALL=en_GB.UTF-8; date +'%b %e'
Mär 30
root@NanoPi-Neo:~# cat /DietPi/dietpi/dietpi-logclear | grep 'local month_day'
                                        local month_day_today=$(LC_ALL=en_GB.UTF-8 date +'%b %e')

Mar 30 10:35:35 dnsmasq[1075]: gestartet, Version 2.72, Cachegröße 10000
Mar 30 10:35:35 dnsmasq[1075]: Übersetzungsoptionen: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntr$
Mar 30 10:35:35 dnsmasq[1075]: Warnung: Ignoriere "resolv-file", weil "no-resolv" aktiv ist
Mar 30 10:35:35 dnsmasq[1075]: Benutze Namensserver 192.168.1.1#53
Mar 30 10:35:35 dnsmasq[1075]: Benutze Namensserver 192.168.1.55#53
Mar 30 10:35:35 dnsmasq[1075]: /etc/hosts gelesen - 6 Adressen
Mar 30 10:35:35 dnsmasq[1075]: /etc/pihole/local.list gelesen - 4 Adressen
Mar 30 10:36:01 dnsmasq[1075]: /etc/pihole/gravity.list gelesen - 347304 Adressen
Mar 30 10:36:01 dnsmasq[1075]: query[A] github.com from 192.168.1.102
Mar 30 10:36:01 dnsmasq[1075]: forwarded github.com to 192.168.1.1
Mar 30 10:36:01 dnsmasq[1075]: forwarded github.com to 192.168.1.55
Mar 30 10:36:01 dnsmasq[1075]: query[AAAA] github.com from 192.168.1.102
Mar 30 10:36:01 dnsmasq[1075]: forwarded github.com to 192.168.1.1
Mar 30 10:36:01 dnsmasq[1075]: forwarded github.com to 192.168.1.55
Mar 30 10:36:01 dnsmasq[1075]: reply github.com is 192.30.253.113
Mar 30 10:36:01 dnsmasq[1075]: reply github.com is 192.30.253.112
....
root@NanoPi-Neo:~# ls -lah /var/log/pihole.log
-rwxrwxr-x 1 dnsmasq www-data 131K Mär 30 13:09 /var/log/pihole.log
  • AdminLTE:

170330-0004


Different language in logfile content depends on device (NanoPi Neo vs. OrangePi One), but it seem to be equal.

@Fourdee
Copy link
Collaborator

Fourdee commented Mar 30, 2017

@k-plan

Thanks, looks to be working with the change?

I believe you need to remove the ;, else, doesn't get applied to next command

LC_ALL=en_GB.UTF-8 date +'%b %e'

I left my C2 running today with the code change, de_DE default locale, seems to be working, logs from 01:00 onwards:
image

Think we can mark this one as closed?

@k-plan
Copy link
Contributor Author

k-plan commented Mar 30, 2017

@Fourdee

I believe you need to remove the ;, else, doesn't get applied to next command

Only copy your command from here 😃

But yes, you're right:

root@RPi-Zero-W:~# date
2017年 03月 30日 星期四 14:22:28 CEST

root@RPi-Zero-W:~# LC_ALL=en_GB.UTF-8; date +'%b %e'
3月 30

root@RPi-Zero-W:~# LC_ALL=en_GB.UTF-8 date +'%b %e'
Mar 30

root@RPi-Zero-W:~# LANG=en_GB.UTF-8 date +'%b %e'
Mar 30
  • My new testing default locale (zh_CN.utf8):
root@RPi-Zero-W:~# locale -a -v

locale: zh_CN.utf8      archive: /usr/lib/locale/locale-archive
-------------------------------------------------------------------------------
    title | Chinese locale for Peoples Republic of China
    email | [email protected]
 language | Chinese
territory | P.R. of China
 revision | 0.1
     date | 2000-07-25
  codeset | UTF-8

locale: en_GB.utf8      archive: /usr/lib/locale/locale-archive
-------------------------------------------------------------------------------
    title | English locale for Britain
   source | RAP
  address | Sankt Jørgens Alle 8, DK-1615 København V, Danmark
  contact | Keld Simonsen
    email | [email protected]
 language | English
territory | Great Britain
 revision | 1.0
     date | 2000-06-28
  codeset | UTF-8

😄 ( ... and British form a Dane - nice!) 🤣

Thanks, looks to be working with the change?

yes, I think so. It seem to be working on my Chinese locale RPi-Zero_W now.

170330-0005

root@RPi-Zero-W:~# ls -lah /var/log/pihole.log
-rwxrwxr-x 1 dnsmasq www-data 81K 3月  30 14:39 /var/log/pihole.log
root@RPi-Zero-W:~# apt-get upgrade
正在读取软件包列表... 完成
正在分析软件包的依赖关系树
正在读取状态信息... 完成
正在对升级进行计算... 完成
下列软件包的版本将保持不变:
  libraspberrypi-bin libraspberrypi0 raspberrypi-bootloader raspberrypi-kernel
升级了 0 个软件包,新安装了 0 个软件包,要卸载 0 个软件包,有 4 个软件包未被升级。

🤣

Think we can mark this one as closed?

Yes, if it will run well on your C2 too.

Failure with locales in scripting, just as difficult every time.
Sorry, better to remove language option in DietPi ... 😺

@k-plan
Copy link
Contributor Author

k-plan commented Mar 30, 2017

@Fourdee

Testing results:

  • RPi-Zero W (zh_CN.utf8):

170330-0006

  • OrangePi One (de_DE.utf8):

170330-0007

  • NanoPi Neo v.1 (de_DE.utf8):

170330-0008


  • Production system (Odroid C2) - non-competitive

170330-0009

@k-plan k-plan closed this as completed Mar 30, 2017
@Fourdee
Copy link
Collaborator

Fourdee commented Mar 30, 2017

@k-plan

Only copy your command from here 😃

Yep, my bad 👍

@Fourdee Fourdee mentioned this issue Apr 8, 2017
Fourdee referenced this issue Apr 8, 2017
**v148**
(08/04/17)

---

**Changes / Improvements / Optimizations:**

DietPi-Config | WiFi: Country code is now also applied to cfg80211 module params (ieee80211_regdom=): https://github.com/Fourdee/DietPi/issues/838

DietPi-Config | Network NAS/Misc: Added option to delay boot, until a valid network connection is established: https://github.com/Fourdee/DietPi/issues/842

DietPi-Config | Audio > Sound Card: Added support for AudioPhonics I-Sabre-K2M (9018K2M): https://github.com/Fourdee/DietPi/issues/716

DietPi-Software | NTPD check/wait: Is now bypassed if no valid network connection is available. It will be checked again, before software installation: https://github.com/Fourdee/DietPi/issues/840

DietPi-Software | Tonido: Now available for installation. Lightweight alternative to NextCloud/Owncloud: http://dietpi.com/phpbb/viewtopic.php?f=8&t=5&p=6476#p6476

DietPi-Software | IceCast: Now available for installation. This is a Shoutcast streaming server, using DarkIce for audio input: http://dietpi.com/phpbb/viewtopic.php?f=8&t=5&p=6526#p6526

DietPi-Software | MotionEye: Now available for installation (RPi only). Web interface and surveillance for your RPi/USB camera. Mainly focused towards motion detection usage: http://dietpi.com/phpbb/viewtopic.php?f=8&t=5&start=100

DietPi-Software | CloudPrint (Google): Now available for installation. Print server with support for Google cloud printing. Many thanks to Sciamano for his assistance with testing: http://dietpi.com/phpbb/viewtopic.php?f=8&t=5&start=100#p6630

DietPi-Set_Hardware | RPi Camera: Now enables bcm2835-v4l2 by default. Required for /dev/video0 stream (mainly for motioneye).

General | Boot: All failed drive mounts will be re-mounted after network connection is established. This is mainly a failsafe for CIFS/net shares, to ensure they are mounted again after network established: https://github.com/Fourdee/DietPi/issues/842#issuecomment-289448261

---

**Bug fixes:**

DietPi-Software | YaCy: Web interface admin details are now set during install. Username = admin, password = dietpi. Many thanks to Pilovali for reporting this!:

DietPi-Software | Gogs: Installation now resolved. Due to https://dl.gogs.io/ removing "latest" links for binary download. Now hosted on dietpi.com: https://github.com/Fourdee/DietPi/issues/833

DietPi-Software | OMV: Has been disabled. We have recieved too many reports from users with issues, due to incompatibility with DietPi. As we cant control what OMV does and how it does it, we cannot support it: https://github.com/Fourdee/DietPi/issues/851

DietPi-RAMlog | PiHole: Resolved an issue where PiHole logs were being cleared every 1 hour, if default locale on system is not en_GB. PiHole logs allow for current day and 5MB size limit. Many thanks to @k-plan for his assistance with this one: https://github.com/Fourdee/DietPi/issues/839

General | Cron-Daily: Removed a df export left over from auto TRIM code (no longer used).

General | RoseapplePi CPU temp: Now functional. Many thanks to @joaofl for this fix: https://github.com/Fourdee/DietPi/issues/848

---

**RPi Stretch:**

DietPi-Software | FFmpeg now sucessfully installs: https://github.com/Fourdee/DietPi/issues/475
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants