Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FTL crashed while importing Teleporter backup #1214

Closed
yubiuser opened this issue Oct 10, 2021 · 6 comments · Fixed by #1222
Closed

FTL crashed while importing Teleporter backup #1214

yubiuser opened this issue Oct 10, 2021 · 6 comments · Fixed by #1222
Labels
Bug Bugfix in progress

Comments

@yubiuser
Copy link
Member

yubiuser commented Oct 10, 2021 •
edited
Loading

Versions

Pi-hole version is development v5.4-38-g19ae9d3 (Latest: v5.5)
AdminLTE version is v5.7 (Latest: v5.7)
FTL version is development vDev-301e97f (Latest: v5.10.2)

Platform

  • OS and version: Debian 10
  • Platform: RockPi

Expected behavior

No crash while importing Teleporter backup.

Actual behavior / bug

There are actually 2 bugs:

  1. Importing a backup makes FTL restarting many time (both on master and on devel - tried to see Modal-Dialog for Teleporter-import web#1231 in action made me noticing this)
  2. while on master, during one of the FTL restarts, Pi-hole crashed

Steps to reproduce

Import a backup.

[2021-10-10 21:13:32.625 2264M]  -> Unknown DNS queries: 0
[2021-10-10 21:13:32.625 2264M]  -> Unique domains: 1013
[2021-10-10 21:13:32.625 2264M]  -> Unique clients: 13
[2021-10-10 21:13:32.625 2264M]  -> Known forward destinations: 2
[2021-10-10 21:13:32.625 2264M] Successfully accessed setupVars.conf
[2021-10-10 21:13:32.626 2264M] listening on 0.0.0.0 port 53
[2021-10-10 21:13:32.626 2264M] listening on :: port 53
[2021-10-10 21:13:32.630 2275M] PID of FTL process: 2275
[2021-10-10 21:13:32.630 2275/T2276] Listening on port 4711 for incoming IPv4 telnet connections
[2021-10-10 21:13:32.631 2275M] INFO: FTL is running as user pihole (UID 998)
[2021-10-10 21:13:32.631 2275/T2277] Error listening on IPv6 port 4711: Cannot assign requested address (99)
[2021-10-10 21:13:32.632 2275/T2278] Listening on Unix socket
[2021-10-10 21:13:32.632 2275M] Reloading DNS cache
[2021-10-10 21:13:32.632 2275M] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2021-10-10 21:13:32.632 2275M] ---------------------------->  FTL crashed!  <----------------------------
[2021-10-10 21:13:32.632 2275M] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2021-10-10 21:13:32.632 2275M] Please report a bug at https://github.com/pi-hole/FTL/issues
[2021-10-10 21:13:32.632 2275M] and include in your report already the following details:
[2021-10-10 21:13:32.633 2275M] FTL has been running for 5 seconds
[2021-10-10 21:13:32.633 2275M] FTL branch: development
[2021-10-10 21:13:32.633 2275M] FTL version: vDev-301e97f
[2021-10-10 21:13:32.633 2275M] FTL commit: 301e97f3
[2021-10-10 21:13:32.633 2275M] FTL date: 2021-10-10 17:56:58 +0200
[2021-10-10 21:13:32.633 2275M] FTL user: started as pihole, ended as pihole
[2021-10-10 21:13:32.633 2275M] Compiled for aarch64 (compiled on CI) using aarch64-linux-gnu-gcc (Debian 6.3.0-18) 6.3.0 20170516
[2021-10-10 21:13:32.633 2275M] Process details: MID: 2275
[2021-10-10 21:13:32.633 2275M]                  PID: 2275
[2021-10-10 21:13:32.633 2275M]                  TID: 2275
[2021-10-10 21:13:32.633 2275M]                  Name: pihole-FTL
[2021-10-10 21:13:32.633 2275M] Received signal: Segmentation fault
[2021-10-10 21:13:32.633 2275M]      at address: 0x11
[2021-10-10 21:13:32.634 2275M]      with code:  SEGV_MAPERR (Address not mapped to object)
[2021-10-10 21:13:32.634 2275M] Backtrace:
[2021-10-10 21:13:32.635 2275M] B[0000]: /usr/bin/pihole-FTL(generate_backtrace+0x4c) [0xaaaad51d2094]
[2021-10-10 21:13:32.704 2275M] L[0000]: /root/project/src/signals.c:98
[2021-10-10 21:13:32.707 2275M] B[0001]: /usr/bin/pihole-FTL(+0x62550) [0xaaaad51d2550]
[2021-10-10 21:13:32.735 2275M] L[0001]: /root/project/src/signals.c:242
[2021-10-10 21:13:32.740 2275M] B[0002]: linux-vdso.so.1(__kernel_rt_sigreturn+0) [0xffff861fb7d8]
[2021-10-10 21:13:32.740 2275M] B[0003]: /lib/aarch64-linux-gnu/libc.so.6(getdelim+0x258) [0xffff85fba648]
[2021-10-10 21:13:32.740 2275M] B[0004]: /usr/bin/pihole-FTL(+0x4a37c) [0xaaaad51ba37c]
[2021-10-10 21:13:32.769 2275M] L[0004]: /root/project/src/config.c:708
[2021-10-10 21:13:32.775 2275M] B[0005]: /usr/bin/pihole-FTL(read_debuging_settings+0x3d8) [0xaaaad51bae90]
[2021-10-10 21:13:32.803 2275M] L[0005]: /root/project/src/config.c:837
[2021-10-10 21:13:32.806 2275M] B[0006]: /usr/bin/pihole-FTL(FTL_dnsmasq_reload+0x80) [0xaaaad51c4b58]
[2021-10-10 21:13:32.826 2275M] L[0006]: /root/project/src/dnsmasq_interface.c:1781
[2021-10-10 21:13:32.829 2275M] B[0007]: /usr/bin/pihole-FTL(clear_cache_and_reload+0x30) [0xaaaad51f59e0]
[2021-10-10 21:13:32.858 2275M] L[0007]: N/A (0x859e0 -> addr2line 0x859e0 -e /usr/bin/pihole-FTL)
[2021-10-10 21:13:32.866 2275M] B[0008]: /usr/bin/pihole-FTL(main_dnsmasq+0x1704) [0xaaaad51f744c]
[2021-10-10 21:13:32.896 2275M] L[0008]: /root/project/src/dnsmasq/dnsmasq.c:1482
[2021-10-10 21:13:32.900 2275M] B[0009]: /usr/bin/pihole-FTL(main+0xf4) [0xaaaad51b8ac4]
[2021-10-10 21:13:32.927 2275M] L[0009]: /root/project/src/main.c:108
[2021-10-10 21:13:32.933 2275M] B[0010]: /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0xe4) [0xffff85f77d24]
[2021-10-10 21:13:32.933 2275M] B[0011]: /usr/bin/pihole-FTL(+0x48c40) [0xaaaad51b8c40]
[2021-10-10 21:13:32.971 2275M] L[0011]: :?
[2021-10-10 21:13:32.978 2275M] ------ Listing content of directory /dev/shm ------
[2021-10-10 21:13:32.978 2275M] File Mode User:Group      Size  Filename
[2021-10-10 21:13:32.978 2275M] rwxrwxrwx root:root       260   .
[2021-10-10 21:13:32.978 2275M] rwxr-xr-x root:root         4K  ..
[2021-10-10 21:13:32.978 2275M] rw------- pihole:pihole     4K  FTL-per-client-regex
[2021-10-10 21:13:32.978 2275M] rw------- pihole:pihole     4K  FTL-dns-cache
[2021-10-10 21:13:32.978 2275M] rw------- pihole:pihole     8K  FTL-overTime
[2021-10-10 21:13:32.978 2275M] rw------- pihole:pihole     1M  FTL-queries
[2021-10-10 21:13:32.979 2275M] rw------- pihole:pihole    20K  FTL-upstreams
[2021-10-10 21:13:32.979 2275M] rw------- pihole:pihole   356K  FTL-clients
[2021-10-10 21:13:32.979 2275M] rw------- pihole:pihole    25K  FTL-domains
[2021-10-10 21:13:32.979 2275M] rw------- pihole:pihole    82K  FTL-strings
[2021-10-10 21:13:32.979 2275M] rw------- pihole:pihole    12   FTL-settings
[2021-10-10 21:13:32.979 2275M] rw------- pihole:pihole   240   FTL-counters
[2021-10-10 21:13:32.979 2275M] rw------- pihole:pihole   104   FTL-lock
[2021-10-10 21:13:32.979 2275M] ---------------------------------------------------
[2021-10-10 21:13:32.979 2275M] Please also include some lines from above the !!!!!!!!! header.
[2021-10-10 21:13:32.979 2275M] Thank you for helping us to improve our FTL engine!
[2021-10-10 21:13:32.980 2275M] ########## FTL terminated after 5s 625ms  (code 1)! ##########
[2021-10-10 21:13:34.079 2333M] Using log file /var/log/pihole-FTL.log
[2021-10-10 21:13:34.079 2333M] ########## FTL started on rockpi-4b! ##########
[2021-10-10 21:13:34.079 2333M] FTL branch: development
[2021-10-10 21:13:34.079 2333M] FTL version: vDev-301e97f
@yubiuser
Copy link
Member Author

The restarts happen due to deleting and importing local dns records/CNAMES

https://github.com/pi-hole/AdminLTE/blob/5588be9c9086ec120cd3bd4dc36bc9d4ed0acd77/scripts/pi-hole/php/teleporter.php#L520-L528

pointing to

https://github.com/pi-hole/AdminLTE/blob/5588be9c9086ec120cd3bd4dc36bc9d4ed0acd77/scripts/pi-hole/php/func.php#L298

to

https://github.com/pi-hole/pi-hole/blob/dad6247cb0c90fe8cc1ad541e0ccf8f02e1dd2be/advanced/Scripts/webpage.sh#L718-L732

Containing a pihole restartdns

@DL6ER
Copy link
Member

DL6ER commented Oct 16, 2021

Sorry for the delay on checking this, somehow it managed to slip my attention.

[2021-10-10 21:13:32.769 2275M] L[0004]: /root/project/src/config.c:708

FTL/src/config.c

Line 708 in 301e97f

while(getline(&conflinebuffer, &size, fp) != -1)

Unfortunately, it is unclear which argument of getline caused the issue here. I wonder if it is a race collision where some parts of FTL were already terminating for the next restart while other parts were still initializing. The terminating parts may have started releasing (= invalidating) memory while the still working parts tried to use this memory -> crash.

@DL6ER DL6ER added the Bug label Oct 16, 2021
@yubiuser
Copy link
Member Author

It will be hard to debug - I've tried multiple times to reproduce but it didn't crash

@DL6ER DL6ER mentioned this issue Oct 17, 2021
Merged
5 tasks
@DL6ER DL6ER linked a pull request Oct 17, 2021 that will close this issue
Prevent race collisions when parsing FTL config file #1222
Merged
5 tasks
@DL6ER
Copy link
Member

DL6ER commented Oct 17, 2021

#1222 will fix this (really difficult to trigger) race collision. However, we should also fix the many-restarts issue as this is just wasting a lot of resources.

@yubiuser
Copy link
Member Author

However, we should also fix the many-restarts issue as this is just wasting a lot of resources.

See
pi-hole/pi-hole#4384
pi-hole/web#1925

@DL6ER
Copy link
Member

DL6ER commented Oct 23, 2021

Update for FTL released

@DL6ER DL6ER closed this as completed Oct 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Bugfix in progress
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Prevent race collisions when parsing FTL config file pi-hole/FTL
2 participants
@DL6ER @yubiuser