Resolving raw.githubusercontent.com (raw.githubusercontent.com)... failed: Connection timed out. #2820

Sopor · 2019-05-18T20:25:11Z

Details:

Date | Sat 18 May 22:21:33 CEST 2019
Bug report | N/A
DietPi version | v6.22.3 (MichaIng/master)
Img creator | DietPi Core Team
Pre-image | Raspbian Lite
SBC device | RPi 3 Model B+ (armv7l) (index=3)
Kernel version | Request | RT Kernel option #1200 SMP Tue Feb 12 20:27:48 GMT 2019
Distro | stretch (index=4)
Command | Connection test: https://raw.githubusercontent.com/MichaIng/DietPi/master/dietpi.txt
Exit code | 4
Software title | DietPi-Set_software

Steps to reproduce:

sudo dietpi-update

Expected behaviour:

...

Actual behaviour:

...

Extra details:

...

Additional logs:

Log file contents:
Spider mode enabled. Check if remote file exists.
--2019-05-18 22:19:31--  https://raw.githubusercontent.com/MichaIng/DietPi/master/dietpi.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... failed: Connection timed out.
wget: unable to resolve host address ‘raw.githubusercontent.com’

The text was updated successfully, but these errors were encountered:

Sopor · 2019-05-18T20:30:04Z

When i tried to ping the site it was very slow

$ ping raw.githubusercontent.com
PING github.map.fastly.net (151.101.84.133) 56(84) bytes of data.
64 bytes from 151.101.84.133 (151.101.84.133): icmp_seq=1 ttl=54 time=13.0 ms
64 bytes from 151.101.84.133 (151.101.84.133): icmp_seq=2 ttl=54 time=13.0 ms
64 bytes from 151.101.84.133 (151.101.84.133): icmp_seq=3 ttl=54 time=13.1 ms
64 bytes from 151.101.84.133 (151.101.84.133): icmp_seq=4 ttl=54 time=13.1 ms
64 bytes from 151.101.84.133 (151.101.84.133): icmp_seq=5 ttl=54 time=13.6 ms
64 bytes from 151.101.84.133 (151.101.84.133): icmp_seq=6 ttl=54 time=13.0 ms
64 bytes from 151.101.84.133 (151.101.84.133): icmp_seq=7 ttl=54 time=13.0 ms
64 bytes from 151.101.84.133 (151.101.84.133): icmp_seq=8 ttl=54 time=13.0 ms
64 bytes from 151.101.84.133 (151.101.84.133): icmp_seq=9 ttl=54 time=13.5 ms

--- github.map.fastly.net ping statistics ---
9 packets transmitted, 9 received, 0% packet loss, time 12032ms
rtt min/avg/max/mdev = 13.036/13.195/13.686/0.237 ms

Sopor · 2019-05-18T20:32:03Z

So now i have a half updated DietPi? What should i do now?

MichaIng · 2019-05-18T20:34:27Z

@Sopor
Many thanks for your report.

Hmm 13 ms but overall 13 seconds? Looks like actually it means 13 seconds for each ping, right?

Jep that is indeed very slow. Not sure if it's temporarily an issue of the GitHub server. Our timeout is 5s by default. This can be changed since v6.23 🤣.

Do the following:

sed -i 's/timeout:-5/timeout:-20/' /DietPi/dietpi/func/dietpi-globals
dietpi-update 1

Sopor · 2019-05-18T20:42:19Z

 $ sudo dietpi-update 1
[  OK  ] DietPi-Update | Root access verified.
[  OK  ] DietPi-Update | RootFS R/W access verified.

 DietPi-Update
─────────────────────────────────────────────────────
 Mode: Checking for DietPi updates

[ INFO ] DietPi-Update | Checking mirror: https://raw.githubusercontent.com/MichaIng/DietPi/master/dietpi/server_version-6
[  OK  ] DietPi-Update | Using update server: https://raw.githubusercontent.com/MichaIng/DietPi/master/dietpi/server_version-6

[ INFO ] DietPi-Update | Current version : v6.24.0
[ INFO ] DietPi-Update | Latest version  : v6.24.0
[  OK  ] DietPi-Update | No updates required, your DietPi installation is up to date.

But there was several errors including some ssh address it couldn't connect to.
I hope it didn't messed up my installation :(

MichaIng · 2019-05-18T20:51:52Z

@Sopor
No worries, nothing that could not be fixed easily. Do you have the error output about ssh connections there? Actually this can only be the survey upload (so no issue at all) and yeah I guess I know where this comes from, since above dietpi.txt file update failed (adding new settings), which contain the new timeout and retry values for URL check.

Please do:

G_CONFIG_INJECT 'CONFIG_G_CHECK_URL_TIMEOUT=' 'CONFIG_G_CHECK_URL_TIMEOUT=20' /DietPi/dietpi.txt
G_CONFIG_INJECT 'CONFIG_G_CHECK_URL_ATTEMPTS=' 'CONFIG_G_CHECK_URL_ATTEMPTS=3' /DietPi/dietpi.txt
/DietPi/dietpi/func/dietpi-set_software verify_dietpi.txt
/DietPi/dietpi/dietpi-survey 1

And if you want to re-apply the update, just to be failsafe: dietpi-update -1

And you can adjust the timeout for our URL connection check via dietpi-config > "Network Options: Misc"

Sopor · 2019-05-18T20:55:11Z

$ G_CONFIG_INJECT 'CONFIG_G_CHECK_URL_TIMEOUT=' 'CONFIG_G_CHECK_URL_TIMEOUT=20' /DietPi/dietpi.txt
[FAILED] File does not exist or cannot be written to by current user                                                 │
               │                                                                                                                      │
               │ Please verify the existence of the file $3                                                                           │
               │         /DietPi/dietpi.txt                                                                                           │
               │                                                                                                                      │
               │ Retry with proper permissions or apply the setting manually:                                                         │
               │         CONFIG_G_CHECK_URL_TIMEOUT=20

If i run with sudo i get:
sudo: G_CONFIG_INJECT: command not found

MichaIng · 2019-05-18T20:57:17Z

@Sopor
Ah non-root, prefix the G_CONFIG_INJECT commands with G_SUDO.

Since those G_* are global "functions" instead of "aliases", those are not available within sudo environment without having a shell loaded. G_SUDO does this.

Sopor · 2019-05-18T21:02:45Z

Same thing :(

This is what i wrote:

 $ G_SUDO G_CONFIG_INJECT 'CONFIG_G_CHECK_URL_TIMEOUT=' 'CONFIG_G_CHECK_URL_TIMEOUT=20' /DietPi/dietpi.txt

MichaIng · 2019-05-18T21:07:17Z

@Sopor
Whoot? Does the file really not exist?
ls -l /DietPi/dietpi.txt

Sopor · 2019-05-18T21:08:24Z

 $ ls -l /DietPi/dietpi.txt
-rwxr-xr-x 1 root root 11554 May 18 22:21 /DietPi/dietpi.txt

Sopor · 2019-05-18T21:11:10Z

I hope i do it right?

$ G_SUDO G_CONFIG_INJECT 'CONFIG_G_CHECK_URL_TIMEOUT=' 'CONFIG_G_CHECK_URL_TIMEOUT=20' /DietPi/dietpi.txt

[FAILED] File does not exist or cannot be written to by current user                                                 │
               │                                                                                                                      │
               │ Please verify the existence of the file $3                                                                           │
               │         /DietPi/dietpi.txt                                                                                           │
               │                                                                                                                      │
               │ Retry with proper permissions or apply the setting manually:                                                         │
               │         CONFIG_G_CHECK_URL_TIMEOUT=20

MichaIng · 2019-05-18T21:11:19Z

@Sopor
Please reload globals first, I think we just fixed the G_command calls as non-root with v6.23:

. /DietPi/dietpi/func/dietpi-globals
G_SUDO G_CONFIG_INJECT 'CONFIG_G_CHECK_URL_TIMEOUT=' 'CONFIG_G_CHECK_URL_TIMEOUT=20' /DietPi/dietpi.txt
G_SUDO G_CONFIG_INJECT 'CONFIG_G_CHECK_URL_ATTEMPTS=' 'CONFIG_G_CHECK_URL_ATTEMPTS=3' /DietPi/dietpi.txt
sudo /DietPi/dietpi/func/dietpi-set_software verify_dietpi.txt
sudo /DietPi/dietpi/dietpi-survey 1

Sopor · 2019-05-18T21:14:23Z

Same thing again
$ . /DietPi/dietpi/func/dietpi-globals
$ G_SUDO G_CONFIG_INJECT 'CONFIG_G_CHECK_URL_TIMEOUT=' 'CONFIG_G_CHECK_URL_TIMEOUT=20' /DietPi/dietpi.txt

[FAILED] File does not exist or cannot be written to by current user

MichaIng · 2019-05-18T21:21:36Z

@Sopor
God damn, jep found the reason... needs fix again, although not yet sure how... however minor one...

However then forget about G_SUDO for now and open a sudo shell to apply :

sudo -s
G_CONFIG_INJECT 'CONFIG_G_CHECK_URL_TIMEOUT=' 'CONFIG_G_CHECK_URL_TIMEOUT=20' /DietPi/dietpi.txt
G_CONFIG_INJECT 'CONFIG_G_CHECK_URL_ATTEMPTS=' 'CONFIG_G_CHECK_URL_ATTEMPTS=3' /DietPi/dietpi.txt
exit
sudo /DietPi/dietpi/func/dietpi-set_software verify_dietpi.txt
sudo /DietPi/dietpi/dietpi-survey 1

MichaIng · 2019-05-18T21:29:22Z

Ah found a solution, although not beautiful due to doubled subshell creation:

sudo bash -c ". /DietPi/dietpi/func/dietpi-globals && G_CONFIG_INJECT 'CONFIG_G_CHECK_URL_TIMEOUT=' 'CONFIG_G_CHECK_URL_TIMEOUT=20' /DietPi/dietpi.txt"

Sopor · 2019-05-18T21:38:03Z

Now you are confusing me ... What should i run?

MichaIng · 2019-05-18T21:55:22Z

@Sopor
Go with the interactive sudo shell for now: #2820 (comment)

Sopor · 2019-05-18T22:03:57Z

$ sudo /DietPi/dietpi/func/dietpi-set_software verify_dietpi.txt
[  OK  ] DietPi-Set_software | Root access verified.
[  OK  ] DietPi-Set_software | RootFS R/W access verified.

 DietPi-Set_software
─────────────────────────────────────────────────────
 Mode: verify_dietpi.txt ()

[  OK  ] DietPi-Set_software | Connection test: https://raw.githubusercontent.com/MichaIng/DietPi/master/dietpi.txt
[  OK  ] DietPi-Set_software | Patching dietpi.txt
[  OK  ] DietPi-Set_software | Downloading current dietpi.txt
[  OK  ] DietPi-Set_software | dietpi.txt verification completed
[  OK  ] verify_dietpi.txt  | Completed

$ sudo /DietPi/dietpi/dietpi-survey 1
[  OK  ] DietPi-Survey | Root access verified.
[  OK  ] DietPi-Survey | Connection test: ssh.dietpi.com

And now i should be able to run sudo dietpi-update -1 to reinstall the update?

MichaIng · 2019-05-18T22:05:21Z

@Sopor
Exactly. Actually just run dietpi-update 1 now, since I just released v6.24.1 which includes some ARMv6 hotfixes.

Sopor · 2019-05-18T22:07:10Z

Too late :(
But it do say i'm running v6.24.1
Do i need to re-run something?

MichaIng · 2019-05-18T22:09:22Z

@Sopor
Jep no issue, has exactly the same effect since you were already on v6.24. Nope no need to do anything else then.

Btw just to test, I added the fix for G_SUDO. Reload updated dietpi-globals into current session once and try it out:

. /DietPi/dietpi/func/dietpi-globals
G_SUDO G_CONFIG_INJECT 'CONFIG_G_CHECK_URL_TIMEOUT=' 'CONFIG_G_CHECK_URL_TIMEOUT=20' /DietPi/dietpi.txt

Sopor · 2019-05-18T22:26:03Z

$ . /DietPi/dietpi/func/dietpi-globals
$ G_SUDO G_CONFIG_INJECT 'CONFIG_G_CHECK_URL_TIMEOUT=' 'CONFIG_G_CHECK_URL_TIMEOUT=20' /DietPi/dietpi.txt
[  OK  ] G_CONFIG_INJECT | Desired setting in /DietPi/dietpi.txt was already set: CONFIG_G_CHECK_URL_TIMEOUT=20

MichaIng · 2019-05-18T22:37:14Z

@Sopor
Perfect, thanks! Okay I mark this as closed.

Sopor · 2019-05-20T22:51:11Z

@MichaIng
I did found the reason for the long ping. For some reason there was an old IP address in the network configuration from when i run it behind a router. For some reason the DNS was 192.168.1.1 instead of my ISP DNS address. I don't know why the old DNS was still there because i shutdown the RPi, moved the network cable from the router and to my switch and then i started the RPi again and got a public IP address. All other RPi OS's have worked perfect without leaving old DNS servers.
I removed the internal IP address and rebooted and now i have much better ping to github and other sites.

MichaIng · 2019-05-21T12:14:28Z

@Sopor
Great to hear. However in other cases ping was long even with g**gle DNS chosen.

I was rethinking our connection checks and download handling yesterday. Actually it is not really required in most cases to check the URL first and do the download afterwards. Checking URLs includes just the same connection + DNS + server response steps as the download itself, so this is just doubled... It would make sense instead to do the download only and use the same connection + DNS timeouts. Then read the exit code/error message to distinguish between connection errors and those that are due to e.g. download target disk size/permissions/corruption or such. Everything else is doubled effort + time.

Also the retry attempts only apply for the connection/URL tests. The actual download attempt itself uses wget defaults. This does not make much sense if something is wrong with the connection that single attempts can fail, then connection check could pass while download connection could fail. The retry option of error prompts anyway allow to retry as often as one wishes.

Was thinking to switch from wget to curl for downloads, since in my tests curl was significantly faster to initiate (of course download speed etc the same). Sadly you cannot specify DNS + server response timeouts separately. However perhaps the default is reasonable and okay (needs testing).

So basically we could use curl without any specific timeouts for usual downloads (in dietpi-software and such). Those play no role anyway in most cases since server downs/failures and wrong URLs and such result in an clear failure response so that curl/wget stop immediately, printing that failure/response code. So we can read the exit code and give reasonable ideas on the error prompt.

Without specifying timeouts:

Interactive run: Worst thing is that some phase of the download takes long. However better that than you face an error prompt even that the connection would have been successful with some seconds more.
Non-interactive run: Worst thing here as well is that it takes longer until it fails. However you will face an unfinished non-interactive script call in both cases and the skipped connection test could even safe more than than the missing timeout.

Last but not least, the wget --spider (as well as curl -I) URL checks fail in some cases, even that the download would succeed. For these cases currently we skip the connection check or do it on the main server URL instead of on the target file. Also here only the final download will show a meaningful error message/exit code.

... So finally I would skip those connection checks for downloads completely, instead implement some exit code based error handling as new G_DOWNLOAD function. If it is possible for curl to wait unlimited on any step and this cannot be prevented than stay with wget. The URL check makes sense then only for verifying entered APT mirrors, DNS servers, NTP mirrors, the general internet connection check within network adapters menu or general debugging reasons.

cURL exit codes: https://ec.haxx.se/usingcurl-returns.html

MichaIng added the Investigating 🤔 label May 18, 2019

MichaIng added Solution available 🥂 Definite solution has been done and removed Investigating 🤔 labels May 18, 2019

MichaIng added this to the v6.24 milestone May 18, 2019

MichaIng closed this as completed May 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resolving raw.githubusercontent.com (raw.githubusercontent.com)... failed: Connection timed out. #2820

Resolving raw.githubusercontent.com (raw.githubusercontent.com)... failed: Connection timed out. #2820

Sopor commented May 18, 2019 •

edited

Loading

Sopor commented May 18, 2019

Sopor commented May 18, 2019

MichaIng commented May 18, 2019

Sopor commented May 18, 2019 •

edited

Loading

MichaIng commented May 18, 2019 •

edited

Loading

Sopor commented May 18, 2019

MichaIng commented May 18, 2019

Sopor commented May 18, 2019 •

edited

Loading

MichaIng commented May 18, 2019

Sopor commented May 18, 2019

Sopor commented May 18, 2019

MichaIng commented May 18, 2019 •

edited

Loading

Sopor commented May 18, 2019

MichaIng commented May 18, 2019 •

edited

Loading

MichaIng commented May 18, 2019

Sopor commented May 18, 2019

MichaIng commented May 18, 2019

Sopor commented May 18, 2019

MichaIng commented May 18, 2019

Sopor commented May 18, 2019 •

edited

Loading

MichaIng commented May 18, 2019

Sopor commented May 18, 2019

MichaIng commented May 18, 2019 •

edited

Loading

Sopor commented May 20, 2019

MichaIng commented May 21, 2019 •

edited

Loading

Resolving raw.githubusercontent.com (raw.githubusercontent.com)... failed: Connection timed out. #2820

Resolving raw.githubusercontent.com (raw.githubusercontent.com)... failed: Connection timed out. #2820

Comments

Sopor commented May 18, 2019 • edited Loading

Details:

Steps to reproduce:

Expected behaviour:

Actual behaviour:

Extra details:

Additional logs:

Sopor commented May 18, 2019

Sopor commented May 18, 2019

MichaIng commented May 18, 2019

Sopor commented May 18, 2019 • edited Loading

MichaIng commented May 18, 2019 • edited Loading

Sopor commented May 18, 2019

MichaIng commented May 18, 2019

Sopor commented May 18, 2019 • edited Loading

MichaIng commented May 18, 2019

Sopor commented May 18, 2019

Sopor commented May 18, 2019

MichaIng commented May 18, 2019 • edited Loading

Sopor commented May 18, 2019

MichaIng commented May 18, 2019 • edited Loading

MichaIng commented May 18, 2019

Sopor commented May 18, 2019

MichaIng commented May 18, 2019

Sopor commented May 18, 2019

MichaIng commented May 18, 2019

Sopor commented May 18, 2019 • edited Loading

MichaIng commented May 18, 2019

Sopor commented May 18, 2019

MichaIng commented May 18, 2019 • edited Loading

Sopor commented May 20, 2019

MichaIng commented May 21, 2019 • edited Loading

Sopor commented May 18, 2019 •

edited

Loading

Sopor commented May 18, 2019 •

edited

Loading

MichaIng commented May 18, 2019 •

edited

Loading

Sopor commented May 18, 2019 •

edited

Loading

MichaIng commented May 18, 2019 •

edited

Loading

MichaIng commented May 18, 2019 •

edited

Loading

Sopor commented May 18, 2019 •

edited

Loading

MichaIng commented May 18, 2019 •

edited

Loading

MichaIng commented May 21, 2019 •

edited

Loading