Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Network | Rock(Pro)64: "ip a" reports "UNKNOWN" connection state #3066

Closed
svh1985 opened this issue Aug 20, 2019 · 13 comments
Closed

Network | Rock(Pro)64: "ip a" reports "UNKNOWN" connection state #3066

svh1985 opened this issue Aug 20, 2019 · 13 comments
Labels
External bug 🐞 For bugs which are not caused by DietPi. ROCK64 ROCKPro64 Solution available 🥂 Definite solution has been done
Milestone

Comments

@svh1985
Copy link
Contributor

svh1985 commented Aug 20, 2019

ADMIN EDIT

  • Bug is that ip a eth0 shows as connection state UNKNOWN, and in some cases the Ethernet connection seems to be unstable, with lots of TX errors.

Apply the workaround

cp -a /boot/boot.scr /boot/boot.scr.bak
identifier='ff540000'
(( $G_HW_MODEL == 42 )) && identifier='fe300000'
sed -i "/^fdt resize/s|$|\nfdt rm /ethernet@$identifier rockchip,bugged_tx_coe\nfdt rm /ethernet@$identifier snps,force_thresh_dma_mode\nfdt set /ethernet@$identifier snps,txpbl <0x21>|" /boot/boot.cmd
mkimage -C none -A arm -T script -d /boot/boot.cmd /boot/boot.scr
  • Please only do this on RockPro64, if you can access the SDcard ext4 partition, to recover boot.scr.bak, if anything fails. On Rock64 however it is safe.
  • Before reboot, check if ifdown eth0 && ifup eth0 already fixes the connection state, so that it shows UP.
    NB: This drops the network connection and restarts it, so at best be connected via keyboard and monitor instead of SSH.
  • After reboot, check if ip a eth0 not shows connection state as UP.
  • If not, then check if ifdown eth0 && ifup eth0 solves this now.

Creating a bug report/issue

Required Information

G_DIETPI_VERSION_CORE=6
G_DIETPI_VERSION_SUB=25
G_DIETPI_VERSION_RC=3
G_GITBRANCH='master'
G_GITOWNER='MichaIng'
Stretch
Linux RockPi 4.4.182-rockchip64 #1 SMP Fri Jun 28 17:34:00 CEST 2019 aarch64 GNU/Linux
Rock64 (aarch64)
Original PSU
EMMC

Steps to reproduce

Login and run

cat /DietPi/dietpi/.network
0
0
NONE
Use dietpi-config to setup a connection
ETH_IP=172.16.0.24
WLAN_IP=

Expected behaviour

  • The 4th line should show the IP, not "Use dietpi-config to setup a connection"
  • It seems to be a bug in the for loop: I tried to run the file ./obtain_network_details with some echo's in the ETH loop but they did not show

Weird because the eth0 is there:

ll /sys/class/net/
total 0
lrwxrwxrwx 1 root root 0 Aug 20 21:55 docker0 -> ../../devices/virtual/net/docker0
lrwxrwxrwx 1 root root 0 Aug 20 21:55 eth0 -> ../../devices/platform/ff540000.ethernet/net/eth0
lrwxrwxrwx 1 root root 0 Aug 20 21:55 lo -> ../../devices/virtual/net/lo
lrwxrwxrwx 1 root root 0 Aug 20 21:55 tun0 -> ../../devices/virtual/net/tun0
lrwxrwxrwx 1 root root 0 Aug 20 21:55 veth2e9149a -> ../../devices/virtual/net/veth2e9149a
lrwxrwxrwx 1 root root 0 Aug 20 21:55 veth8c30c6a -> ../../devices/virtual/net/veth8c30c6a
@MichaIng
Copy link
Owner

MichaIng commented Aug 20, 2019

@svh1985
Many thanks for your report. Can you paste, please: ip a

Since ETH_IP is set, the loop runs through. 172.16.0.24 is the correct IP applied to eth0?

I tried to run the file ./obtain_network_details with some echo's in the ETH loop but they did not show

Could you paste the ETH loop block, where/how you placed the echos?

Note that an iface is only recognised as "active", when ip a s <iface> contains the "UP" state string, which should always be there if the device is connected. line 3 contains "NONE" which means that eth0 obviously shows "DOWN" or something else. And when no active iface has been found, no related IP is estimated.


I see you have quite some virtual interfaces, VLANs, docker and OpenVPN. The latter two should not interfere, but I am not sure if the VLANs attached to eth0 have an effect on its connection state or lets say the ip a s eth0 output in general 🤔.

@svh1985
Copy link
Contributor Author

svh1985 commented Aug 21, 2019

The output. I'm guessing it has something to do with the UNKNOWN state no?

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether 92:26:27:9c:2b:7a brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.24/24 brd 172.16.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::9026:27ff:fe9c:2b7a/64 scope link
       valid_lft forever preferred_lft forever
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:03:95:c0:e1 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:3ff:fe95:c0e1/64 scope link
       valid_lft forever preferred_lft forever
6: veth8c30c6a@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
    link/ether 36:de:57:7e:eb:b0 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::34de:57ff:fe7e:ebb0/64 scope link
       valid_lft forever preferred_lft forever
8: veth2e9149a@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
    link/ether 6a:b2:a6:07:6f:14 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::68b2:a6ff:fe07:6f14/64 scope link
       valid_lft forever preferred_lft forever
9: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none
    inet 10.8.2.15/24 brd 10.8.2.255 scope global tun0
       valid_lft forever preferred_lft forever

I read that there were some fixes in the ayufan kernel, maybe that can fix the UNKNOWN state. Not sure how to install tho 🙈

https://forum.armbian.com/topic/9812-rock64-random-nic-failure/

Other than this, my NIC works as expected btw.

@MichaIng
Copy link
Owner

MichaIng commented Aug 21, 2019

@svh1985
Probably related, at least state UNKNOWN shows up in your link as well. That is the reason why obtain_network_details does not show an active interface+IP. Probably we can implement a workaround for Rock64 explicitly to allow "UNKOWN" as well, until there is definitely a fix 🤔.

To add/try the fix:

  • Edit /boot/boot.cmd
  • Directly after the line beginning with fdt resize, add:
fdt rm /ethernet@ff540000 rockchip,bugged_tx_coe
fdt rm /ethernet@ff540000 snps,force_thresh_dma_mode
fdt set /ethernet@ff540000 snps,txpbl <0x21>
  • Then run: mkimage -C none -A arm -T script -d /boot/boot.cmd /boot/boot.scr

If this helps, we can add it with next update to all RockPis.


Actually I am thinking if this is related to: #2028 (comment)
For this we already implemented a workaround: https://github.com/MichaIng/DietPi/blob/dev/rootfs/etc/network/if-up.d/dietpi-disable_offload

EDIT: Nope, symptoms are different and our workaround is in place.


Just to sort it out, if the above bootloader additions do not help, you could try to disable docker, OpenVPN and VLAN interfaces temporarily and ifdown eth0 && ifup eth0 to reinit the interface, as long as you have local monitor + keyboard connection of course, so only eth0 (and lo) is up, then check ip a s eth0 for the state string again. Just to verify that virtual interfaces, using the physical ones, do not affect their reported states somehow.

@svh1985
Copy link
Contributor Author

svh1985 commented Aug 21, 2019

I added the lines, ran the command and rebooted but still UNKNOWN.

When I'm home I'll try to disable the adapters and do the if up/down.

Do you think a kernel upgrade will help?

@svh1985
Copy link
Contributor Author

svh1985 commented Aug 22, 2019

I just logged in and the IP now shows on the dietpi-banner, so it seems it DID solve the issue!

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 92:26:27:9c:2b:7a brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.24/24 brd 172.16.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::9026:27ff:fe9c:2b7a/64 scope link
       valid_lft forever preferred_lft forever
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:a1:07:6d:f4 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:a1ff:fe07:6df4/64 scope link
       valid_lft forever preferred_lft forever
6: vethe1f98d9@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
    link/ether 02:99:c2:c6:f7:cf brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::99:c2ff:fec6:f7cf/64 scope link
       valid_lft forever preferred_lft forever
8: vetha8676e8@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
    link/ether 4a:2c:21:8d:56:68 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::482c:21ff:fe8d:5668/64 scope link
       valid_lft forever preferred_lft forever
9: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none
    inet 10.8.0.13/24 brd 10.8.0.255 scope global tun0
       valid_lft forever preferred_lft forever

@svh1985
Copy link
Contributor Author

svh1985 commented Aug 26, 2019

Small update:
I've ran ifdown eth0 && ifup eth0 without disabling the virtual adapters and the state was set to UP.
I then logged out/in and the banner also showed the correct IP.

@MichaIng
Copy link
Owner

@svh1985
Okay that is great. I will check our survey uploads if this issue is apparent on most Rock64 or even RockPro64.

If there is no kernel update available until v6.26 release, we will patch this workaround in. Might you give us a hint here, if you receive one? apt update should tell you, apt upgrade holds them back by default, apt full-upgrade will apply them.

Btw good find about the issue on ARMbian 👍.

@MichaIng MichaIng added the Workaround available 🆗 Workaround is available/has been implemented, but a definite solution should be found when possible. label Aug 26, 2019
@MichaIng MichaIng added this to the v6.26 milestone Aug 26, 2019
@MichaIng MichaIng changed the title obtain_network_details does not recognize ETH IP Network | Rock64: "ip a" reports "UNKNOWN" connection state Aug 26, 2019
@svh1985
Copy link
Contributor Author

svh1985 commented Aug 27, 2019

Ran the command this is the output.
Don't know how the kernel upgrade should look like, but I would guess linux-stretch-root-rock64/stretch 5.90 arm64 is the one?

root@RockPi:~# apt update
Hit:1 https://apt.armbian.com stretch InRelease
Hit:2 https://download.docker.com/linux/debian stretch InRelease
Ign:3 https://downloads.plex.tv/repo/deb public InRelease
Hit:4 https://downloads.plex.tv/repo/deb public Release
Hit:5 https://download.mono-project.com/repo/debian stretch InRelease
Ign:6 https://cdn-aws.deb.debian.org/debian stretch InRelease
Get:7 https://cdn-aws.deb.debian.org/debian stretch-updates InRelease [91.0 kB]
Get:8 https://cdn-aws.deb.debian.org/debian-security stretch/updates InRelease [94.3 kB]
Get:9 https://cdn-aws.deb.debian.org/debian stretch-backports InRelease [91.8 kB]
Hit:11 https://cdn-aws.deb.debian.org/debian stretch Release
Get:12 https://cdn-aws.deb.debian.org/debian-security stretch/updates/main arm64 Packages [482 kB]
Get:13 https://cdn-aws.deb.debian.org/debian-security stretch/updates/main armhf Packages [488 kB]
Fetched 1,247 kB in 14s (88.0 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
3 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@RockPi:~# apt list --upgradable
Listing... Done
armbian-tools-stretch/stretch 5.75 arm64 [upgradable from: 5.69]
linux-stretch-root-rock64/stretch 5.90 arm64 [upgradable from: 5.73]
plexmediaserver/public 1.16.5.1554-1e5ff713d arm64 [upgradable from: 1.16.4.1469-6d5612c2f]
root@RockPi:~#

@MichaIng
Copy link
Owner

@svh1985
Nope, the kernel update would be linux-image-* package. linux-<distro>-root- is the package that contains some ARMbian specific services and binaries + system config files and such. Most of this is actually not what we want or need, their services either double with ours or are even incompatible/cause issues, like their zRam implementation.

Out of interest, does the following print any active processes, after you applied the apt upgrade?
systemctl status armbian*

@MichaIng
Copy link
Owner

MichaIng commented Aug 27, 2019

# grep 'Rock64' [a-f]* | wc -l
202
# grep '((aNETWORK_INTERFACE\[NONE\]++))' $(grep 'Rock64' [a-f]* | sed 's/:.*$//') | wc -l
78
# grep 'RockPro64' [a-f]* | wc -l
57
# grep '((aNETWORK_INTERFACE\[NONE\]++))' $(grep 'RockPro64' [a-f]* | sed 's/:.*$//') | wc -l
26
  • ~40% of all Rock64 and RockPro64 report no active network adapter.
  • More than a quarter in average run a WiFi adapter: https://dietpi.com/survey/#network
  • So the value for Ethernet adapters should be even worse.
  • Taking into account that these include reports from beginning of the year, including old Jessie images, I would not wonder if nearly all current Stretch Rock(Pro)64 are affected.

To compare with other devices:

# grep 'Odroid' [a-f]* | wc -l
744
# grep '((aNETWORK_INTERFACE\[NONE\]++))' $(grep 'Odroid' [a-f]* | sed 's/:.*$//') | wc -l
0
# grep 'RPi' [a-f]* | wc -l
7306
# grep '((aNETWORK_INTERFACE\[NONE\]++))' $(grep 'RPi' [a-f]* | sed 's/:.*$//') | wc -l
45
# grep '((aNETWORK_INTERFACE\[NONE\]++))' $(grep 'Nano' [a-f]* | sed 's/:.*$//') | wc -l
0
# grep 'Asus' [a-f]* | wc -l
252
# grep '((aNETWORK_INTERFACE\[NONE\]++))' $(grep 'Asus' [a-f]* | sed 's/:.*$//') | wc -l
9
# grep 'Virtual' [a-f]* | wc -l
639
# grep '((aNETWORK_INTERFACE\[NONE\]++))' $(grep 'Virtual' [a-f]* | sed 's/:.*$//') | wc -l
13
# grep 'Native' [a-f]* | wc -l
243
# grep '((aNETWORK_INTERFACE\[NONE\]++))' $(grep 'Native' [a-f]* | sed 's/:.*$//') | wc -l
5

Or the other way round:

# grep '((aNETWORK_INTERFACE\[NONE\]++))' $(grep -E 'Rock(Pro)?64' [a-f]* | sed 's/:.*$//') | wc -l
104
# grep '((aNETWORK_INTERFACE\[NONE\]++))' $(grep -E 'Rock(Pro)?64' [0-4]* | sed 's/:.*$//') | wc -l
77
# grep '((aNETWORK_INTERFACE\[NONE\]++))' $(grep -E 'Rock(Pro)?64' [5-9]* | sed 's/:.*$//') | wc -l
96

Must be scanned in smaller portions as 92428 files are too much input arguments for grep 😄!

  • 277 out of 477 devices reported without active network interface are Rock64 or RockPro64.
  • But they are only 713 out of 27209 overall reported devices: https://dietpi.com/survey/#device

If you ask me, the data give a VERY clear picture 👍.

@MichaIng MichaIng added External bug 🐞 For bugs which are not caused by DietPi. ROCKPro64 and removed Investigating 🤔 labels Aug 27, 2019
MichaIng added a commit that referenced this issue Aug 31, 2019
v6.26

Verified

This commit was signed with the committer’s verified signature.
Two-Hearts Patrick Zheng
+ DietPi-PREP | Replace obsolete Rock64 GPU acceleration fix for Ayufan images with Ethernet fix for Rock64 and RockPro64: #3066
+ DietPi-PREP | Apply Meveric backports APT preference on Meveric images only. E.g. out official Odroid C1 image is not based on Meveric, thus APT preferences or obsolete.
@MichaIng
Copy link
Owner

PR up to apply the fix to all new and existing Rock64 and RockPro64 images: #3087

MichaIng added a commit that referenced this issue Aug 31, 2019
+ DietPi-Patch | Rock(Pro)64: Apply workaround for kernel-related Ethernet issues: #3066
@svh1985
Copy link
Contributor Author

svh1985 commented Sep 1, 2019

I did some additional testing and I have some bad news: it only works when you run ifdown eth0 && ifup eth0 after every reboot. The fix does not create a persistent fix.
So, I don't think this fixes the issue in a way that is wanted.

@MichaIng
Copy link
Owner

MichaIng commented Sep 1, 2019

@svh1985
Okay. It could then be tested, if the 3 lines in /boot/bood.cmd are even required to solve the UNKNOWN interface state, of if ifdown eth0 && ifup eth0 alone solves this:

sed-Ei '/(f[ef](54|30)0000/d' /boot/bood.cmd
mkimage -C none -A arm -T script -d /boot/boot.cmd /boot/boot.scr
reboot
ifdown eth0 && ifup eth0
ip a eth0

However we'll probably leave it inside, as it seems to fix some TX errors and NIC crashes according to discussion on ARMbian forum. However I am not sure if/when some fix will be in the kernel, or if it is already? Another reason to check if the fix really has any effect.

To fix UNKNOWN state, we could otherwise add eth0 restart to DietPi-Boot script. I will add a testing request to MOTD. Would be good to find some more testers, especially ones with RockPro64.

@MichaIng MichaIng changed the title Network | Rock64: "ip a" reports "UNKNOWN" connection state Network | Rock(Pro)64: "ip a" reports "UNKNOWN" connection state Sep 1, 2019
MichaIng added a commit that referenced this issue Sep 16, 2019
+ DietPi-PREP | Replace obsolete Rock64 GPU acceleration fix for Ayufan images with Ethernet fix for Rock64 and RockPro64: #3066
+ DietPi-PREP | Apply Meveric backports APT preference on Meveric images only. E.g. our official Odroid C1 image is not based on Meveric, thus APT preferences are obsolete.
+ DietPi-Patch | Rock(Pro)64: Apply workaround for kernel-related Ethernet issues: #3066
@MichaIng MichaIng mentioned this issue Sep 16, 2019
@MichaIng MichaIng added Solution available 🥂 Definite solution has been done and removed Workaround available 🆗 Workaround is available/has been implemented, but a definite solution should be found when possible. labels Mar 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
External bug 🐞 For bugs which are not caused by DietPi. ROCK64 ROCKPro64 Solution available 🥂 Definite solution has been done
Projects
None yet
Development

No branches or pull requests

2 participants