Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

raspberrypi 3b wifi 802.1x benchmark, kernel Internal error: Oops: 17 [#1] SMP ARM for BCM2835 #2794

Open
cxueqin opened this issue Jan 5, 2019 · 20 comments
Labels
Close within 30 days Issue will be closed within 30 days unless requested to stay open Waiting for external input Waiting for a comment from the originator of the issue, or a collaborator.

Comments

@cxueqin
Copy link

cxueqin commented Jan 5, 2019

Hi friend:
I use raspberrypi 3B board as a wifi station mode to connect to an AP. The wifi connection was made by wpa_supplicant with 802.1X authtication params. The 802.1x params list below:

  • key_mgmt : WPA-EAP
  • eap: PEAP
  • proto: WPA RSN
  • pairwise: CCMP
  • group: CCMP
  • NetworkIdentity
  • password

The wifi connection was fine at first. I used wpa_cli to reassociate the AP with the same params every two minutes. In other words, the rp 3b board test 802.1x access authentication every two minutes. After hundreds of successful reassociations, the kernel went wrong. The syslog shown

Jan 4 07:14:40 NetFrog kernel: [21393.314945] brcmfmac: brcmf_cfg80211_scan: scan error (-11)
Jan 4 07:14:40 NetFrog kernel: [21393.315365] brcmfmac: brcmf_cfg80211_escan: Connecting: status (3)
Jan 4 07:14:40 NetFrog kernel: [21393.315375] brcmfmac: brcmf_cfg80211_scan: scan error (-11)
Jan 4 07:14:45 NetFrog kernel: [21398.610986] ------------[ cut here ]------------
Jan 4 07:14:45 NetFrog kernel: [21398.611241] WARNING: CPU: 0 PID: 42 at net/wireless/sme.c:752 __cfg80211_connect_result+0x398/0x3e4 [cfg80211]
Jan 4 07:14:45 NetFrog kernel: [21398.611248] Modules linked in: fuse rfcomm cmac bnep hci_uart btbcm serdev bluetooth ecdh_generic brcmfmac brcmutil cfg80211 rfkill snd_bcm2835(C) snd_pcm snd_timer snd asix uio_pdrv_genirq fixed uio i2c_dev ip_tables x_tables ipv6
Jan 4 07:14:45 NetFrog kernel: [21398.611376] CPU: 0 PID: 42 Comm: kworker/u8:1 Tainted: G WC 4.14.90-v7+ #1183
Jan 4 07:14:45 NetFrog kernel: [21398.611381] Hardware name: BCM2835
Jan 4 07:14:45 NetFrog kernel: [21398.611571] Workqueue: cfg80211 cfg80211_event_work [cfg80211]
Jan 4 07:14:45 NetFrog kernel: [21398.611605] [<8010ff30>] (unwind_backtrace) from [<8010c174>] (show_stack+0x20/0x24)
Jan 4 07:14:45 NetFrog kernel: [21398.611622] [<8010c174>] (show_stack) from [<8078b864>] (dump_stack+0xd4/0x118)
Jan 4 07:14:45 NetFrog kernel: [21398.611639] [<8078b864>] (dump_stack) from [<8011dd70>] (__warn+0xf8/0x110)
Jan 4 07:14:45 NetFrog kernel: [21398.611654] [<8011dd70>] (__warn) from [<8011de58>] (warn_slowpath_null+0x30/0x38)
Jan 4 07:14:45 NetFrog kernel: [21398.611850] [<8011de58>] (warn_slowpath_null) from [<7f22ed4c>] (__cfg80211_connect_result+0x398/0x3e4 [cfg80211])
Jan 4 07:14:45 NetFrog kernel: [21398.612227] [<7f22ed4c>] (__cfg80211_connect_result [cfg80211]) from [<7f202ec4>] (cfg80211_process_wdev_events+0x10c/0x14c [cfg80211])
Jan 4 07:14:45 NetFrog kernel: [21398.612598] [<7f202ec4>] (cfg80211_process_wdev_events [cfg80211]) from [<7f202f40>] (cfg80211_process_rdev_events+0x3c/0x70 [cfg80211])
Jan 4 07:14:45 NetFrog kernel: [21398.612965] [<7f202f40>] (cfg80211_process_rdev_events [cfg80211]) from [<7f1fd294>] (cfg80211_event_work+0x24/0x2c [cfg80211])
Jan 4 07:14:45 NetFrog kernel: [21398.613158] [<7f1fd294>] (cfg80211_event_work [cfg80211]) from [<801379f0>] (process_one_work+0x158/0x454)
Jan 4 07:14:45 NetFrog kernel: [21398.613173] [<801379f0>] (process_one_work) from [<80137d50>] (worker_thread+0x64/0x5b8)
Jan 4 07:14:45 NetFrog kernel: [21398.613187] [<80137d50>] (worker_thread) from [<8013ddd4>] (kthread+0x13c/0x16c)
Jan 4 07:14:45 NetFrog kernel: [21398.613205] [<8013ddd4>] (kthread) from [<801080ac>] (ret_from_fork+0x14/0x28)
Jan 4 07:14:45 NetFrog kernel: [21398.613293] ---[ end trace 5eac627de54b823b ]---
Jan 4 07:14:46 NetFrog kernel: [21399.528228] Unable to handle kernel NULL pointer dereference at virtual address 00000000
Jan 4 07:14:46 NetFrog kernel: [21399.528253] pgd = a50b0000
Jan 4 07:14:46 NetFrog kernel: [21399.528264] [00000000] *pgd=373d8835, *pte=00000000, *ppte=00000000
Jan 4 07:14:46 NetFrog kernel: [21399.528293] Internal error: Oops: 17 [#1] SMP ARM
Jan 4 07:14:46 NetFrog kernel: [21399.528304] Modules linked in: fuse rfcomm cmac bnep hci_uart btbcm serdev bluetooth ecdh_generic brcmfmac brcmutil cfg80211 rfkill snd_bcm2835(C) snd_pcm snd_timer snd asix uio_pdrv_genirq fixed uio i2c_dev ip_tables x_tables ipv6
Jan 4 07:14:46 NetFrog kernel: [21399.528432] CPU: 1 PID: 1176 Comm: wpa_supplicant Tainted: G WC 4.14.90-v7+ #1183
Jan 4 07:14:46 NetFrog kernel: [21399.528442] Hardware name: BCM2835
Jan 4 07:14:46 NetFrog kernel: [21399.528453] task: b674cb00 task.stack: a790e000
Jan 4 07:14:46 NetFrog kernel: [21399.528476] PC is at call_commit_handler.part.2+0x18/0x44
Jan 4 07:14:46 NetFrog kernel: [21399.528490] LR is at ioctl_standard_call+0x270/0x4bc
Jan 4 07:14:46 NetFrog kernel: [21399.528502] pc : [<80785b08>] lr : [<80785da4>] psr: 20000013
Jan 4 07:14:46 NetFrog kernel: [21399.528513] sp : a790fde0 ip : a790fdf0 fp : a790fdec
Jan 4 07:14:46 NetFrog kernel: [21399.528523] r10: b9272800 r9 : 00000034 r8 : 00008b1a
Jan 4 07:14:46 NetFrog kernel: [21399.528535] r7 : 0000000e r6 : a790fea0 r5 : 00000003 r4 : 00000002
Jan 4 07:14:46 NetFrog kernel: [21399.528547] r3 : 00000000 r2 : 60000013 r1 : 60000013 r0 : b9272800
Jan 4 07:14:46 NetFrog kernel: [21399.528560] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
Jan 4 07:14:46 NetFrog kernel: [21399.528572] Control: 10c5383d Table: 250b006a DAC: 00000055
Jan 4 07:14:46 NetFrog kernel: [21399.528584] Process wpa_supplicant (pid: 1176, stack limit = 0xa790e210)
Jan 4 07:14:46 NetFrog kernel: [21399.528595] Stack: (0xa790fde0 to 0xa7910000)
Jan 4 07:14:46 NetFrog kernel: [21399.528613] fde0: a790fe5c a790fdf0 80785da4 80785afc 806ab2ac a790fe6c a790fe6c a790feb0
Jan 4 07:14:46 NetFrog kernel: [21399.528630] fe00: 00000000 a536aa80 80697fd0 636ab284 616e6968 6566696c 6669772d 8068bc69
Jan 4 07:14:46 NetFrog kernel: [21399.528648] fe20: a790fe44 807a4150 a790fe4c a790fe38 807a4150 00008b1a a790fea0 80c7a840
Jan 4 07:14:46 NetFrog kernel: [21399.528665] fe40: 7efe67b8 00000000 a790e000 00000036 a790fe94 a790fe60 807862e8 80785b40
Jan 4 07:14:46 NetFrog kernel: [21399.528682] fe60: 7f249c5c 80c7a840 7efe67b8 00008b1a 00000004 00008b1a 80c7a840 7efe67b8
Jan 4 07:14:46 NetFrog kernel: [21399.528699] fe80: 80c7a840 00000004 a790feec a790fe98 806b59b4 8078613c a790ff14 a790fea8
Jan 4 07:14:46 NetFrog kernel: [21399.528717] fea0: 6e616c77 00000030 00000000 00000000 7efe67d8 0001000e 00000000 00000000
Jan 4 07:14:46 NetFrog kernel: [21399.528734] fec0: a790ff14 00008b1a 0000012a 7efe67b8 80c7a840 00000004 a790e000 00000036
Jan 4 07:14:46 NetFrog kernel: [21399.528752] fee0: a790ff0c a790fef0 80671330 806b5924 7efe67b8 abf6eaa0 b67fcd80 00008b1a
Jan 4 07:14:46 NetFrog kernel: [21399.528769] ff00: a790ff7c a790ff10 8029f060 80671218 807a1834 80145efc 00000128 801081c4
Jan 4 07:14:46 NetFrog kernel: [21399.528787] ff20: a790e000 00000128 a790ff94 a790ff38 808024bc 397d3000 807a1e98 ba361d40
Jan 4 07:14:46 NetFrog kernel: [21399.528803] ff40: 00000000 00000000 a790fe88 a790e000 a790e010 b67fcd80 7efe67b8 b67fcd80
Jan 4 07:14:46 NetFrog kernel: [21399.528820] ff60: 00008b1a 00000004 a790e000 00000036 a790ffa4 a790ff80 8029f7bc 8029efc0
Jan 4 07:14:46 NetFrog kernel: [21399.528839] ff80: 00000016 0000000e 019773c0 7efe67d8 00000036 801081c4 00000000 a790ffa8
Jan 4 07:14:46 NetFrog kernel: [21399.528855] ffa0: 80108000 8029f784 0000000e 019773c0 00000004 00008b1a 7efe67b8 00000016
Jan 4 07:14:46 NetFrog kernel: [21399.528872] ffc0: 0000000e 019773c0 7efe67d8 00000036 01989704 00000000 001bc730 019773d0
Jan 4 07:14:46 NetFrog kernel: [21399.528889] ffe0: 001bcbac 7efe67b4 00141478 76c7680c 20000010 00000004 00000000 00000000
Jan 4 07:14:46 NetFrog kernel: [21399.528920] [<80785b08>] (call_commit_handler.part.2) from [<80785da4>] (ioctl_standard_call+0x270/0x4bc)
Jan 4 07:14:46 NetFrog kernel: [21399.528942] [<80785da4>] (ioctl_standard_call) from [<807862e8>] (wext_handle_ioctl+0x1b8/0x20c)
Jan 4 07:14:46 NetFrog kernel: [21399.528967] [<807862e8>] (wext_handle_ioctl) from [<806b59b4>] (dev_ioctl+0x9c/0x8e8)
Jan 4 07:14:46 NetFrog kernel: [21399.528992] [<806b59b4>] (dev_ioctl) from [<80671330>] (sock_ioctl+0x124/0x2d8)
Jan 4 07:14:46 NetFrog kernel: [21399.529016] [<80671330>] (sock_ioctl) from [<8029f060>] (do_vfs_ioctl+0xac/0x7c4)
Jan 4 07:14:46 NetFrog kernel: [21399.529037] [<8029f060>] (do_vfs_ioctl) from [<8029f7bc>] (SyS_ioctl+0x44/0x6c)
Jan 4 07:14:46 NetFrog kernel: [21399.529059] [<8029f7bc>] (SyS_ioctl) from [<80108000>] (ret_fast_syscall+0x0/0x28)
Jan 4 07:14:46 NetFrog kernel: [21399.529079] Code: e24cb004 e52de004 e8bd4000 e5903120 (e5933000)
Jan 4 07:14:46 NetFrog kernel: [21399.529094] ---[ end trace 5eac627de54b823c ]---

The syslog told kernel Internal error: Oops: 17 [#1] SMP ARM for BCM2835. After that the wpa_supplicant didn't work, the system cpu usage increased gradually which would reach 100 or higher(measured by uptime). And then the linux system didn't work . SSH service couldn't be reach, but icmp ping was still alive.

I tried the above test several times on the same and different raspberry pi 3B boards. The kernel error problem could be reproduced. The kernel firmware was upgraded to lastest 4.14.90-v7+.

Any suggestion is welcome. Thank you !

@pelwell
Copy link
Contributor

pelwell commented Jan 6, 2019

We like reproducible errors, and we like ones that can be reproduced in minutes (not hours) even more. Here's what I suggest you do:

  1. Try reducing the 2 minute interval and see if it fails quicker. The aim is to have a guaranteed (or very likely) failure as quickly as possible.
  2. Attach (or provide a link to) the test script, along with any modifications needed to stock Raspbian (e.g. a custom wpa_supplicant.conf).

@cxueqin cxueqin changed the title raspberrypi 3b kernel Internal error: Oops: 17 [#1] SMP ARM for BCM2835 #195 raspberrypi 3b wifi 802.1x benchmark, kernel Internal error: Oops: 17 [#1] SMP ARM for BCM2835 Jan 8, 2019
@cxueqin
Copy link
Author

cxueqin commented Jan 8, 2019

@pelwell I used nodejs to send command to wpa_supplicant ctrl_interface to send wifi test command. The test logic:

  1. enable a specific wifi network
  2. setNetworkSSID
  3. setNetworkVariable key_mgmt
  4. setNetworkVariable eap
  5. setNetworkVariable proto WPA RSN
  6. setNetworkVariable pairwise CCMP
  7. setNetworkVariable group CCMP
  8. setNetworkIdentity
  9. setNetworkVariable password
  10. enableNetwork
  11. reassociate

The test logic is repeated every two minutes. Sometimes after hundreds of tests , the kernel gave many error messages, such as
image

And in some cases , kernel will failed.
image

The test script and wpa_supplicant config file attached here.
Just unzip it and run node wifi_test.js to reproduce the problem.

wifi_test.zip

Many thanks.

@pelwell
Copy link
Contributor

pelwell commented Jan 8, 2019

Does it fail quicker with a shorter interval?

@cxueqin
Copy link
Author

cxueqin commented Jan 8, 2019

Does it fail quicker with a shorter interval?

I will try to do the test with a shorter interval, and tell the result.

But in syslog , there was an noteable line, "Unable to handle kernel NULL pointer deference at virtual address 000000". Maybe some bug in wifi driver or other modules cause the problem?

@pelwell
Copy link
Contributor

pelwell commented Jan 8, 2019

Yes, there is almost certainly a bug in the driver - it shouldn't crash.

There are never no outstanding issues, so you are competing with other users for our support. The time-to-failure is important because the shorter it is, the more chance an engineer will take the time to look at it.

@cxueqin
Copy link
Author

cxueqin commented Jan 8, 2019

@pelwell ok, thank u for reply. The kernel message file , some kernel log message like this
image
Maybe the log message has some relation with the bug.

@cxueqin
Copy link
Author

cxueqin commented Jan 10, 2019

@pelwell The wifi kernel crash could be easily using my test script in one minute interval. I reproduced the crash in two respberry pi 3B board. One kernel version is 4.14.79-v7+ #1159, and the other one is 4.14.90-v7+ #1183. The crash backstrace is almost same.

4.14.90-v7 crash log:
image

4.14.79-v7+ #1159 crash log:
image

Is there any way to avoid this problem in wifi 802.1x mode? Thanks

@pelwell
Copy link
Contributor

pelwell commented Jan 10, 2019

I've been running a modified version of your script for about an hour now, reconnecting every 20 seconds. In that time the kernel log has accumulated 4 "scan error (-16)"s, and no crashes.

However, the script modification was to switch to WPA-PSK authentation rather than WPA-EAP, which I am unable to test. Are you able to try with WPA-PSK to see if it is more or less reliable?

@cxueqin
Copy link
Author

cxueqin commented Jan 10, 2019

@pelwell In my wifi 802.1x test, sometimes the raspberry pi system was stable for serveral days, sometimes kernel crash in hours. General speaking, the chance of kernel crash was high in my enviroment.

Now I am doing some PSK authentation test. I will report the result later. Thanks.

@cxueqin
Copy link
Author

cxueqin commented Jan 10, 2019

@pelwell I found when the wifi kernel crash problem happened, the raspberry pi system worked in abnormal way. The system didn't response to reboot command. The ssh service didn't work. The cpu usage became very high , sometimes load avg reached 100 or higher. I think it is a critical problem.

@cxueqin
Copy link
Author

cxueqin commented Jan 11, 2019

@pelwell Since we have no idea the reason of the kernel crash , could some action to be taken to detect the kernel crash. When kernel crash is detected, the system will be rebooted automatically to avoid the bad situaction. Any way to achieve that goal? Thanks.

The newest kernel crash log with official kernel version 4.14.79-v7+ #1159.

Jan 11 01:19:00 NetFrog kernel: [ 8360.000887] Unable to handle kernel NULL pointer dereference at virtual address 00000000
Jan 11 01:19:00 NetFrog kernel: [ 8360.000909] pgd = a7950000
Jan 11 01:19:00 NetFrog kernel: [ 8360.000920] [00000000] *pgd=3708a835, *pte=00000000, *ppte=00000000
Jan 11 01:19:00 NetFrog kernel: [ 8360.000949] Internal error: Oops: 17 [#1] SMP ARM
Jan 11 01:19:00 NetFrog kernel: [ 8360.000960] Modules linked in: fuse rfcomm cmac bnep hci_uart btbcm serdev bluetooth ecdh_generic brcmfmac brcmutil cfg80211 rfkill snd_bcm2835(C) snd_pcm snd_timer snd uio_pdrv_genirq fixed uio i2c_dev ip_tables x_tables ipv6
Jan 11 01:19:00 NetFrog kernel: [ 8360.001083] CPU: 2 PID: 802 Comm: wpa_supplicant Tainted: G         C      4.14.79-v7+ #1159
Jan 11 01:19:00 NetFrog kernel: [ 8360.001092] Hardware name: BCM2835
Jan 11 01:19:00 NetFrog kernel: [ 8360.001103] task: b6b03c00 task.stack: a79d4000
Jan 11 01:19:00 NetFrog kernel: [ 8360.001126] PC is at call_commit_handler.part.2+0x18/0x44
Jan 11 01:19:00 NetFrog kernel: [ 8360.001140] LR is at ioctl_standard_call+0x270/0x4bc
Jan 11 01:19:00 NetFrog kernel: [ 8360.001151] pc : [<8078324c>]    lr : [<807834e8>]    psr: 20000013
Jan 11 01:19:00 NetFrog kernel: [ 8360.001162] sp : a79d5de0  ip : a79d5df0  fp : a79d5dec
Jan 11 01:19:00 NetFrog kernel: [ 8360.001173] r10: b6853000  r9 : 00000034  r8 : 00008b1a
Jan 11 01:19:00 NetFrog kernel: [ 8360.001184] r7 : 0000000e  r6 : a79d5ea0  r5 : 00000007  r4 : 00000002
Jan 11 01:19:00 NetFrog kernel: [ 8360.001196] r3 : 00000000  r2 : 60000013  r1 : 60000013  r0 : b6853000
Jan 11 01:19:00 NetFrog kernel: [ 8360.001209] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Jan 11 01:19:00 NetFrog kernel: [ 8360.001220] Control: 10c5383d  Table: 2795006a  DAC: 00000055
Jan 11 01:19:00 NetFrog kernel: [ 8360.001232] Process wpa_supplicant (pid: 802, stack limit = 0xa79d4210)
Jan 11 01:19:00 NetFrog kernel: [ 8360.001243] Stack: (0xa79d5de0 to 0xa79d6000)
Jan 11 01:19:00 NetFrog kernel: [ 8360.001261] 5de0: a79d5e5c a79d5df0 807834e8 80783240 806a8be0 a79d5e6c a79d5e6c a79d5eb0
Jan 11 01:19:00 NetFrog kernel: [ 8360.001278] 5e00: 00000000 a783f180 8069591c 636a8bb8 616e6968 6566696c 6669772d 80689569
Jan 11 01:19:00 NetFrog kernel: [ 8360.001295] 5e20: a79d5e44 807a1890 a79d5e4c a79d5e38 807a1890 00008b1a a79d5ea0 80c7a7c0
Jan 11 01:19:00 NetFrog kernel: [ 8360.001312] 5e40: 7e9c77b8 00000000 a79d4000 00000036 a79d5e94 a79d5e60 80783a2c 80783284
Jan 11 01:19:00 NetFrog kernel: [ 8360.001328] 5e60: 7f284c5c 80c7a7c0 7e9c77b8 00008b1a 00000004 00008b1a 80c7a7c0 7e9c77b8
Jan 11 01:19:00 NetFrog kernel: [ 8360.001345] 5e80: 80c7a7c0 00000004 a79d5eec a79d5e98 806b32e8 80783880 a79d5f14 a79d5ea8
Jan 11 01:19:00 NetFrog kernel: [ 8360.001362] 5ea0: 6e616c77 00000030 00000000 00000000 7e9c77d8 0001000e 00000000 00000000
Jan 11 01:19:00 NetFrog kernel: [ 8360.001379] 5ec0: a79d5f14 00008b1a 0000012a 7e9c77b8 80c7a7c0 00000004 a79d4000 00000036
Jan 11 01:19:00 NetFrog kernel: [ 8360.001396] 5ee0: a79d5f0c a79d5ef0 8066ecec 806b3258 7e9c77b8 b8b02e20 b9068c00 00008b1a
Jan 11 01:19:00 NetFrog kernel: [ 8360.001413] 5f00: a79d5f7c a79d5f10 8029ea5c 8066ebd4 8079ef74 80145ec0 00000128 801081c4
Jan 11 01:19:00 NetFrog kernel: [ 8360.001430] 5f20: a79d4000 00000128 a79d5f94 a79d5f38 808024bc 397e4000 8079f5d8 ba372d40
Jan 11 01:19:00 NetFrog kernel: [ 8360.001446] 5f40: 00000000 00000000 a79d5e88 a79d4000 a79d4010 b9068c00 7e9c77b8 b9068c00
Jan 11 01:19:00 NetFrog kernel: [ 8360.001463] 5f60: 00008b1a 00000004 a79d4000 00000036 a79d5fa4 a79d5f80 8029f1b8 8029e9bc
Jan 11 01:19:00 NetFrog kernel: [ 8360.001480] 5f80: 00000016 0000000e 020fe3c0 7e9c77d8 00000036 801081c4 00000000 a79d5fa8
Jan 11 01:19:00 NetFrog kernel: [ 8360.001497] 5fa0: 80108000 8029f180 0000000e 020fe3c0 00000004 00008b1a 7e9c77b8 00000016
Jan 11 01:19:00 NetFrog kernel: [ 8360.001513] 5fc0: 0000000e 020fe3c0 7e9c77d8 00000036 0211cf4c 00000000 001bc730 020fe3d0
Jan 11 01:19:00 NetFrog kernel: [ 8360.001530] 5fe0: 001bcbac 7e9c77b4 00141478 76c6880c 20000010 00000004 00000000 00000000
Jan 11 01:19:00 NetFrog kernel: [ 8360.001559] [<8078324c>] (call_commit_handler.part.2) from [<807834e8>] (ioctl_standard_call+0x270/0x4bc)
Jan 11 01:19:00 NetFrog kernel: [ 8360.001582] [<807834e8>] (ioctl_standard_call) from [<80783a2c>] (wext_handle_ioctl+0x1b8/0x20c)
Jan 11 01:19:00 NetFrog kernel: [ 8360.001605] [<80783a2c>] (wext_handle_ioctl) from [<806b32e8>] (dev_ioctl+0x9c/0x8e8)
Jan 11 01:19:00 NetFrog kernel: [ 8360.001630] [<806b32e8>] (dev_ioctl) from [<8066ecec>] (sock_ioctl+0x124/0x2d8)
Jan 11 01:19:00 NetFrog kernel: [ 8360.001653] [<8066ecec>] (sock_ioctl) from [<8029ea5c>] (do_vfs_ioctl+0xac/0x7c4)
Jan 11 01:19:00 NetFrog kernel: [ 8360.001674] [<8029ea5c>] (do_vfs_ioctl) from [<8029f1b8>] (SyS_ioctl+0x44/0x6c)
Jan 11 01:19:00 NetFrog kernel: [ 8360.001696] [<8029f1b8>] (SyS_ioctl) from [<80108000>] (ret_fast_syscall+0x0/0x28)
Jan 11 01:19:00 NetFrog kernel: [ 8360.001714] Code: e24cb004 e52de004 e8bd4000 e5903120 (e5933000) 
Jan 11 01:19:00 NetFrog kernel: [ 8360.001729] ---[ end trace a931f3365b4fc949 ]---

@pelwell
Copy link
Contributor

pelwell commented Jan 11, 2019

You could try with a script like this - I called it wpawatch:

#!/usr/bin/perl
my $ph;
die "* failed to open pipe\n" if (!open($ph, '-|', "dmesg -w"));
while (my $line = <$ph>)
{
        system("reboot") if ($line =~ /wpa_supplicant/);
}

Run with sudo ./wpawatch &, or from /etc/rc.local, etc.

Any appearance of the string wpa_supplicant in the kernel log will trigger a reboot. You can test it with:

$ sudo sh -c "echo Fake wpa_supplicant crash >/dev/kmsg"

I did try to use a simple shell script, but the line buffering wasn't co-operating, hence the Perl - sorry (not sorry).

@pelwell pelwell added the Waiting for external input Waiting for a comment from the originator of the issue, or a collaborator. label Jan 17, 2019
@cxueqin
Copy link
Author

cxueqin commented Jan 23, 2019

@pelwell Thank you for your perl watchdog script. It works. I will add the watch dog daemon to restart the system when wpa_supplicant and kernel bug occured. But anyway, any idea about the kernel bug? I think the best way is to fix the kernel bug directly.

@pelwell
Copy link
Contributor

pelwell commented Jan 23, 2019

We can't fix the kernel bug until we can reproduce it, and so far that hasn't been possible. Your kernel is not a standard kernel, your environment is different, and your use case (repeatedly connecting and disconnecting) is very niche.

@JamesH65
Copy link
Contributor

@cxueqin Have you tried this with the latest kernel? Does that make any difference?

This issue will be closed within 30 days unless further interactions are posted. If you wish this issue to remain open, please add a comment. A closed issue may be reopened if requested.

@JamesH65 JamesH65 added the Close within 30 days Issue will be closed within 30 days unless requested to stay open label Sep 19, 2019
@cxueqin
Copy link
Author

cxueqin commented Sep 21, 2019

This issue will be closed within 30 days unless further interactions are posted. If you wish this issue to remain open, please add a comment. A closed issue may be reopened if requested.

@JamesH65 I haven't test in the lastest kernel. Can you tell me which kernel version I should use for testing? Thanks

@JamesH65
Copy link
Contributor

Just the latest release one, from apt.

sudo apt update
sudo apt upgrade

@chrispollitt
Copy link

I am also having this issue.

@spaetow
Copy link

spaetow commented Feb 15, 2021

I have been able to reproduce this today. As with the OP, this effectively causes the RPi to stop responding to reboots (well, it shuts down but then hangs and a hard reset is required).

Feb 15 22:19:54 raspberrypi kernel: [   67.564603] pgd = 39a51f6e
Feb 15 22:19:54 raspberrypi kernel: [   67.564622] [00000000] *pgd=06eba835, *pte=00000000, *ppte=00000000
Feb 15 22:19:54 raspberrypi kernel: [   67.564679] Internal error: Oops: 17 [#1] SMP ARM
Feb 15 22:19:54 raspberrypi kernel: [   67.564692] Modules linked in: rfcomm fuse bnep hci_uart btbcm bluetooth ecdh_generic ecc 8021q garp stp llc nft_counter nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink joydev brcmfmac brcmutil sha256_generic cfg80211 rfkill raspberrypi_hwmon bcm2835_codec(C) bcm2835_v4l2(C) snd_bcm2835(C) bcm2835_isp(C) v4l2_mem2mem bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_pcm videobuf2_common snd_timer videodev snd vc_sm_cma(C) mc uio_pdrv_genirq uio fixed i2c_dev ip_tables x_tables ipv6
Feb 15 22:19:54 raspberrypi kernel: [   67.565022] CPU: 2 PID: 503 Comm: hostapd Tainted: G         C        5.10.11-v7+ #1399
Feb 15 22:19:54 raspberrypi kernel: [   67.565033] Hardware name: BCM2835
Feb 15 22:19:54 raspberrypi kernel: [   67.565209] PC is at brcmf_p2p_send_action_frame+0x288/0xa60 [brcmfmac]
Feb 15 22:19:54 raspberrypi kernel: [   67.565368] LR is at brcmf_p2p_send_action_frame+0x270/0xa60 [brcmfmac]
Feb 15 22:19:54 raspberrypi kernel: [   67.565382] pc : [<7f470f34>]    lr : [<7f470f1c>]    psr: 80000013
Feb 15 22:19:54 raspberrypi kernel: [   67.565394] sp : 85bf9a88  ip : 00000000  fp : 85bf9aec
Feb 15 22:19:54 raspberrypi kernel: [   67.565406] r10: 85ae04e0  r9 : 00000004  r8 : 7f48b6a8
Feb 15 22:19:54 raspberrypi kernel: [   67.565419] r7 : 00000000  r6 : 8401500c  r5 : 858a1000  r4 : 84015000
Feb 15 22:19:54 raspberrypi kernel: [   67.565432] r3 : 00000000  r2 : 858a1000  r1 : 7f492504  r0 : 00000000
Feb 15 22:19:54 raspberrypi kernel: [   67.565447] Flags: Nzcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Feb 15 22:19:54 raspberrypi kernel: [   67.565461] Control: 10c5383d  Table: 05bfc06a  DAC: 00000055
Feb 15 22:19:54 raspberrypi kernel: [   67.565476] Process hostapd (pid: 503, stack limit = 0x2963c431)
Feb 15 22:19:54 raspberrypi kernel: [   67.565489] Stack: (0x85bf9a88 to 0x85bfa000)
Feb 15 22:19:54 raspberrypi kernel: [   67.565507] 9a80:                   00000000 00000000 80e057c8 85ae04e0 84014d80 00000000
Feb 15 22:19:54 raspberrypi kernel: [   67.565527] 9aa0: ffffffff 7f492538 7f492384 7f492504 7f491b48 ffffa52f 80e03d00 00000000
Feb 15 22:19:54 raspberrypi kernel: [   67.565546] 9ac0: 85bf9aec 80e05008 0000006d 858a1000 9303002c 85bf9b80 84015000 00000000
Feb 15 22:19:54 raspberrypi kernel: [   67.565564] 9ae0: 85bf9b44 85bf9af0 7f466980 7f470cb8 00000000 00000000 0000006d 0000099e
Feb 15 22:19:54 raspberrypi kernel: [   67.565583] 9b00: 7f324110 85ae04e0 86d00004 00000085 0000099e a1908b93 808d4454 7f4667fc
Feb 15 22:19:54 raspberrypi kernel: [   67.565602] 9b20: 85ae01c0 85ae0000 85bf9bac 85bf9b80 83a8db40 84e9f014 85bf9b74 85bf9b48
Feb 15 22:19:54 raspberrypi kernel: [   67.565622] 9b40: 7f2eb5a0 7f466808 808d9784 808d4424 85bf9c54 80e05008 85ae0000 86d00004
Feb 15 22:19:54 raspberrypi kernel: [   67.565640] 9b60: 00000000 83a8db40 85bf9bfc 85bf9b78 7f2e1c3c 7f2eb498 00000000 0000003b
Feb 15 22:19:54 raspberrypi kernel: [   67.565660] 9b80: 00000000 00000000 809e5c7c 80196f54 84019e30 00000000 0000099e 00000000
Feb 15 22:19:54 raspberrypi kernel: [   67.565680] 9ba0: 00000000 00000000 00000000 84019e30 00000001 00000000 9303002c 00000085
Feb 15 22:19:54 raspberrypi kernel: [   67.565699] 9bc0: 00000000 00000000 00000000 a1908b93 85bf9cb0 00000000 7f324110 80e05008
Feb 15 22:19:54 raspberrypi kernel: [   67.565718] 9be0: 858a5000 83a8dc00 80f16400 85bf9cb0 85bf9cac 85bf9c00 808db758 7f2e1a0c
Feb 15 22:19:54 raspberrypi kernel: [   67.565736] 9c00: 00000001 809eb3b4 00000000 85bf9c18 00000000 80e05008 00000000 00000000
Feb 15 22:19:54 raspberrypi kernel: [   67.565755] 9c20: 00000000 85bf9c24 85bf9c24 a1908b93 0000173f 8ca790c0 7f2e1a00 00000000
Feb 15 22:19:54 raspberrypi kernel: [   67.565774] 9c40: 00000000 00000000 7f31a3cc 00000129 03101c3b 602af30f 2f0001ec 93030000
Feb 15 22:19:54 raspberrypi kernel: [   67.565793] 9c60: 93030010 93030014 858a5000 80f16400 85ae0000 86d00004 85bf9cb0 a1908b93
Feb 15 22:19:54 raspberrypi kernel: [   67.565813] 9c80: 85bf9cbc 83a8dc00 80e05008 808db598 93030000 000000b4 00000000 80e05008
Feb 15 22:19:54 raspberrypi kernel: [   67.565832] 9ca0: 85bf9cfc 85bf9cb0 808d903c 808db5a4 00000000 00000000 00000000 00000000
Feb 15 22:19:54 raspberrypi kernel: [   67.565850] 9cc0: 00000000 00000000 00000000 00000000 00000000 a1908b93 85bf9cfc 80f1cf54
Feb 15 22:19:54 raspberrypi kernel: [   67.565869] 9ce0: 83a8dc00 000000b4 83a8dc00 84efe184 85bf9d14 85bf9d00 808d98c8 808d8f80
Feb 15 22:19:54 raspberrypi kernel: [   67.565888] 9d00: 816b8000 84efe000 85bf9d4c 85bf9d18 808d86d4 808d98a0 7fffffff a1908b93
Feb 15 22:19:54 raspberrypi kernel: [   67.565907] 9d20: 000001c0 00000008 85bf9f40 80e05008 83a8dc00 84efe000 000000b4 00000000
Feb 15 22:19:54 raspberrypi kernel: [   67.565928] 9d40: 85bf9dbc 85bf9d50 808d8998 808d8538 00000001 80608704 85bf9d8c 85bf9d68
Feb 15 22:19:54 raspberrypi kernel: [   67.565947] 9d60: 8060ceec 806086f8 00000000 00000000 85bf9e40 82cefec0 00000000 000001f7
Feb 15 22:19:54 raspberrypi kernel: [   67.565965] 9d80: 00000000 00000000 00000000 a1908b93 000000b4 85bf9f40 862c8780 00000000
Feb 15 22:19:54 raspberrypi kernel: [   67.565984] 9da0: 80e05008 862c8780 00000000 85bf9de4 85bf9dd4 85bf9dc0 80853b20 808d8790
Feb 15 22:19:54 raspberrypi kernel: [   67.566003] 9dc0: 85bf9f40 00000000 85bf9e34 85bf9dd8 80854168 80853ae8 00000000 80e05008
Feb 15 22:19:54 raspberrypi kernel: [   67.566022] 9de0: 85bf9e34 85bf9df0 80855c20 80612178 85bf9e40 85bf9f48 85bf9e3c 00000010
Feb 15 22:19:54 raspberrypi kernel: [   67.566041] 9e00: 7e8c5f64 a1908b93 00004000 80e05008 85bf9f40 00000000 862c8780 00000000
Feb 15 22:19:54 raspberrypi kernel: [   67.566059] 9e20: 00000000 00000128 85bf9f2c 85bf9e38 80855cbc 80853f74 00000000 80612008
Feb 15 22:19:54 raspberrypi kernel: [   67.566080] 9e40: 00000000 01c49280 000000b4 85bf9ebc 85bf9ec0 80e05008 85bf9eac 8032dd0c
Feb 15 22:19:54 raspberrypi kernel: [   67.566099] 9e60: 80855c20 80612178 85bf9ebc 80855d6c 00000043 000001c0 00000000 859cfe40
Feb 15 22:19:54 raspberrypi kernel: [   67.566117] 9e80: 85bf9ec4 00000010 00000000 00000000 862caa80 a1908b93 00000000 859cfe40
Feb 15 22:19:54 raspberrypi kernel: [   67.566136] 9ea0: 000001c0 80e05008 00000000 85bf9f58 000001c0 00000000 85bf9f54 85bf9ec8
Feb 15 22:19:54 raspberrypi kernel: [   67.566155] 9ec0: 8032fb4c 8028c874 85bf9f44 85bf9ed8 8028c874 84efe000 862c8780 0000000b
Feb 15 22:19:54 raspberrypi kernel: [   67.566174] 9ee0: 84efe000 85bf9f08 00000004 00000000 80e05008 803527b4 85bf9f14 85bf9f08
Feb 15 22:19:54 raspberrypi kernel: [   67.566192] 9f00: 803527b4 a1908b93 80e05008 7e8c5f18 00000000 862c8780 80100204 85bf8000
Feb 15 22:19:54 raspberrypi kernel: [   67.566211] 9f20: 85bf9f94 85bf9f30 808560fc 80855c4c 00000000 00000000 00000000 fffffff7
Feb 15 22:19:54 raspberrypi kernel: [   67.566230] 9f40: 85bf9e84 0000000c 00000005 00000000 00000000 85bf9e4c 00000000 00000000
Feb 15 22:19:54 raspberrypi kernel: [   67.566248] 9f60: 00000000 7e8c5f91 00000000 00000000 00000000 a1908b93 01c44398 01c4afd8
Feb 15 22:19:54 raspberrypi kernel: [   67.566268] 9f80: 01c44320 00000128 85bf9fa4 85bf9f98 80856154 808560a8 00000000 85bf9fa8
Feb 15 22:19:54 raspberrypi kernel: [   67.566287] 9fa0: 80100040 80856144 01c44398 01c4afd8 00000005 7e8c5f18 00000000 00000000
Feb 15 22:19:54 raspberrypi kernel: [   67.566306] 9fc0: 01c44398 01c4afd8 01c44320 00000128 76ea2000 7e8c5fd8 00000001 00000004
Feb 15 22:19:54 raspberrypi kernel: [   67.566324] 9fe0: 0000006c 7e8c5ed0 76e875bc 76a4c980 60000010 00000005 00000000 00000000
Feb 15 22:19:54 raspberrypi kernel: [   67.566335] Backtrace:
Feb 15 22:19:54 raspberrypi kernel: [   67.566646] [<7f470cac>] (brcmf_p2p_send_action_frame [brcmfmac]) from [<7f466980>] (brcmf_cfg80211_mgmt_tx+0x184/0x2e4 [brcmfmac])
Feb 15 22:19:54 raspberrypi kernel: [   67.566670]  r10:00000000 r9:84015000 r8:85bf9b80 r7:9303002c r6:858a1000 r5:0000006d
Feb 15 22:19:54 raspberrypi kernel: [   67.566684]  r4:80e05008
Feb 15 22:19:54 raspberrypi kernel: [   67.567142] [<7f4667fc>] (brcmf_cfg80211_mgmt_tx [brcmfmac]) from [<7f2eb5a0>] (cfg80211_mlme_mgmt_tx+0x114/0x330 [cfg80211])
Feb 15 22:19:54 raspberrypi kernel: [   67.567164]  r10:84e9f014 r9:83a8db40 r8:85bf9b80 r7:85bf9bac r6:85ae0000 r5:85ae01c0
Feb 15 22:19:54 raspberrypi kernel: [   67.567177]  r4:7f4667fc
Feb 15 22:19:54 raspberrypi kernel: [   67.567650] [<7f2eb48c>] (cfg80211_mlme_mgmt_tx [cfg80211]) from [<7f2e1c3c>] (nl80211_tx_mgmt+0x23c/0x36c [cfg80211])
Feb 15 22:19:54 raspberrypi kernel: [   67.567672]  r9:83a8db40 r8:00000000 r7:86d00004 r6:85ae0000 r5:80e05008 r4:85bf9c54
Feb 15 22:19:54 raspberrypi kernel: [   67.567915] [<7f2e1a00>] (nl80211_tx_mgmt [cfg80211]) from [<808db758>] (genl_rcv_msg+0x1c0/0x370)
Feb 15 22:19:54 raspberrypi kernel: [   67.567937]  r10:85bf9cb0 r9:80f16400 r8:83a8dc00 r7:858a5000 r6:80e05008 r5:7f324110
Feb 15 22:19:54 raspberrypi kernel: [   67.567950]  r4:00000000
Feb 15 22:19:54 raspberrypi kernel: [   67.567971] [<808db598>] (genl_rcv_msg) from [<808d903c>] (netlink_rcv_skb+0xc8/0x120)
Feb 15 22:19:54 raspberrypi kernel: [   67.567991]  r10:80e05008 r9:00000000 r8:000000b4 r7:93030000 r6:808db598 r5:80e05008
Feb 15 22:19:54 raspberrypi kernel: [   67.568004]  r4:83a8dc00
Feb 15 22:19:54 raspberrypi kernel: [   67.568024] [<808d8f74>] (netlink_rcv_skb) from [<808d98c8>] (genl_rcv+0x34/0x44)
Feb 15 22:19:54 raspberrypi kernel: [   67.568042]  r8:84efe184 r7:83a8dc00 r6:000000b4 r5:83a8dc00 r4:80f1cf54
Feb 15 22:19:54 raspberrypi kernel: [   67.568062] [<808d9894>] (genl_rcv) from [<808d86d4>] (netlink_unicast+0x1a8/0x258)
Feb 15 22:19:54 raspberrypi kernel: [   67.568076]  r5:84efe000 r4:816b8000
Feb 15 22:19:54 raspberrypi kernel: [   67.568095] [<808d852c>] (netlink_unicast) from [<808d8998>] (netlink_sendmsg+0x214/0x468)
Feb 15 22:19:54 raspberrypi kernel: [   67.568115]  r10:00000000 r9:000000b4 r8:84efe000 r7:83a8dc00 r6:80e05008 r5:85bf9f40
Feb 15 22:19:54 raspberrypi kernel: [   67.568127]  r4:00000008
Feb 15 22:19:54 raspberrypi kernel: [   67.568150] [<808d8784>] (netlink_sendmsg) from [<80853b20>] (sock_sendmsg+0x44/0x54)
Feb 15 22:19:54 raspberrypi kernel: [   67.568169]  r10:85bf9de4 r9:00000000 r8:862c8780 r7:80e05008 r6:00000000 r5:862c8780
Feb 15 22:19:54 raspberrypi kernel: [   67.568182]  r4:85bf9f40
Feb 15 22:19:54 raspberrypi kernel: [   67.568203] [<80853adc>] (sock_sendmsg) from [<80854168>] (____sys_sendmsg+0x200/0x22c)
Feb 15 22:19:54 raspberrypi kernel: [   67.568218]  r5:00000000 r4:85bf9f40
Feb 15 22:19:54 raspberrypi kernel: [   67.568239] [<80853f68>] (____sys_sendmsg) from [<80855cbc>] (___sys_sendmsg+0x7c/0xa8)
Feb 15 22:19:54 raspberrypi kernel: [   67.568258]  r10:00000128 r9:00000000 r8:00000000 r7:862c8780 r6:00000000 r5:85bf9f40
Feb 15 22:19:54 raspberrypi kernel: [   67.568270]  r4:80e05008
Feb 15 22:19:54 raspberrypi kernel: [   67.568291] [<80855c40>] (___sys_sendmsg) from [<808560fc>] (__sys_sendmsg+0x60/0x9c)
Feb 15 22:19:54 raspberrypi kernel: [   67.568310]  r9:85bf8000 r8:80100204 r7:862c8780 r6:00000000 r5:7e8c5f18 r4:80e05008
Feb 15 22:19:54 raspberrypi kernel: [   67.568330] [<8085609c>] (__sys_sendmsg) from [<80856154>] (sys_sendmsg+0x1c/0x20)
Feb 15 22:19:54 raspberrypi kernel: [   67.568348]  r7:00000128 r6:01c44320 r5:01c4afd8 r4:01c44398
Feb 15 22:19:54 raspberrypi kernel: [   67.568370] [<80856138>] (sys_sendmsg) from [<80100040>] (ret_fast_syscall+0x0/0x28)
Feb 15 22:19:54 raspberrypi kernel: [   67.568383] Exception stack(0x85bf9fa8 to 0x85bf9ff0)
Feb 15 22:19:54 raspberrypi kernel: [   67.568400] 9fa0:                   01c44398 01c4afd8 00000005 7e8c5f18 00000000 00000000
Feb 15 22:19:54 raspberrypi kernel: [   67.568419] 9fc0: 01c44398 01c4afd8 01c44320 00000128 76ea2000 7e8c5fd8 00000001 00000004
Feb 15 22:19:54 raspberrypi kernel: [   67.568436] 9fe0: 0000006c 7e8c5ed0 76e875bc 76a4c980
Feb 15 22:19:54 raspberrypi kernel: [   67.568454] Code: e51b1040 e3530002 05943034 1594302c (e5930000)
Feb 15 22:19:54 raspberrypi kernel: [   67.568476] ---[ end trace 8d5fba1f93e042f8 ]---

This is on Raspbian GNU/Linux 10 (buster). Kernel version is Linux raspberrypi 5.10.11-v7+ #1399 SMP Thu Jan 28 12:06:05 GMT 2021 armv7l GNU/Linux

I am attempting to use the RPi as a Passpoint/Hotspot 2.0 access point, which requires some configuration. It appears to have this moment whilst going through the ANQP phase of the association. Configuration file can be provided (it contains RADIUS secrets to a third-party server that are sensitive).

@spaetow
Copy link

spaetow commented Feb 15, 2021

As an addendum: When the line hs20=1 in my hostapd.conf is commented out, the problem no longer occurs. So it appears that the firmware as provided by Broadcom/Cypress does not support Hotspot 2.0/Passpoint, or at least freaks out substantially to cause a kernel hang. I'll have to check this on Fedora as a comparison.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Close within 30 days Issue will be closed within 30 days unless requested to stay open Waiting for external input Waiting for a comment from the originator of the issue, or a collaborator.
Projects
None yet
Development

No branches or pull requests

5 participants