rt2x00 - usb system still broken

[username7291]

Describe the bug
Wifi crashes without a Hardware reason (3 different Wifi sticks, about 15 different cables 0.2M-3M, about 6 PSUs actual a 5 Amp with a short thick cable, direct USB power, a Rasberry 2 and a Raspberry 3, different SD cards, many different OS/firmware). Power is stable on the Wifi stick, no drops in dmesg, power is uninterruptable.

To reproduce
Use a wifi stick, with other traffic intensive Usb/Network load. Network will go down after 1 Hour-3 Month. In this case 2 720p cameras running via usb, 2 streams running over the Network, airmon-ng is running. Happends more in the cold month cause it is darker the Camera jpegs increase in size and the taffic goes up. The CPU temprature is low 52°C max.

Expected behaviour
Wireless Network shouldnt go down.

Actual behaviour
Wireless Network goes down

System

• Which model of Raspberry Pi? e.g. Pi3B+, PiZeroW: PI2 and PI3

• Which OS and version (cat /etc/rpi-issue)?
  Raspberry Pi reference 2017-07-05
  Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, dd7de2554c08e6db89806eb45d955356d0456c66, stage5
  (happends with any OS working with RPI2 and above, never happends with RPIB and wheezy 24/7 but this had only one camera)

• Which firmware version (vcgencmd version)?
  Nov 29 2019 18:46:15
  Copyright (c) 2012 Broadcom
  version 7d5bdbd77968098c981dadefae06aaec79354c69 (clean) (release) (start)

• Which kernel version (uname -a)?
  Linux rpi3 4.19.86-v7+ Can't enable serial5 and serial enumeration is weird  #1283 SMP Fri Nov 29 18:32:25 GMT 2019 armv7l GNU/Linux
  (happends with many different tryed versions rpiupdate <>)

Logs
If applicable, add the relevant output from dmesg or similar.
[Die Dez 3 17:56:52 2019] ieee80211 phy0: rt2x00lib_request_firmware: Info - Loading firmware file 'rt2870.bin'
[Die Dez 3 17:56:52 2019] ieee80211 phy0: rt2x00lib_request_firmware: Info - Firmware detected - version: 0.29
[Die Dez 3 17:56:53 2019] device wlan0 entered promiscuous mode
[Die Dez 3 17:56:56 2019] tun: Universal TUN/TAP device driver, 1.6
[Mit Dez 4 09:29:14 2019] sh (6349): drop_caches: 1
[Don Dez 5 09:29:42 2019] sh (12234): drop_caches: 1
[Don Dez 5 15:25:14 2019] ------------[ cut here ]------------
[Don Dez 5 15:25:14 2019] WARNING: CPU: 0 PID: 13126 at drivers/usb/core/urb.c:363 usb_submit_urb+0x2e0/0x4f4
[Don Dez 5 15:25:14 2019] URB 5dccb931 submitted while active
[Don Dez 5 15:25:14 2019] Modules linked in: tun arc4 rt2800usb rt2800lib rt2x00usb rt2x00lib mac80211 sha256_generic cfg80211 rfkill crc_ccitt evdev raspberrypi_hwmon bcm2835_codec(C) spidev bcm2835_v4l2(C) hwmon v4l2_mem2mem bcm2835_mmal_vchiq(C) snd_bcm2835(C) i2c_bcm2835 vc_sm_cma(C) videobuf2_dma_contig v4l2_common spi_bcm2835 snd_usb_audio snd_hwdep snd_usbmidi_lib uvcvideo snd_rawmidi videobuf2_vmalloc videobuf2_memops snd_seq_device videobuf2_v4l2 videobuf2_common snd_pcm videodev snd_timer fixed snd uio_pdrv_genirq media uio i2c_dev fuse ipv6
[Don Dez 5 15:25:14 2019] CPU: 0 PID: 13126 Comm: kworker/u8:0 Tainted: G WC 4.19.86-v7+ #1283
[Don Dez 5 15:25:14 2019] Hardware name: BCM2835
[Don Dez 5 15:25:14 2019] Workqueue: phy0 rt2x00usb_work_rxdone [rt2x00usb]
[Don Dez 5 15:25:14 2019] [<801120c0>] (unwind_backtrace) from [<8010d5f4>] (show_stack+0x20/0x24)
[Don Dez 5 15:25:14 2019] [<8010d5f4>] (show_stack) from [<80844228>] (dump_stack+0xe0/0x124)
[Don Dez 5 15:25:14 2019] [<80844228>] (dump_stack) from [<80120cbc>] (__warn+0x104/0x11c)
[Don Dez 5 15:25:14 2019] [<80120cbc>] (__warn) from [<80120d2c>] (warn_slowpath_fmt+0x58/0x74)
[Don Dez 5 15:25:14 2019] [<80120d2c>] (warn_slowpath_fmt) from [<80648350>] (usb_submit_urb+0x2e0/0x4f4)
[Don Dez 5 15:25:14 2019] [<80648350>] (usb_submit_urb) from [<7f50b6c0>] (rt2x00usb_kick_rx_entry+0xc0/0x100 [rt2x00usb])
[Don Dez 5 15:25:14 2019] [<7f50b6c0>] (rt2x00usb_kick_rx_entry [rt2x00usb]) from [<7f50b730>] (rt2x00usb_clear_entry+0x30/0x34 [rt2x00usb])
[Don Dez 5 15:25:14 2019] [<7f50b730>] (rt2x00usb_clear_entry [rt2x00usb]) from [<7f4f6b94>] (rt2x00lib_rxdone+0x1a4/0x608 [rt2x00lib])
[Don Dez 5 15:25:14 2019] [<7f4f6b94>] (rt2x00lib_rxdone [rt2x00lib]) from [<7f50c064>] (rt2x00usb_work_rxdone+0x70/0xb0 [rt2x00usb])
[Don Dez 5 15:25:14 2019] [<7f50c064>] (rt2x00usb_work_rxdone [rt2x00usb]) from [<8013c46c>] (process_one_work+0x170/0x458)
[Don Dez 5 15:25:14 2019] [<8013c46c>] (process_one_work) from [<8013c7b0>] (worker_thread+0x5c/0x5a4)
[Don Dez 5 15:25:14 2019] [<8013c7b0>] (worker_thread) from [<80142a9c>] (kthread+0x138/0x168)
[Don Dez 5 15:25:14 2019] [<80142a9c>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
[Don Dez 5 15:25:14 2019] Exception stack(0xb4615fb0 to 0xb4615ff8)
[Don Dez 5 15:25:14 2019] 5fa0: 00000000 00000000 00000000 00000000
[Don Dez 5 15:25:14 2019] 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[Don Dez 5 15:25:14 2019] 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[Don Dez 5 15:25:14 2019] ---[ end trace ecea6ba1c3ce15af ]---
[Don Dez 5 15:32:15 2019] ieee80211 phy0: rt2x00queue_flush_queue: Warning - Queue 14 failed to flush

(this sometimes is only "Queue failed to flush" without the trace but wifi goes down too)
Additional context

Bus 001 Device 007: ID 148f:3070 Ralink Technology, Corp. RT2870/RT3070 Wireless Adapter
Bus 001 Device 005: ID 045e:0779 Microsoft Corp. LifeCam HD-3000
Bus 001 Device 004: ID 045e:0779 Microsoft Corp. LifeCam HD-3000
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Add any other relevant context for the problem.
Most times wifi works again after "ifconfig wlan0 down" and "ifconfig wlan up" again,
but there are cases that lxterminals freezes completely and it require a full reboot.
This never happends on RPIB, cause the trace points to "drivers/usb/core/urb.c" i assume its
the new USB driver and it has still Bugs.
The same stuff runs on x86 Hardware and Linux in various kernel/OS versions 24/7 since years, so
this Bug is rasberry/raspbian realated and not the fault of external developers.
In German forums different peoples complain about wifi issues.

[JamesH65]

Do you have any specific way of making it go wrong? If it takes days to go wrong this is going to be extremely difficult to track down. Does the Pi4 show the same issue?

[username7291]

It always goes wrong, just a matter of time. Webcam&Traffic is necessary, without camera and only 1/3 of the traffic i dont got it to crash within 3 Month.
Shouldnt tell the trace together with the known code where the bug is located? I would guess increasing the usb load would trigger the bug faster, means if you use 3 usb cameras/1080p, just getting usb to the maximum rate that the pi can reach. I dont have a Pi4, but its possible that it doesnt have this(differend usb driver) or its more difficult to trigger the bug. The Pi4 has no Ethernet over USB and USB3, means you cant reach the total throughput limit, for PI2/3 at about 20-30 MB/s, so fast.

General the failure handling of usb pretty goes wrong. Wifi goes down, no packets are received, the device is still listed under lsusb and usual can resumed by end user with ifconfig wlan0 down/up. Why the driver layer dont resume on ANY unknow error? Other operating systems would simply powercycle/reinit a "dead" usb device automatic.

[ganzgustav22]

I found that the latest kernel that works reliably with Ralink and Atheros Wifi USB cards is the 4.9 line. Everything higher and there's all kinds of weird problems.

[username7291]

PI4 shows the same issue, right now i would say less often.
By the way the hanging read() is set to a timeout of 1 sec
by select(). Who is responsible for the timeout rt2x00?