Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

isr_rfcoreerrors while pinging between CC2538DKs #7020

Closed
Gr3yh0und opened this issue May 8, 2017 · 14 comments
Closed

isr_rfcoreerrors while pinging between CC2538DKs #7020

Gr3yh0und opened this issue May 8, 2017 · 14 comments
Assignees
@rfuentess
Labels
Platform: ARM Platform: This PR/issue effects ARM-based platforms Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors)

Comments

@Gr3yh0und
Copy link

Hey guys,
I'm currently trying to play around with the TinyDTLS package maintained by @rfuentess . However while trying to make it fly I often get the following errors on the node (dtls-echo):

RFCORE_ASSERT(len <= RFCORE_XREG_RXFIFOCNT) failed at line 91 in rfcore_read_fifo()!
RFCORE_SFR_RFERRF = 0x02

RFCORE_ASSERT(NOT(flags & RXUNDERF)) failed at line 40 in isr_rfcoreerr()!
RFCORE_SFR_RFERRF = 0x00

...

RFCORE_ASSERT(RFCORE_XREG_RXFIFOCNT > 0) failed at line 77 in rfcore_read_byte()!
RFCORE_SFR_RFERRF = 0x00

I'm using one CC2538DK with the gnrc_border_router image as a border router to access the second device.

Sometimes similar errors occur on the border router as well:

gnrc_uhcpc: uhcp_handle_prefix(): got same prefix again

RFCORE_ASSERT(RFCORE_XREG_RXFIFOCNT > 0) failed at line 77 in rfcore_read_byte()!
RFCORE_SFR_RFERRF = 0x02

RFCORE_ASSERT(NOT(flags & RXUNDERF)) failed at line 40 in isr_rfcoreerr()!

RFCORE_SFR_uhcp_client(): sending REQ...

The DTLS handshake doesn't happen in the end (my guess because of those errors). Any ideas?
@Gr3yh0und
Copy link
Author

It looks like this topic has already came up earlier on a couple of issues and PRs. Obviously the CC2538 still has some RF problems with possibly too fast received packets? When i do massive ICMP spam with 250B huge packets it occurs every ~30s.
gnrc_netdev: possibly lost interrupt.

I guess in my situation the three fast coming DTLS client packets (key ex, cipher, finished) are sometimes crashing it.

@rfuentess
Copy link
Contributor

I guess in my situation the three fast coming DTLS client packets (key ex, cipher, finished) are sometimes crashing it.

Hi @Gr3yh0und, maybe you can try ifconfig <interface-ID> -ack_req ?
This helped to improve the performance of TinyDTLS for my testbed.

@Gr3yh0und
Copy link
Author

Hey @rfuentess ,
Thanks, it helps a bit. I'm currently using the version 0.8.2 where I needed to change one line because my compiler complained about gettimeofday() as non-existent (, so I set it to 0). However I'm now getting closer and it stops at

Jan 01 00:00:14 WARN decryption failed
Jan 01 00:00:14 INFO decrypt_verify() failed

I'll look closer into that tomorrow, somehow the MAC check failed.

@rfuentess
Copy link
Contributor

Jan 01 00:00:14 WARN decryption failed
Jan 01 00:00:14 INFO decrypt_verify() failed

I fear that the master secret for temporary keys are wrongly done due the absence of gettimeofday(). I'm assuming you are not using Linux for compiling, right? (I didn't have this particular problem when was testing the package).

By the way, the DEFINE that you disabled was HAVE_SYS_TIME_H or HAVE_TIME_H?

@Gr3yh0und
Copy link
Author

Gr3yh0und commented May 9, 2017 •
edited
Loading

After doing some testing the ifconfig 7 -ack_req doesn't help much. I just need to use the normal ping6 with 64 Bytes every second and I can wait for the RF errors to occur.

I'm building the BOARD=cc2538dk on debian. As the gettimeofday function returned an implicit declaration on compilation for the board, I simply set tv.tv_sec = 0; Changing the *t to simply return 0 doesn't change much as well. I'm using PSK.

I just checked the logs and compared keys. You are right, the master_key is differing but the pre_master is correct.

@rfuentess
Copy link
Contributor

Just a quick notification,

I have been working on improving the performance of tinyDTLS for RIOT (also, upgrading it to 0.8.6). I'm still doing tests before updating the PR #6430. But, you can see some of the changes on my fork.

The issues of encryption were related to allocation of memory. For now, I have solved it, temporarily, by increasing the thread stacks for tinyDTLS. This is tested only for iotlab-m3 boards. Other boards are having issues with memory when peers are added to tinyDTLS.

Finally, I was recently with a Debian machine and the issues related to gettimeofday() were not present.

@Gr3yh0und
Copy link
Author

Gr3yh0und commented May 24, 2017 •
edited
Loading

Hi @rfuentess ,
Thanks for the reply. For the gettimeofday() issue, simply do a make BOARD=cc2538dk and it will fail.

I've seen your branch on tinydtls and I am already using it. I've implemented my CoAPs server based on YaCoAP here on your old example. As I am now trying to do the same for the client I've tried your old and new client implementation. The old implementation runs into memory problems as it overwrites (like after 3-4 times of execution) some areas in the stack. The newer one directly crashes with a Stack Pointer error after receiving the Server Hello/Done.

dtlsc fd00:dead:beef::1 1
Client support PSK
CLIENT: DTLS Channel started
DBG-Client: Msg received from
fd00:dead:beef::1 Addr Src: []:6666
Stack pointer corrupted, reset to top of stack
FSR/FAR:
CFSR: 0x00008600
HFSR: 0x40000000
DFSR: 0x00000000
AFSR: 0x00000000
BFAR: 0xffffffff
Misc
EXC_RET: 0xfffffff1`

Anyway keep up the good work!

@rfuentess rfuentess mentioned this issue Jun 12, 2017
Closed
6 tasks
@miri64 miri64 added the Platform: ARM Platform: This PR/issue effects ARM-based platforms label Oct 25, 2017
@miri64 miri64 added the Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors) label Oct 25, 2017
@MrKevinWeiss
Copy link
Contributor

Just so I don't reopen another issue I found the following related bug:

Description

During testing 2018.10-RC2 04-single-hop-6lowpan-icmp Task #06 (Experimental) - ICMPv6 link-local echo with samr21-xpro/remote
I had some assert fails on the remote-revb board:

2018-11-07 12:29:57,820 - INFO # RFCORE_ASSERT(RFCORE_XREG_RXFIFOCNT > 0) failed at line 77 in rfcore_read_byte()!
2018-11-07 12:29:57,822 - INFO #   RFCORE_SFR_RFERRF = 0x02
2018-11-07 12:29:57,825 - INFO # RFCORE_ASSERT(len <= RFCORE_XREG_RXFIFOCNT) failed at line 91 in rfcore_read_fifo()!
2018-11-07 12:29:57,826 - INFO #   RFCORE_SFR_RFERRF = 0x00
2018-11-07 12:29:57,841 - INFO # RFCORE_ASSERT(NOT(flags & RXUNDERF)) failed at line 40 in isr_rfcoreerr()!
2018-11-07 12:29:57,842 - INFO #   RFCORE_SFR_RFERRF = 0x00
...
2018-11-07 12:29:58,420 - INFO # RFCORE_ASSERT(NOT(flags & RXUNDERF)) failed at line 40 in isr_rfcoreerr()!
2018-11-07 12:29:58,421 - INFO #   RFCORE_SFR_RFERRF = 0x00
2018-11-07 12:29:58,422 - INFO # RFCORE_ASSERT(RFCORE_XREG_RXFIFOCNT > 0) failed at line 77 in rfcore_read_byte()!
2018-11-07 12:29:58,423 - INFO #   RFCORE_SFR_RFERRF = 0x00
2018-11-07 12:29:58,442 - INFO # RFCORE_ASSERT(NOT(flags & RXUNDERF)) failed at line 40 in isr_rfcoreerr()!
2018-11-07 12:29:58,443 - INFO #   RFCORE_SFR_RFERRF = 0x00
2018-11-07 12:29:58,444 - INFO # RFCORE_ASSERT(RFCORE_XREG_RXFIFOCNT > 0) failed at line 77 in rfcore_read_byte()!
2018-11-07 12:29:58,445 - INFO #   RFCORE_SFR_RFERRF = 0x00
...

Steps to reproduce the issue

BOARD=<samr21-xpro and remote-revb> make flash term -C tests/gnrc_udp samr21-xpro: ping6 1000 100 100at the same time remote-revb:ping6 1000 100 100`

Expected results

No RFCORE_ASSERT failures and packet loss < 10%

Actual results

many RFCORE_ASSERT failures (see above) with 5% packet loss (so still acceptable)

Versions

Operating System Environment
-----------------------------
       Operating System: "Ubuntu" "16.04.5 LTS (Xenial Xerus)"
                 Kernel: Linux 4.15.0-38-generic x86_64 x86_64

Installed compiler toolchains
-----------------------------
             native gcc: gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
      arm-none-eabi-gcc: arm-none-eabi-gcc (GNU Tools for Arm Embedded Processors 7-2017-q4-major) 7.2.1 20170904 (release) [ARM/embedded-7-branch revision 255204]
                avr-gcc: avr-gcc (GCC) 4.9.2
       mips-mti-elf-gcc: missing
             msp430-gcc: missing
   riscv-none-embed-gcc: missing
                  clang: clang version 5.0.0-3~16.04.1 (tags/RELEASE_500/final)

Installed compiler libs
-----------------------
   arm-none-eabi-newlib: "2.5.0"
    mips-mti-elf-newlib: missing
riscv-none-embed-newlib: missing
               avr-libc: "1.8.0svn" ("20111229")

Installed development tools
---------------------------
                  cmake: cmake version 3.12.3
               cppcheck: missing
                doxygen: missing
                 flake8: 3.5.0 (mccabe: 0.6.1, pycodestyle: 2.3.1, pyflakes: 1.6.0) CPython 3.5.2 on Linux
                    git: git version 2.7.4
                openocd: Open On-Chip Debugger 0.10.0+dev-00399-g09076d1 (2018-04-12-17:12)
                 python: Python 2.7.12
                python2: Python 2.7.12
                python3: Python 3.5.2
             coccinelle: missing

@MrKevinWeiss
Copy link
Contributor

@jia200x ping

@stale
Copy link

stale bot commented Aug 10, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.

@stale stale bot added the State: stale State: The issue / PR has no activity for >185 days label Aug 10, 2019
@MrKevinWeiss
Copy link
Contributor

Since it passed a few release cycles where this was tested can we say that it is closed? Does someone want to retest?

@stale stale bot removed the State: stale State: The issue / PR has no activity for >185 days label Aug 12, 2019
@Gr3yh0und
Copy link
Author

I sadly don't have access to these boards anymore... :-/

@MrKevinWeiss
Copy link
Contributor

Since I didn't discover this issue in the last release tests I am going to close it!

@MBradbury
Copy link

I've been trying to set up gnrc_border_router and several instances of gnrc_networking on Zolertia RE Mote Rev.Bs and have been encountering these assertions.

On all nodes I encounter asserts for the rssi_value, where rssi_value is usually -99 or -98 and CC2538_RF_SENSITIVITY is -97.

RFCORE_ASSERT(rssi_val > CC2538_RF_SENSITIVITY) failed at line 338 in _recv()!
  RFCORE_SFR_RFERRF = 0x00

However, on some of the remote-revbs running gnrc_networking I also encounter:

RFCORE_ASSERT(RFCORE_XREG_RXFIFOCNT > 0) failed at line 77 in rfcore_read_byte()!
  RFCORE_SFR_RFERRF = 0x00
RFCORE_ASSERT(NOT(flags & RXUNDERF)) failed at line 40 in isr_rfcoreerr()!
  RFCORE_SFR_RFERRF = 0x00
RFCORE_ASSERT(RFCORE_XREG_RXFIFOCNT > 0) failed at line 77 in rfcore_read_byte()!
  RFCORE_SFR_RFERRF = 0x00
RFCORE_ASSERT(NOT(flags & RXUNDERF)) failed at line 40 in isr_rfcoreerr()!
  RFCORE_SFR_RFERRF = 0x00

I believe this is causing issues where I cannot ping certain nodes. For example on the RPL root node:

ifconfig
Iface  7  HWaddr: 2F:0A  Channel: 26  Page: 0  NID: 0x23
          Long HWaddr: 00:12:4B:00:14:D5:2F:0A
           TX-Power: 0dBm  State: IDLE
          AUTOACK  L2-PDU:102 MTU:1280  HL:64  RTR
          RTR_ADV  6LO  IPHC
          Source address length: 8
          Link type: wireless
          inet6 addr: fe80::212:4b00:14d5:2f0a  scope: link  VAL
          inet6 addr: 2001:db8::212:4b00:14d5:2f0a  scope: global  VAL
          inet6 group: ff02::2
          inet6 group: ff02::1
          inet6 group: ff02::1:ffd5:2f0a
          inet6 group: ff02::1a

Iface  6  HWaddr: 02:DD:CB:72:02:3A
          L2-PDU:1500 MTU:1500  HL:64  RTR
          Source address length: 6
          Link type: wired
          inet6 addr: fe80::dd:cbff:fe72:23a  scope: link  VAL
          inet6 addr: fe80::2  scope: link  VAL
          inet6 group: ff02::2
          inet6 group: ff02::1
          inet6 group: ff02::1:ff72:23a
          inet6 group: ff02::1:ff00:2

> rpl
rpl
instance table: [X]
parent table:   [ ]     [ ]     [ ]

instance [0 | Iface: 7 | mop: 2 | ocp: 0 | mhri: 256 | mri 0]
        dodag [2001:db8::212:4b00:14d5:2f0a | R: 256 | OP: Router | PIO: on | TR(I=[8,20], k=10, c=16, TC=64s)]
> ping6 2001:db8::212:4b00:14d5:2ba9

--- 2001:db8::212:4b00:14d5:2ba9 PING statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

the target:

Iface  7  HWaddr: 2B:A9  Channel: 26  Page: 0  NID: 0x23
          Long HWaddr: 00:12:4B:00:14:D5:2B:A9
           TX-Power: 0dBm  State: IDLE
          AUTOACK  ACK_REQ  L2-PDU:102 MTU:1280  HL:64  RTR
          6LO  IPHC
          Source address length: 8
          Link type: wireless
          inet6 addr: fe80::212:4b00:14d5:2ba9  scope: link  VAL
          inet6 addr: 2001:db8::212:4b00:14d5:2ba9  scope: global  VAL
          inet6 group: ff02::2
          inet6 group: ff02::1
          inet6 group: ff02::1:ffd5:2ba9
          inet6 group: ff02::1a

          Statistics for Layer 2
            RX packets 388  bytes 25237
            TX packets 217 (Multicast: 83)  bytes 14607
            TX succeeded 0 errors 0
          Statistics for IPv6
            RX packets 325  bytes 25482
            TX packets 217 (Multicast: 83)  bytes 18302
            TX succeeded 217 errors 0
> rpl
instance table: [X]
parent table:   [X]     [ ]     [ ]

instance [0 | Iface: 7 | mop: 2 | ocp: 0 | mhri: 256 | mri 0]
        dodag [2001:db8::212:4b00:14d5:2f0a | R: 512 | OP: Router | PIO: on | TR(I=[8,20], k=10, c=1, TC=35s)]
                parent [addr: fe80::212:4b00:14d5:2f0a | rank: 256]

However, another node can ping this target:

> ifconfig
Iface  7  HWaddr: 2B:E6  Channel: 26  Page: 0  NID: 0x23
          Long HWaddr: 00:12:4B:00:14:D5:2B:E6
           TX-Power: 0dBm  State: IDLE
          AUTOACK  L2-PDU:102 MTU:1280  HL:64  RTR
          6LO  IPHC
          Source address length: 8
          Link type: wireless
          inet6 addr: fe80::212:4b00:14d5:2be6  scope: link  VAL
          inet6 addr: 2001:db8::212:4b00:14d5:2be6  scope: global  VAL
          inet6 group: ff02::2
          inet6 group: ff02::1
          inet6 group: ff02::1:ffd5:2be6
          inet6 group: ff02::1a

          Statistics for Layer 2
            RX packets 202  bytes 11685
            TX packets 132 (Multicast: 87)  bytes 8239
            TX succeeded 0 errors 0
          Statistics for IPv6
            RX packets 202  bytes 15540
            TX packets 132 (Multicast: 87)  bytes 10594
            TX succeeded 132 errors 0

rpl
> rpl
instance table: [X]
parent table:   [X]     [ ]     [ ]

instance [0 | Iface: 7 | mop: 2 | ocp: 0 | mhri: 256 | mri 0]
        dodag [2001:db8::212:4b00:14d5:2f0a | R: 768 | OP: Router | PIO: on | TR(I=[8,20], k=10, c=1, TC=483s)]
                parent [addr: fe80::212:4b00:14d5:2ba9 | rank: 512]
ping6 2001:db8::212:4b00:14d5:2ba9
> ping6 2001:db8::212:4b00:14d5:2ba9
12 bytes from 2001:db8::212:4b00:14d5:2ba9: icmp_seq=0 ttl=64 time=7.313 ms
12 bytes from 2001:db8::212:4b00:14d5:2ba9: icmp_seq=1 ttl=64 time=7.313 ms
12 bytes from 2001:db8::212:4b00:14d5:2ba9: icmp_seq=2 ttl=64 time=7.314 ms

--- 2001:db8::212:4b00:14d5:2ba9 PING statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 7.313/7.313/7.314 ms

This is on master

version
2020.07-devel-59-g7aa62

Other environment information:

Operating System Environment
----------------------------
         Operating System: "Debian GNU/Linux" "10 (buster)"
                   Kernel: Linux 4.19.0-8-amd64 x86_64 unknown
             System shell: /usr/bin/dash (probably dash)
             make's shell: /usr/bin/dash (probably dash)

Installed compiler toolchains
-----------------------------
               native gcc: gcc (Debian 8.3.0-6) 8.3.0
        arm-none-eabi-gcc: arm-none-eabi-gcc (15:7-2018-q2-6) 7.3.1 20180622 (release) [ARM/embedded-7-branch revision 261907]
                  avr-gcc: avr-gcc (AVR_8_bit_GNU_Toolchain_3.6.2_1759) 5.4.0
         mips-mti-elf-gcc: missing
               msp430-gcc: msp430-gcc (GCC) 4.7.3 20130411 (mspgcc dev 20120911)
     riscv-none-embed-gcc: missing
     xtensa-esp32-elf-gcc: missing
   xtensa-esp8266-elf-gcc: missing
                    clang: missing

Installed compiler libs
-----------------------
     arm-none-eabi-newlib: "3.1.0"
      mips-mti-elf-newlib: missing
  riscv-none-embed-newlib: missing
  xtensa-esp32-elf-newlib: missing
xtensa-esp8266-elf-newlib: missing
                 avr-libc: "2.0.0" ("20150208")

Installed development tools
---------------------------
                   ccache: missing
                    cmake: missing
                 cppcheck: missing
                  doxygen: missing
                      git: git version 2.20.1
                     make: GNU Make 4.2.1
                  openocd: missing
                   python: Python 3.7.5
                  python2: Python 2.7.16
                  python3: Python 3.7.5
                   flake8: 3.7.9 (mccabe: 0.6.1, pycodestyle: 2.5.0, pyflakes: 2.1.1) CPython 3.7.5 on Linux
               coccinelle: missing

@MBradbury MBradbury mentioned this issue Apr 20, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rfuentess rfuentess

Labels
Platform: ARM Platform: This PR/issue effects ARM-based platforms Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@miri64 @Gr3yh0und @MBradbury @rfuentess @MrKevinWeiss