Multiple instance of mavlink crashes when arming

[x-logan]

I can run two mavlink instances without problems but the moment i arm the system both instances stop responding.

I initialize them in the /etc/extras.txt like this

mavlink stop-all
sleep 2
mavlink start -d /dev/ttyS1 -b 57600
sleep 2
mavlink start -d /dev/ttyS2 -b 115200

The radio is connected to /dev/ttyS1 and the offboard computer is connected to /dev/ttyS2.
Both work fine until y arm the system.

Lorenz said that i should remove all the instructions on the extras.txt file and change the parameter SYS_COMPANION to 921600, and i did but I realice that this parameter only affects the FMUV2 and i am using a PX4+IO.

So i manually start the mavlink like this in the extas.txt

mavlink start -d /dev/ttyS2 -b 921600 -m onboard

and the problem still remains..

[SimonWilks]

It looks like it is a memory usage issue. I tested this out today and it looks like it is only just using too much memory when both mavlinks are started. It worked for me twice but failed on the remaining few attempts.

@x-logan can you check the amount of space left by issuing the "free" command? For me it was around only 2k after the crash.

Either we need to reduce the amount of memory used by mavlink ( @LorenzMeier can this be done by reducing the number of streams for example?) or, as I am doing with ROS, channel QGC connection over the same connection as the offboard.

[x-logan]

@SimonWilks I did what you said and this is the result, the output of the free command before and after arming the system
__________total_____used____free______largest
Mem:____173776___168528___5248_____1168
and after arming
__________total_____used____free______largest
Mem:____173776___91248___21056______5600

Both instances of mavlink are still running but not answering, i can't stop them with stop-all

I also reduce the number of the streams that mavlink sends, i just leave the SYS_STATUS and VFR_HUD, and it didn't crash. Here are the output of the free command
__________total_____used____free______largest
Mem:_____173776___165312__8464_____3600
after arming
__________total_____used____free______largest
Mem:____173776___169584___4192_____1536

So it is a memory problem i don't know if this can be solved or we just have to be careful with the amount of streams we send with mavlink.

[SimonWilks]

Can you just use a single mavlink instance and proxy qgc over it? I've
never used it but mavproxy might help here.

On Fri, Feb 20, 2015 at 3:28 PM, Logan [email protected] wrote:

@SimonWilks https://github.com/SimonWilks I did what you said and this
is the result, the output of the free command before and after arming the
system

           total          used           free      largest

Mem: 173776 168528 5248 1168
and after arming

           total          used           free       largest

Mem: 173776 91248 21056 5600

Both instances of mavlink are still running but not answering, i can't
stop them with stop-all

I also reduce the number of the streams that mavlink sends, i just leave
the SYS_STATUS and VFR_HUD, and it didn't crash. Here are the output of the
free command

            total         used          free        largest

Mem: 173776 165312 8464 3600
after arming
total used free largest
Mem: 173776 169584 4192 1536

So it is a memory problem i don't know if this can be solved or we just
have to be careful with the amount of streams we send with mavlink.

—
Reply to this email directly or view it on GitHub
#1812 (comment).

[x-logan]

I'll try mavproxy but i don't know if that is going to suit my needs because in my offboard program i already have the mavlink's headers and already made an interface to communicate to the PX4, so i don't know if mavproxy can share the serial port but i don't think so.

[LorenzMeier]

Current RAM usage on FMUv1:

 PID COMMAND                   CPU(ms) CPU(%)  USED/STACK PRIO(BASE) STATE
   0 Idle Task                  916438 69.298     0/    0   0 (  0)  READY
   1 hpwork                      27555  2.144   692/ 1792 192 (192)  w:sem
   2 lpwork                       3448  0.194   364/ 1792  50 ( 50)  READY
   3 init                          172  0.000  1444/ 2992 100 (100)  w:sem
   6 nshterm                         5  0.000   844/ 1496  70 ( 70)  w:sem
 132 top                           849  0.877  1172/ 1696 100 (100)  RUN
  75 dataman                        15  0.000   612/ 1792 100 (100)  w:sem
  77 commander                    1922  0.097  2692/ 3192 215 (215)  w:sig
  78 commander_low_prio            182  0.000   476/ 2392  50 ( 50)  w:sem
  80 mavlink_if0                  8433  0.584  1780/ 2392 100 (100)  w:sig
  81 mavlink_rcv_if0                84  0.000   780/ 2096 175 (175)  w:sem
 101 sensors_task                26784  2.144  1148/ 1992 250 (250)  w:sem
 106 sdlog2                       1463  0.097  1972/ 2992  70 ( 70)  w:sig
 111 gps                          5889  1.169   692/ 1496 220 (220)  w:sem
 119 ekf_att_pos_estimator      265363 20.760  3708/ 7496 215 (215)  w:sem
 121 mc_att_control              12111  0.877   996/ 1792 250 (250)  w:sem
 123 mc_pos_control               7906  0.584  1044/ 1792 250 (250)  w:sem
 125 land_detector                2033  0.097   524/  992 100 (100)  READY
 127 navigator                    6842  0.487   756/ 1792 120 (120)  w:sem

[LorenzMeier]

Fixed wing config:

Processes: 19 total, 4 running, 15 sleeping
CPU usage: 32.04% tasks, 0.58% sched, 67.38% idle
Uptime: 27.978s total, 19.291s idle

 PID COMMAND                   CPU(ms) CPU(%)  USED/STACK PRIO(BASE) STATE 
   0 Idle Task                   19290 67.380     0/    0   0 (  0)  READY 
   1 hpwork                        540  2.044   828/ 1792 192 (192)  w:sig 
   2 lpwork                         71  0.194   364/ 1792  50 ( 50)  READY 
   3 init                          166  0.000  1128/ 2992 100 (100)  w:sem 
   6 nshterm                         4  0.000   712/ 1496  70 ( 70)  w:sem 
 127 top                           157  0.876  1164/ 1696 100 (100)  RUN   
  75 dataman                        15  0.000   612/ 1792 100 (100)  w:sem 
  77 commander                      81  0.097  2692/ 3192 215 (215)  w:sig 
  78 commander_low_prio              3  0.000   468/ 2392  50 ( 50)  w:sem 
  80 mavlink_if0                   181  0.584  1764/ 2392 100 (100)  w:sig 
  81 mavlink_rcv_if0                 2  0.000   780/ 2096 175 (175)  w:sem 
 101 sensors_task                  513  1.947  1156/ 1992 250 (250)  w:sem 
 106 sdlog2                        280  0.000  1972/ 2992  70 ( 70)  w:sig 
 111 gps                           108  0.000   660/ 1496 220 (220)  w:sem 
 118 ekf_att_pos_estimator        6048 25.413  3708/ 7496 215 (215)  w:sem 
 120 fw_att_control                 77  0.292   884/ 2040 250 (250)  w:sem 
 122 fw_pos_control_l1              15  0.000   820/ 1992 250 (250)  w:sem 
 124 land_detector                  34  0.097   468/  992 100 (100)  READY 
 126 navigator                     141  0.486   756/ 1792 120 (120)  w:sem 
nsh> free
             total       used       free    largest
Mem:        181808     160832      20976      16464

[LorenzMeier]

Now we're on master at:

Processes: 19 total, 4 running, 15 sleeping
CPU usage: 33.59% tasks, 0.68% sched, 65.73% idle
Uptime: 37.480s total, 25.841s idle

 PID COMMAND                   CPU(ms) CPU(%)  USED/STACK PRIO(BASE) STATE 
   0 Idle Task                   25840 65.725     0/    0   0 (  0)  READY 
   1 hpwork                        738  2.044   684/ 1792 192 (192)  w:sig 
   2 lpwork                         97  0.194   364/ 1792  50 ( 50)  READY 
   3 init                          168  0.000  1128/ 2992 100 (100)  w:sem 
   6 nshterm                         1  0.000   780/ 1496  70 ( 70)  w:sem 
 127 top                            10  0.876  1164/ 1696 100 (100)  RUN   
  75 dataman                        15  0.000   612/ 1792 100 (100)  w:sem 
  77 commander                      94  0.097  2716/ 3192 215 (215)  w:sig 
  78 commander_low_prio              4  0.000   476/ 1992  50 ( 50)  w:sem 
  80 mavlink_if0                   242  0.681  1764/ 2392 100 (100)  w:sig 
  81 mavlink_rcv_if0                 2  0.000   748/ 1792 175 (175)  w:sem 
 101 sensors_task                  734  1.947  1140/ 1992 250 (250)  w:sem 
 106 sdlog2                        294  0.000  1972/ 2992  70 ( 70)  READY 
 111 gps                           160  1.168   684/ 1496 220 (220)  w:sem 
 118 ekf_att_pos_estimator        8466 25.705  3708/ 7496 215 (215)  w:sem 
 120 fw_att_control                108  0.292   884/ 1592 250 (250)  w:sem 
 122 fw_pos_control_l1              17  0.000   752/ 1592 250 (250)  w:sem 
 124 land_detector                  48  0.097   380/  992 100 (100)  READY 
 126 navigator                     192  0.486   756/ 1792 120 (120)  w:sem 
nsh> free
             total       used       free    largest
Mem:        182048     159232      22816      18208

[LorenzMeier]

And for multicopters:

Processes: 19 total, 3 running, 16 sleeping
CPU usage: 28.71% tasks, 0.68% sched, 70.61% idle
Uptime: 17.766s total, 12.912s idle

 PID COMMAND                   CPU(ms) CPU(%)  USED/STACK PRIO(BASE) STATE 
   0 Idle Task                   12912 70.611     0/    0   0 (  0)  READY 
   1 hpwork                        325  2.036   828/ 1792 192 (192)  w:sig 
   2 lpwork                         44  0.193   364/ 1792  50 ( 50)  w:sig 
   3 init                          173  0.000  1540/ 2992 100 (100)  w:sem 
   6 nshterm                         4  0.000   804/ 1496  70 ( 70)  w:sem 
 132 top                            31  0.969  1204/ 1696 100 (100)  RUN   
  75 dataman                        15  0.000   612/ 1792 100 (100)  w:sem 
  77 commander                      66  0.096  2708/ 3192 215 (215)  w:sig 
  78 commander_low_prio              2  0.000   460/ 1992  50 ( 50)  READY 
  80 mavlink_if0                   111  0.581  1764/ 2392 100 (100)  w:sig 
  81 mavlink_rcv_if0                 1  0.000   732/ 1792 175 (175)  w:sem 
 101 sensors_task                  309  1.939  1148/ 1992 250 (250)  w:sem 
 106 sdlog2                        279  0.096  1972/ 2992  70 ( 70)  w:sig 
 111 gps                            67  0.193   684/ 1496 220 (220)  w:sem 
 119 ekf_att_pos_estimator        2815 20.659  3708/ 7496 215 (215)  w:sem 
 121 mc_att_control                120  0.775   996/ 1592 250 (250)  w:sem 
 123 mc_pos_control                 84  0.581  1156/ 1592 250 (250)  w:sem 
 125 land_detector                  23  0.096   444/  992 100 (100)  READY 
 127 navigator                      83  0.484   852/ 1792 120 (120)  w:sem 
nsh> free
             total       used       free    largest
Mem:        182048     156352      25696      21152

[LorenzMeier]

I'm pretty confident that things work fine now again, closing this.