Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory leak of facedetectioncpu example on Android (RK3399 specific) #205

Closed
zhouy1017 opened this issue Oct 30, 2019 · 15 comments
Closed

Memory leak of facedetectioncpu example on Android (RK3399 specific) #205

zhouy1017 opened this issue Oct 30, 2019 · 15 comments
Assignees
@jiuqiant

Comments

@zhouy1017
Copy link

Hello,
We noticed a crash issue of facedetectioncpu example Android app directly built from repo after running hours on a 2G RAM platform.
We tested and monitored the memory usage of the application for over 12 hours on another device with 4G RAM and get a record of unusual growing memory usage.
The memory usage at the first minutes after application booted:

USER PID PPID VSZ RSS WCHAN ADDR S NAME
u0_a83 1963 399 5623744 463512 0 0 S com.google.mediapipe.apps.facedetectioncpu
USER PID PPID VSZ RSS WCHAN ADDR S NAME
u0_a83 1963 399 5623744 462452 0 0 S com.google.mediapipe.apps.facedetectioncpu

The memory usage after about 14 hours running:

USER PID PPID VSZ RSS WCHAN ADDR S NAME
u0_a83 1963 399 14258664 2922496 0 0 S com.google.mediapipe.apps.facedetectioncpu
USER PID PPID VSZ RSS WCHAN ADDR S NAME
u0_a83 1963 399 14430984 2925108 0 0 S com.google.mediapipe.apps.facedetectioncpu

(And we have the log by 30s interval indicating the growth by time).

We suspect there are some memory related issues in the application.

Here is our environment information:
Build OS: Ubuntu 18.04.3
Bazel Version: 0.29.1
Repo commit:c6fea4c9d9f3bdf081c10387d2b42ed854199d06
Testing Device: RK3399Pro
Android: 8.1.0 Kernel 4.4.167
@zhouy1017 zhouy1017 changed the title Memory leak of example of facedetectioncpu example on Android Memory leak of facedetectioncpu example on Android Oct 30, 2019
@teleger
Copy link

teleger commented Oct 30, 2019

Hello,
We noticed a crash issue of facedetectioncpu example Android app directly built from repo after running hours on a 2G RAM platform.
We tested and monitored the memory usage of the application for over 12 hours on another device with 4G RAM and get a record of unusual growing memory usage.
The memory usage at the first minutes after application booted:

USER PID PPID VSZ RSS WCHAN ADDR S NAME
u0_a83 1963 399 5623744 463512 0 0 S com.google.mediapipe.apps.facedetectioncpu
USER PID PPID VSZ RSS WCHAN ADDR S NAME
u0_a83 1963 399 5623744 462452 0 0 S com.google.mediapipe.apps.facedetectioncpu

The memory usage after about 14 hours running:

USER PID PPID VSZ RSS WCHAN ADDR S NAME
u0_a83 1963 399 14258664 2922496 0 0 S com.google.mediapipe.apps.facedetectioncpu
USER PID PPID VSZ RSS WCHAN ADDR S NAME
u0_a83 1963 399 14430984 2925108 0 0 S com.google.mediapipe.apps.facedetectioncpu

(And we have the log by 30s interval indicating the growth by time).

We suspect there are some memory related issues in the application.

Here is our environment information:
Build OS: Ubuntu 18.04.3
Bazel Version: 0.29.1
Repo commit:c6fea4c9d9f3bdf081c10387d2b42ed854199d06
Testing Device: RK3399Pro
Android: 8.1.0 Kernel 4.4.167

My environment similar to you, is also in ubuntu for android development, hardware environment is rk3399. Want to know if you have some problems, such as building the app, the terminal was slow, but also download third-party libraries, for example opencv4.1, etc., those things are particularly large, slow download, do you have any solution? Also, are you building a new app? As for the 0.6 version of mediapipe, some new functions have been added, such as aar module can be built, there are still many problems, I look forward to discussing with you, ok? You are here, just reply me directly, thank you

@zhouy1017
Copy link
Author

@teleger
Currently we foucus on the memory leak issue on the original application provided as-it-is and no extended development is planned. We are curious about your testing result on memory issues, do you have memory leak/abnormal usage on those examples?
Furthermore, for building issues, if you are in mainland China, be sure to set up correct source servers in Ubuntu to speed up dependecies downloading.

@mgyong
Copy link

mgyong commented Oct 30, 2019 •
edited
Loading

@zhouy1017 @jiuqiant is doing facedetectiongpu testing on pixel 2 phone for past 2+ hours but currently it looks like there are no memory leak issues to date. We will report more once we have done cpu.
A few questions

  1. Did the memory leak gradually such that it takes 14 hours to see? or was there a sudden jump over time?
  2. The memory leak could be something specific to RK3399Pro. It would be good if you could get another non-RK phone to test and see if this is something specific to Rockwell
  3. What is your use case for MediaPipe? Curious your interest in getting MediaPipe working on Rockwell

@jiuqiant
Copy link
Contributor

Hi, thanks for reporting this issue to us!

To reproduce this issue, I install the MediaPipe face detection cpu and gpu demo on a Google Pixel 2 XL and utilize Android Studio profiler to monitor the memory usage of each app for 2 hours. AFAICT, I don't see any significant memory leak on the Google Pixel 2 XL.

For the face detection cpu demo (in the debug mode), the memory usage of the app ranges from 380 MB to 395 MB. Here are two data points for your reference.
face_detection_cpu_memory_profiling_start
face_detection_cpu_memory_profiling_after_2_hrs

For the face detection gpu demo, the memory footprint is between 260 MB to 275 MB. We provide two data points for your reference as well. You can build the face detection gpu demo in Android Studio by cloning this repo directly.
face_detection_gpu_memory_profiling_after_start
face_detection_gpu_memory_profiling_after_2_hrs

As the next step, we probably need to get a RK3399Pro to reproduce your issue. But, it may take longer time.

@zhouy1017
Copy link
Author

@mgyong @jiuqiant
Thank you for paying atttention to this issue.

@mgyong
For your questions:

1.Did the memory leak gradually such that it takes 14 hours to see? or was there a sudden jump over time?

The memory leaks gradually. It becomes sensible after hours (crashed on a 2GB RK3288 in about 6 hours), and reaches unacceptable size after 14 hours.

2.The memory leak could be something specific to RK3399Pro. It would be good if you could get another non-RK phone to test and see if this is something specific to Rockwell

Yes. After hours of testing, I could confirm this. We tested on RK3399Pro and Samsung S7e. And the result is that no memory leak on S7e while issue presists on RK3399. You can check the results below.

3.What is your use case for MediaPipe? Curious your interest in getting MediaPipe working on Rockwell

We plan to use RK3399 to control an automatic door based on facedetection(just open the door for human faces, no face identity needed).

@jiuqiant
Thanks for your test results.
We performed test on RK3399Pro and Samsung S7e for 2 hours.
On S7e, it's pretty similar to your result on Pixel. No memory leak.
S7e

On 3399, memory leak could be significant after 2 hours.
3399

It is confirmed that the issue is RK3399 related.
And by comparing 2 profiling result, we suspect that some memory release functions are not working normally and it might be opengl related.

@zhouy1017 zhouy1017 changed the title Memory leak of facedetectioncpu example on Android Memory leak of facedetectioncpu example on Android (RK3399 specific) Oct 31, 2019
@mgyong
Copy link

mgyong commented Oct 31, 2019

@zhouy1017 Curious why you are using RK3399? Is this the production board? We love to get more info on your use case. Pls email me mgyong at google dot com

@zhouy1017
Copy link
Author

@jiuqiant @mgyong
We detected similar memory leak on Xiaomi Note3 cellphone with snapdragon 660 Soc.
The profiling is below:
xiaomi

The memory allocated for NDK increased from around 100MB to 160MB during 1-2 hours.

@jiuqiant
Copy link
Contributor

jiuqiant commented Nov 6, 2019

JFYI, we also tested the face detection cpu demo on a Xiaomi Redmi Note 5 for 2 hours and 20 minutes and I don't see significant memory leak.
xiaomi redmi note 5

@mgyong
Copy link

mgyong commented Nov 6, 2019

@zhouy1017 @jiuqiant tested on MIUI version 10.3.2 Stable, Android version 9 PKQ1.180 Redmi Note 5. Can you retest on your Note 3 (let us know what is the version?)? Thanks

@jiuqiant
Copy link
Contributor

We worked with Rockchip (the manufacturer of RK3399) and got the issue solved by upgrading the mali drivers.

@jiuqiant jiuqiant mentioned this issue Dec 18, 2019
Closed
@ToNYLin66
Copy link

We worked with Rockchip (the manufacturer of RK3399) and got the issue solved by upgrading the mali drivers.

I have Memory leak of handtrackinggpu now in RK3399 X99 TV Box, and my TV Box firmware system is the latest version.
How Can I solve this problem?

@zhouy1017
Copy link
Author

zhouy1017 commented Mar 27, 2020 via email

@jiuqiant jiuqiant mentioned this issue Sep 3, 2020
Closed
@lovehuanhuan
Copy link

Hello,
We received an system update from Rockchip and that fixed the issue we encountered.
I don’t know if Rockchip updates the update files listed on their website but maybe you can contact them directly to figure it out.

Get Outlook for iOShttps://aka.ms/o0ukef
…

From: Ting-Yu Lin [email protected]
Sent: Thursday, March 26, 2020 1:47:53 PM
To: google/mediapipe [email protected]
Cc: zhouy1017 [email protected]; Mention [email protected]
Subject: Re: [google/mediapipe] Memory leak of facedetectioncpu example on Android (RK3399 specific) (#205)

We worked with Rockchip (the manufacturer of RK3399) and got the issue solved by upgrading the mali drivers.

I have Memory leak of handtrackinggpu now in RK3399 X99 TV Box, and my TV Box firmware system is the latest version.
How Can I solve this problem?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fmediapipe%2Fissues%2F205%23issuecomment-604242560&data=02%7C01%7C%7C07e80d4a6b0b4483c35c08d7d1493b62%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637207984760125304&sdata=mcMrjYeniaWekLfvKlKS4KAL2kDv7tR3CjFM%2FYM1hn0%3D&reserved=0, or unsubscribehttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAE25QT6AIJJ4B5KBOBHBRKDRJLUATANCNFSM4JGSCT2A&data=02%7C01%7C%7C07e80d4a6b0b4483c35c08d7d1493b62%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637207984760125304&sdata=6WvMaQukq1F5yiwxEK9rxqJRAVhFbIzt1QHfz83Ji3Y%3D&reserved=0.

能否把补丁私信我一份 [email protected]

@lovehuanhuan
Copy link

I found that there is some problem with rk3399's "GLES20.glDeleteTextures" , it doesn't work well,and will cause memory leak problem. So, I created a texture pool to avoid calling GLES20.glGenTextures ererytime.

@ortegatron
Copy link

@lovehuanhuan Could you please provide code example for the change you made? I would be very thanked if you do, thanks

@sgowroji sgowroji mentioned this issue Apr 1, 2021
Closed
@sureshdagooglecom sureshdagooglecom mentioned this issue Mar 31, 2022
Closed
arttupii pushed a commit to arttupii/mediapipe that referenced this issue Nov 18, 2023
@PaulTR
Merge pull request google-ai-edge#205 from markmcd/main
819f656 
Initial import of customization notebooks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jiuqiant jiuqiant

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@mgyong @jiuqiant @ortegatron @lovehuanhuan @zhouy1017 @teleger @ToNYLin66