Skip to content

Commit

Permalink
add memory growth test case
Browse files Browse the repository at this point in the history
  • Loading branch information
timotheecour committed Jan 25, 2019
1 parent 09bdc2b commit e5c8de5
Show file tree
Hide file tree
Showing 6 changed files with 97 additions and 5 deletions.
12 changes: 7 additions & 5 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
*/tests/**
!*/tests/**/*.*

*/tests_imported/**
!*/tests_imported/**/*.*
## ignore all files wo extension D20190125T022032
*
!/**/
!*.*

htmldocs

*.dylib
!Makefile
15 changes: 15 additions & 0 deletions testcases/tests/t0126/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# NIMBASE = ~/lib/Nim/lib/nimbase.h
NIMBASE = $nimc_D/lib/nimbase.h

.PHONY: run

run: a.out
./a.out

a.out: test.cpp csrc/clib.cpp csrc/nimbase.h
g++ -Icsrc test.cpp

csrc/clib.cpp: clib.nim
nim cpp --checks:on -d:traceGC --memTracker:on -d:memProfiler -d:lineDir -d:leakDetector -d:useSysAssert -d:useGcAssert --header --nimcache:csrc --compileOnly ./clib.nim

csrc/nimbase.h: $(NIMBASE) csrc/clib.cpp
12 changes: 12 additions & 0 deletions testcases/tests/t0126/build.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
build(){
(
set -e
temp_D=/tmp/D20190124T204416/
nim c -o:$temp_D/libclib.dylib --app:lib --nimcache:$temp_D clib.nim
clang++ -o $temp_D/test test.cpp -lclib -L$temp_D/
$temp_D/test > $temp_D/log.txt
grep 'occupied memory' $temp_D/log.txt |less
)
}

build
19 changes: 19 additions & 0 deletions testcases/tests/t0126/clib.nim
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import strformat
import strutils
import sequtils

echo "clib initialized"
echo GC_getStatistics()

var s0 = "foobar"
var s1 = s0.cstring
proc hello_echo*(message:cstring):cstring {.exportc.} =
## Echo a message back
echo GC_getStatistics()
when defined(case1):
var s_message = $message
result = &"{s_message} echo"
echo &"Nim echo result: {result}"
else:
result = s1

13 changes: 13 additions & 0 deletions testcases/tests/t0126/readme.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
## command
running
```
sh build.sh
```
shows no memory growth

## links
* Nim memory growth
* [node-nim-memory-bug/clib.nim at master · iffy/node-nim-memory-bug](https://github.com/iffy/node-nim-memory-bug/blob/master/clib.nim)
* https://gist.github.com/iffy/c0e5e962bf3f14acf41dc6969f322747 (first version)
* https://gist.github.com/iffy/7ad0755869240ebe295b1306179b7042 (2nd version)

31 changes: 31 additions & 0 deletions testcases/tests/t0126/test.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#include <iostream>
// #include "csrc/clib.h"
using namespace std;

// N_CDECL(void, NimMain)(void);

extern "C"
char* hello_echo(char* message);

std::string echo(std::string arg) {
const char* message = arg.c_str();
cout << "C++ passing message: " << message << "\n";
cout << "C++ message addr: " << (void*)message << "\n";
char* retval = hello_echo(
(char *)(message)
);
cout << "C++ retval addr : " << (void*)retval << "\n";
return std::string(retval);
}

int main() {
// NimMain();
cout << "Hello world\n";
int i = 0;
for (i = 0; i < 1000; i++) {
std::string msg = std::string("Hello");
std::string ret = echo(msg);
cout << "Got reply: " << ret << "\n";
}
return 0;
}

7 comments on commit e5c8de5

@timotheecour
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/cc @iffy @rayman22201

from discussion here https://irclogs.nim-lang.org/25-01-2019.html#04:39:05

@timotheecour I'll have to try using --app:lib, but I'm seeing memory growth with this version: https://gist.github.com/iffy/7ad0755869240ebe295b1306179b7042 (And thanks for the clang++ example)

please check test case (btw simpler when it's in a git repo instead of a gist); it shows no memory growth after 1K iterations

the way you were doing in https://gist.github.com/iffy/7ad0755869240ebe295b1306179b7042#file-test-cpp (calling NimMain(); from C++) seems weird to me, did you find this pattern from somewhere?
using --app:lib , as i did above, works

@iffy
Copy link

@iffy iffy commented on e5c8de5 Jan 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@timotheecour

In your test case, when I change line 26 to std::string msg = "Hello " + std::string(i, '-'); I see that memory goes up and down, but it seems to inflate much more than it should. The string is never longer than ~1000 characters but I'm seeing memory usage go up to 206K.

If I change the iterations to 10000 memory usage still goes up and down, but ends up at ~1.5MB

Do you see this same behavior?

I got the NimMain thing from some work I did with nimx trying to get iOS and Android things to work. I like what you've done in getting rid of it. Thanks for the example!

@timotheecour
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got the NimMain thing from some work I did with nimx trying to get iOS and Android things to work

if u got it from somewhere else please tell them to use --app:lib :)

investigating ur other point

@timotheecour
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added plotting so we see clearly it goes in cycles, as expected:
image

@timotheecour
Copy link
Owner Author

@timotheecour timotheecour commented on e5c8de5 Jan 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but it seems to inflate much more than it should. The string is never longer than ~1000 characters but I'm seeing memory usage go up to 206K

200 iters
image

10000 iters truncated to 200 for visibility

image

=>

If I change the iterations to 10000 memory usage still goes up and down, but ends up at ~1.5MB

i see no difference in max mem regardless of number of iters

@rayman22201
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but it seems to inflate much more than it should.

I think you are on to something! Similar inflation seems to be happening with gc:regions as well. (though I haven't graphed it so nicely)
Maybe it has something to do with the string implementation?

@timotheecour
Copy link
Owner Author

@timotheecour timotheecour commented on e5c8de5 Jan 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • string implementation has issues (eg for a in 0..<n: let s = "bar" allocates n times, whereas in D it doens't) but i don't think that's the issue here

  • as u see, memory grows linearly then halves; and it repeats; that makes sense; memory isn't collected after each string allocation but only after GC deems it's necessary; so it'll have not 1 string of 1K chars but P strings; and then when GC does the P+1 allocation, a threshold is reached and a GC collection happens; but instead of releasing all memory (which would be inefficient, I think that's what GC bug: seems very slow where it shouldn't; maybe it leaks? nim-lang/Nim#10040 fixed), it just halves it ; @Araq is my analysis correct?

Please sign in to comment.