Skip to content
This repository has been archived by the owner on Feb 15, 2022. It is now read-only.

javascript out of memory #32

Closed
BarnumD opened this issue Aug 20, 2016 · 13 comments
Closed

javascript out of memory #32

BarnumD opened this issue Aug 20, 2016 · 13 comments

Comments

@BarnumD
Copy link
Contributor

BarnumD commented Aug 20, 2016

I've been seeing some out of memory errors when doing backfill. I'm running in a docker container (barnumd/zenbot:3.2.4). There's no memory limit on the container. And my host has 16GB of RAM, 12 available.

`08/20/2016 05:51:56 PM UTC [ reducer] processing 9/ticks sec, thought queue: 297075
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

<--- Last few GCs --->

7925875 ms: Mark-sweep 1292.6 (1434.9) -> 1292.1 (1434.9) MB, 2954.8 / 0 ms [allocation failure] [GC in old space requested].
7928486 ms: Mark-sweep 1292.1 (1434.9) -> 1292.0 (1434.9) MB, 2611.1 / 0 ms [allocation failure] [GC in old space requested].
7930980 ms: Mark-sweep 1292.0 (1434.9) -> 1291.8 (1434.9) MB, 2494.4 / 0 ms [last resort gc].
7933455 ms: Mark-sweep 1291.8 (1434.9) -> 1291.8 (1434.9) MB, 2474.5 / 0 ms [last resort gc].

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1cb8528c9e31
1: parse [native json.js:~51] [pc=0x3479c74f0a43] (this=0x1cb8528c37b1 <a JSON with map 0x3012fee095e9>,x=0x37a77b01d4a9 <String[254]: {"app":"zb","id":"zb:poloniex-43478444","key":"trade","value":{"id":"poloniex-43478444","trade_id":43478444,"time":1469526543000,"asset":"ETH","currency":"BTC","size":1.71176473,"price":0.02030962,"side":"buy","exchange":"poloniex"},"time":1469526543000}>,s=...

1: node::Abort() [node]
2: 0xfb1c8c [node]
3: v8::Utils::ReportApiFailure(char const_, char const_) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const_, bool) [node]
5: v8::internal::Factory::NewTransitionArray(int) [node]
6: v8::internal::TransitionArray::Allocate(v8::internal::Isolate_, int, int) [node]
7: v8::internal::TransitionArray::Insert(v8::internal::Handlev8::internal::Map, v8::internal::Handlev8::internal::Name, v8::internal::Handlev8::internal::Map, v8::internal::SimpleTransitionFlag) [node]
8: v8::internal::Map::CopyReplaceDescriptors(v8::internal::Handlev8::internal::Map, v8::internal::Handlev8::internal::DescriptorArray, v8::internal::Handlev8::internal::LayoutDescriptor, v8::internal::TransitionFlag, v8::internal::MaybeHandlev8::internal::Name, char const_, v8::internal::SimpleTransitionFlag) [node]
9: v8::internal::Map::CopyAddDescriptor(v8::internal::Handlev8::internal::Map, v8::internal::Descriptor_, v8::internal::TransitionFlag) [node]
10: v8::internal::Map::CopyWithField(v8::internal::Handlev8::internal::Map, v8::internal::Handlev8::internal::Name, v8::internal::Handlev8::internal::FieldType, v8::internal::PropertyAttributes, v8::internal::Representation, v8::internal::TransitionFlag) [node]
11: v8::internal::Map::TransitionToDataProperty(v8::internal::Handlev8::internal::Map, v8::internal::Handlev8::internal::Name, v8::internal::Handlev8::internal::Object, v8::internal::PropertyAttributes, v8::internal::Object::StoreFromKeyed) [node]
12: v8::internal::LookupIterator::PrepareTransitionToDataProperty(v8::internal::Handlev8::internal::JSObject, v8::internal::Handlev8::internal::Object, v8::internal::PropertyAttributes, v8::internal::Object::StoreFromKeyed) [node]
13: v8::internal::Object::AddDataProperty(v8::internal::LookupIterator_, v8::internal::Handlev8::internal::Object, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow, v8::internal::Object::StoreFromKeyed) [node]
14: v8::internal::JSObject::DefineOwnPropertyIgnoreAttributes(v8::internal::LookupIterator_, v8::internal::Handlev8::internal::Object, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow, v8::internal::JSObject::AccessorInfoHandling) [node]
15: v8::internal::JSObject::DefinePropertyOrElementIgnoreAttributes(v8::internal::Handlev8::internal::JSObject, v8::internal::Handlev8::internal::Name, v8::internal::Handlev8::internal::Object, v8::internal::PropertyAttributes) [node]
16: v8::internal::JsonParser::ParseJsonObject() [node]
17: v8::internal::JsonParser::ParseJsonValue() [node]
18: v8::internal::JsonParser::ParseJson() [node]
19: v8::internal::Runtime_ParseJson(int, v8::internal::Object*, v8::internal::Isolate) [node]
20: 0x3479c710961b
08/20/2016 05:52:40 PM UTC [ reducer] processing 4/ticks sec, thought queue: 292575
`

@grigio
Copy link
Contributor

grigio commented Aug 20, 2016

Can you link the dockerfile?

@BarnumD
Copy link
Contributor Author

BarnumD commented Aug 20, 2016

Actual run command I'm using... (The default run doesn't backfill so I pass this command the first run after removing my db for a new version.)
docker run -d -v /docker/mounts/zenbot/config.js:/usr/src/zenbot/config.js -p 3013:3013 --link mongodb:mongodb --cpu-shares=2 --name zenbot barnumd/zenbot:3.2.4 zenbot launch map --backfill reduce run server

Here's the dockerfile.
https://github.com/BarnumD/docker-zenbot/blob/master/Dockerfile

@BarnumD
Copy link
Contributor Author

BarnumD commented Aug 20, 2016

Host is Ubuntu 16.04. 1.12.0-rc4 (I should update :) )

@grigio
Copy link
Contributor

grigio commented Aug 20, 2016

I run it on Ubuntu 16.04 with node 5.9.1 and node 6.4 and I never had that kind of problem.. 2gb RAM 6gb swap. Probably your node version is buggy

@BarnumD
Copy link
Contributor Author

BarnumD commented Aug 20, 2016

I'll try node 6.4.0. Also updated to docker 1.12.1

@carlos8f
Copy link
Contributor

I'm using Ubuntu 14.04 2gb ram no swap, node 6.3.1, mongo 3.2.8, reduce command when processing a large queue sometimes crashes with a mongo timeout, respawns and then clears the queue quickly.

@BarnumD from your paste it looks like 297075 items in your thought queue, which means the reduce command is way behind. Thought queue should be around 2000-8000 items during backfill. If the thought queue keeps creeping up, check your mongodb.log for slow queries, check your config.js and remove exchanges/pairs you're not using, relax the backfill timeout, until the thought queue can be efficiently processed.

Hope that helps

@carlos8f
Copy link
Contributor

you can also try backfilling a single exchange or pair at a time, if your reducer can't keep up with all of them at once

@BarnumD
Copy link
Contributor Author

BarnumD commented Aug 20, 2016

OK, I do have more than the default pairs and exchanges enabled. I also saw some timeouts from the mongodb. So that sounds accurate. Is backfilling a single exchange at a time in the readme/docs?

@carlos8f
Copy link
Contributor

Is backfilling a single exchange at a time in the readme/docs?

No, but I plan on updating readme/docs with conversations like these under the FAQ section as they come up.

If anyone would like to add things to the FAQ please submit Pull Requests :)

@BarnumD
Copy link
Contributor Author

BarnumD commented Aug 20, 2016

My assets were:
"BTC",
"ETH",
And currencies were:
"USD",
"USDT",
"BTC"
Exchanges:
"bitfinex",
"gdax",
"poloniex",
"server"

I removed the BTC currency and so far my thought queue is <= 100

@carlos8f
Copy link
Contributor

what would be interesting is if Zenbrain (once the auto-learn engine is done) could tweak its own map/reduce parameters to adjust to the system it's running on. realtime perf analysis of its own perf :)

@grigio
Copy link
Contributor

grigio commented Aug 20, 2016

I think the config.js should only have BTC/USD by default.. also because the logic doesn't support other pairs as it is now https://github.com/carlos8f/zenbot/blob/master/default_logic.js#L45-L46

Another ambiguos part is that is that also the currencies specified in config.js could be valid for an exchange but invalid for another

@BarnumD
Copy link
Contributor Author

BarnumD commented Aug 20, 2016

I did let it run for a while with both ETH and BTC assets running and the queue did get out of hand again with very few tick/s.

Alright. I'll just separate asset into their own running instances and mongo databases.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants