increasing CPU and memory usage after running for several hours

[timbleck]

After starting hklifx on my C.H.I.P. (ARMv7, Debian based Linux) i do see an increase of CPU and memory usage after running for several hours in the background. Regarding top and ps looks like hklifx is starting more and more threads that are not closed anymore. Even looks like the process then crashes at some point (see hklifx-log.txt).

[steventroughtonsmith]

I see the same on my iMac with 24GB RAM—it expands to fill all available memory over time. Not safe to keep running for long periods

[brutella]

I think the reason why this happens is that the connection to the HomeKit clients is never closed. iOS tries to keep the connection alive as long as possible. We should set a reasonable connection timeout. I've opened an issue in the hc project.

[broglea]

I run this on my raspberry pi model b+ and get OOM after it has been running for about a day.

[medazzouzi]

Hello,
I run or on my synology or has gone OOM in a day.

Le 4 oct. 2016 ? 18:54, Austin Brogle <[email protected]mailto:[email protected]> a ?crit :

I run this on my raspberry pi model b+ and get OOM after it has been running for about a day.

You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/22#issuecomment-251446360, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AQ070IztUUMBQ4YII6aLaMMrIMuhFh0Gks5qwoShgaJpZM4Jkoi7.

[brutella]

I've added a possible fix to the hc project.

Since hklifx and hc now support vendor directories, you don't have to manually fetch the changes of all dependencies. Just fetch the changes of the hklifx master branch.

[steventroughtonsmith]

Will keep an eye on it and report back. It does still seem to be climbing in memory usage before any clients interact with it through the UI. As much as 0.5MB a second for a period, every few seconds

[lstemple]

To add my experience -- I too had the out-of memory error on an Raspberry Pi 2 Model b, after about 5 hours. I was redirecting std-err to a log file which ended up being around 50MB. There appears to be a goroutine leak of some kind, since the log contained about 90k goroutine stack traces -- the vast majority of which looked like this:

goroutine 1444 [chan send, 440 minutes]: hklifx/vendor/github.com/pdf/golifx/protocol/v2/device.(*Device).Handle(0x10918000, 0x116119f0) /home/pi/dev/go/src/hklifx/vendor/github.com/pdf/golifx/protocol/v2/device/device.go:571 +0x70 hklifx/vendor/github.com/pdf/golifx/protocol.(*V2).process(0x10877570, 0x116119f0, 0x109fec00) /home/pi/dev/go/src/hklifx/vendor/github.com/pdf/golifx/protocol/v2.go:552 +0x8254 created by hklifx/vendor/github.com/pdf/golifx/protocol.(*V2).dispatcher /home/pi/dev/go/src/hklifx/vendor/github.com/pdf/golifx/protocol/v2.go:356 +0x640

I hope this helps! I am really enjoying learning from this project 👍

[brutella]

In my experience all other hc based projects run without memory issues for month. I think that the memory leak happens in golifx – but I currently don't have time to look into that.

[taz030485]

Yeah I'm getting OOM errors as well.
Launched at 11:52pm and crashed sometime around 4:22am
stdout stderr both logged out log.txt

[bland328]

I opened Issue 31 (#31) just minutes ago, before realizing my hklifx problems might also be due to OOM failures.

I'm now witnessing that after running for just 20 minutes, hklifxd is (according to top) consuming 10% of memory, and still climbing.

[pdf]

@brutella I'm not sure how you're doing vendoring, or I'd submit a PR, but if you make this change:

diff --git a/hklifxd.go b/hklifxd.go
index 4e67063..1eac00e 100644
--- a/hklifxd.go
+++ b/hklifxd.go
@@ -47,7 +47,7 @@ func Connect() {
 
        client.SetDiscoveryInterval(30 * time.Second)
 
-       sub, _ := client.NewSubscription()
+       sub := client.Subscribe()
        for {
                event := <-sub.Events()
                switch event.(type) {
@@ -81,7 +81,7 @@ func NewDevice(device common.Device) {
        if light, ok := device.(common.Light); ok {
                hkLight := GetHKLight(light)
 
-               hkLight.sub, _ = light.NewSubscription()
+               hkLight.sub = light.Subscribe()
                for {
                        event := <-hkLight.sub.Events()
                        switch event.(type) {
@@ -115,7 +115,7 @@ func NewDevice(device common.Device) {
 func ExpireDevice(device common.Device) {
        if light, ok := device.(common.Light); ok {
                if hkLight, found := lights[light.ID()]; found == true {
-                       light.CloseSubscription(hkLight.sub)
+                       hkLight.sub.Close()
                        hkLight.transport.Stop()
 
                        delete(lights, light.ID())

And update golifx to v1.0.1, I've made changes to golifx that may help, but I don't have any Apple stuff to test this particular code with. The changes are not well tested by me at this stage though, as I've had very little time to work that code for quite a while.

If the issue is not resolved by this, users who can reproduce the problem will need to provide some profiling (ie - pprof) data to have any hope of working out where the leak is.

[brutella]

@pdf Thanks, golifx v1.0.1 is now available in golifx-dnssd branch. This branch also includes the dnssd branch from hc which eliminates connection issues.

Everybody interested in those changes, do the following:

1. Checkout the golifx-dnssd branch git checkout golifx-dnssd
2. Run go get
3. Run hklifxd.go

[brutella]

Anyone still experiences this issue?