Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exceptions collecting container.stats.memory.percent and container.stats.cpu.percent #379

Open
aledsage opened this issue Nov 11, 2016 · 3 comments
Open

Exceptions collecting container.stats.memory.percent and container.stats.cpu.percent #379

aledsage opened this issue Nov 11, 2016 · 3 comments

Comments

@aledsage
Copy link
Member

In the logs, I see the two exceptions shown below:

2016-10-25 21:50:16,015 WARN  117 o.a.b.c.f.AttributePollHandler [r-NmGPUwnZ-30374] unable to compute VanillaSoftwareProcessImpl{id=cb3pwep5pc}->Sensor: container.stats.memory.percent (java.lang.Double); on val=org.apache.brooklyn.feed.ssh.SshPollValue@1df4a1b0
org.apache.brooklyn.util.javalang.coerce.ClassCoercionException: Cannot coerce "70   0.00%" to java.lang.Double (70   0.00%): adapting failed
        at org.apache.brooklyn.util.javalang.coerce.PrimitiveStringTypeCoercions.stringToPrimitive(PrimitiveStringTypeCoercions.java:202)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.PrimitiveStringTypeCoercions.tryCoerce(PrimitiveStringTypeCoercions.java:47)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.TypeCoercerExtensible.tryCoerceInternal(TypeCoercerExtensible.java:131)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.TypeCoercerExtensible.tryCoerce(TypeCoercerExtensible.java:107)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.TypeCoercerExtensible.coerce(TypeCoercerExtensible.java:97)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.TypeCoercerExtensible.coerce(TypeCoercerExtensible.java:93)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.core.flags.TypeCoercions.coerce(TypeCoercions.java:79)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.sensor.ssh.SshCommandSensor$3.apply(SshCommandSensor.java:130)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.sensor.ssh.SshCommandSensor$3.apply(SshCommandSensor.java:127)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at com.google.common.base.Functions$FunctionComposition.apply(Functions.java:211)[67:com.google.guava:16.0.1]
        at org.apache.brooklyn.core.feed.AttributePollHandler.transformValueOnSuccess(AttributePollHandler.java:114)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.AttributePollHandler.onSuccess(AttributePollHandler.java:101)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.DelegatingPollHandler.onSuccess(DelegatingPollHandler.java:51)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.Poller$PollJob$1.run(Poller.java:78)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.Poller$1$1.call(Poller.java:156)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.Poller$1$1.call(Poller.java:149)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:359)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$ScheduledTaskCallable$1.call(BasicExecutionManager.java:436)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745)[:1.8.0_101]


2016-11-10 19:06:31,400 WARN  117 o.a.b.c.f.AttributePollHandler [-NmGPUwnZ-229662] unable to compute VanillaSoftwareProcessImpl{id=q9wq6umn7m}->Sensor: container.stats.cpu.percent (java.lang.Double); on val=org.apache.brooklyn.feed.ssh.SshPollValue@66242ba5
org.apache.brooklyn.util.javalang.coerce.ClassCoercionException: Cannot coerce "205b7205497b8f7887f5" to java.lang.Double (205b7205497b8f7887f5): adapting failed
        at org.apache.brooklyn.util.javalang.coerce.PrimitiveStringTypeCoercions.stringToPrimitive(PrimitiveStringTypeCoercions.java:202)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.PrimitiveStringTypeCoercions.tryCoerce(PrimitiveStringTypeCoercions.java:47)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.TypeCoercerExtensible.tryCoerceInternal(TypeCoercerExtensible.java:131)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.TypeCoercerExtensible.tryCoerce(TypeCoercerExtensible.java:107)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.TypeCoercerExtensible.coerce(TypeCoercerExtensible.java:97)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.TypeCoercerExtensible.coerce(TypeCoercerExtensible.java:93)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.core.flags.TypeCoercions.coerce(TypeCoercions.java:79)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.sensor.ssh.SshCommandSensor$3.apply(SshCommandSensor.java:130)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.sensor.ssh.SshCommandSensor$3.apply(SshCommandSensor.java:127)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at com.google.common.base.Functions$FunctionComposition.apply(Functions.java:211)[67:com.google.guava:16.0.1]
        at org.apache.brooklyn.core.feed.AttributePollHandler.transformValueOnSuccess(AttributePollHandler.java:114)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.AttributePollHandler.onSuccess(AttributePollHandler.java:101)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.DelegatingPollHandler.onSuccess(DelegatingPollHandler.java:51)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.Poller$PollJob$1.run(Poller.java:78)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.Poller$1$1.call(Poller.java:156)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.Poller$1$1.call(Poller.java:149)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:359)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$ScheduledTaskCallable$1.call(BasicExecutionManager.java:436)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745)[:1.8.0_101]

This tells us that the bash below (from https://github.com/brooklyncentral/clocker/blob/master/common/catalog/docker/docker.bom#L543-L562) is incorrect - i.e. the cut commands are picking up the wrong characters. It looks really brittle. I'd have though we should split it on the separator, rather than looking for a specific character range.

        - type: org.apache.brooklyn.core.sensor.ssh.SshCommandSensor
          brooklyn.config:
            name: container.stats.cpu.percent
            period: 1m
            targetType: double
            command: |
              test -f pid.txt &&
                ( docker stats --no-stream $(cat pid.txt) |
                  tail -n +2 |
                  cut -c21-40 )
        - type: org.apache.brooklyn.core.sensor.ssh.SshCommandSensor
          brooklyn.config:
            name: container.stats.memory.percent
            period: 1m
            targetType: double
            command: |
              test -f pid.txt &&
                ( docker stats --no-stream $(cat pid.txt) |
                  tail -n +2 |
                  cut -c63-82 )
@aledsage
Copy link
Member Author

My output from docker stats --no-stream $(cat pid.txt) is:

CONTAINER                                                          CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
bf29e117ac83803eb92dbec77d5a5e29d3453cc5c04a6a5ad98537160b77f90d   1.39%               685.9MiB / 3.518GiB   19.04%              0B / 0B             909kB / 760MB       0

We should be able to instead usedocker stats --no-stream --format "table {{.CPUPerc}}" $(cat pid.txt), which gives:

CPU %
1.46%

And docker stats --no-stream --format "table {{.MemPerc}}" $(cat pid.txt), which gives:

MEM %
19.04%

(But we'd still need to strip off the % for this to be coerced to a double).

@geomacy
Copy link
Contributor

geomacy commented Jan 26, 2018

In that case you could do

docker stats --no-stream --format "table {{.CPUPerc}}" $(cat pid.txt) | sed -n '2s/%//p'

to get just the 1.46 and similarly for the memory percent.

@aledsage
Copy link
Member Author

See fix in #446

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@aledsage @geomacy