cgroup quota shows up as -1 #8427

bparees · 2016-04-08T16:05:31Z

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/14128/testReport/(root)/Extended/_builds__Conformance__s2i_build_with_a_quota_Building_from_a_template_should_create_an_s2i_build_with_a_quota_and_run_it/

build knows the quota limit is 6000:

 I0407 00:33:34.039640       1 builder.go:145] Running build with cgroup limits: api.CGroupLimits{MemoryLimitBytes:209715200, CPUShares:61, CPUPeriod:100000, CPUQuota:6000, MemorySwap:209715200}

build creates a container with a quota value of 6000:

 Creating container with options {Name:"" Config:&{Hostname: Domainname: User: Memory:0 MemorySwap:0 MemoryReservation:0 KernelMemory:0 CPUShares:0 CPUSet: AttachStdin:false AttachStdout:true AttachStderr:false PortSpecs:[] ExposedPorts:map[] StopSignal: Tty:false OpenStdin:true StdinOnce:true Env:[OPENSHIFT_BUILD_NAME=s2i-build-quota-1 OPENSHIFT_BUILD_NAMESPACE=extended-test-s2i-build-quota-73hkx-7vwau] Cmd:[/bin/sh -c tar -C /tmp -xf - && /tmp/scripts/assemble] DNS:[] Image:centos/ruby-22-centos7:latest Volumes:map[] VolumeDriver: VolumesFrom: WorkingDir: MacAddress: Entrypoint:[] NetworkDisabled:false SecurityOpts:[] OnBuild:[] Mounts:[] Labels:map[]} HostConfig:&{Binds:[] CapAdd:[] CapDrop:[KILL MKNOD SETGID SETUID SYS_CHROOT] GroupAdd:[] ContainerIDFile: LxcConf:[] Privileged:false PortBindings:map[] Links:[] PublishAllPorts:false DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] VolumesFrom:[] NetworkMode:container:b03447dab75ae8295f5acc07474232038a65f0b95de206775138835ba0699138 IpcMode: PidMode: UTSMode: RestartPolicy:{Name: MaximumRetryCount:0} Devices:[] LogConfig:{Type: Config:map[]} ReadonlyRootfs:false SecurityOpt:[] CgroupParent: Memory:209715200 MemorySwap:209715200 MemorySwappiness:0 OOMKillDisable:false CPUShares:61 CPUSet: CPUSetCPUs: CPUSetMEMs: CPUQuota:6000 CPUPeriod:100000 BlkioWeight:0 Ulimits:[] VolumeDriver: OomScoreAdj:0}}

the container dumps the cgroup filesystem value and sees -1:

    I0407 00:33:36.030147       1 sti.go:581] MEMORY=209715200
    I0407 00:33:36.031367       1 sti.go:581] MEMORYSWAP=209715200
    I0407 00:33:36.032448       1 sti.go:581] QUOTA=-1
    I0407 00:33:36.033398       1 sti.go:581] SHARES=61
    I0407 00:33:36.034333       1 sti.go:581] PERIOD=100000

The text was updated successfully, but these errors were encountered:

bparees · 2016-04-08T16:06:15Z

@smarterclayton fyi. i made it a p2 since the exposure is that some builds might be able to exceed an enforced cpu quota. move up if you think it should be higher.

smarterclayton · 2016-04-08T16:06:56Z

Is that not a blocker for online?

bparees · 2016-04-08T16:17:51Z

i don't know. it could be, i'd say it depends on how frequently it occurs and whether that makes it an avenue for abuse. if 1/100 builds run w/o cpu quota i'm not sure that's a problem? if it's frequent enough that someone could explicitly take advantage of it, maybe it is. hence my putting the question to you :) i wouldn't argue against p1, but i wasn't going to make it one either.

mrunalp · 2016-04-08T16:37:38Z

We have added #8406 to gather additional debugging information.

mrunalp · 2016-04-08T16:38:12Z

@bparees I am not sure if that is available in the link that you posted in this bug as the link doesn't open right now.

bparees · 2016-04-08T16:44:46Z

@mrunalp looks like jenkins is down at the moment. and of course the build itself may get cleared at some point. but i put the relevant details from the failure in the description.

mrunalp · 2016-04-08T16:47:52Z

@bparees Were you able to capture the output of /proc/self/cgroup that we added?

bparees · 2016-04-08T16:58:54Z

@mrunalp sorry if i confused you, the job i linked was the original case where @smarterclayton hit it, it is not a new incident w/ your debugging added.

mrunalp · 2016-04-08T16:59:20Z

@bparees ack

bparees · 2016-04-13T20:19:09Z

@mrunalp looks like it recreated w/ your debug here: https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/167/testReport/junit/%28root%29/Extended/_builds__Conformance__s2i_build_with_a_quota_Building_from_a_template_should_create_an_s2i_build_with_a_quota_and_run_it/

mrunalp · 2016-04-13T21:27:51Z

@bparees Thanks! I am looking into it.

mrunalp · 2016-04-19T19:32:55Z

Looking into systemd as possible cause per the debug output.

mrunalp · 2016-04-26T17:09:27Z

@bparees Is the image used in the test fedora based? Does it have the findmnt command in it?
I would like to add it to gather more debug information.

mrunalp · 2016-04-26T17:46:50Z

Created #8635 to collect additional debug information.

mrunalp · 2016-05-02T16:55:32Z

From debug output in #8707, I see that the cgroups mounts are correct ruling out a race there. More likely the value is getting overwritten. I will check with the kernel team.

        I0502 12:51:10.645812       1 docker.go:623] Attaching to container "0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a" ...
        I0502 12:51:10.646351       1 docker.go:632] Starting container "0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a" ...
        I0502 12:51:11.168104       1 docker.go:656] Waiting for container "0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a" to stop ...
        I0502 12:51:11.372743       1 sti.go:583] MEMORY=209715200
        I0502 12:51:11.373543       1 sti.go:583] MEMORYSWAP=209715200
        I0502 12:51:11.374877       1 sti.go:583] QUOTA=-1
        I0502 12:51:11.375617       1 sti.go:583] SHARES=61
        I0502 12:51:11.376372       1 sti.go:583] PERIOD=100000
        I0502 12:51:11.377177       1 sti.go:583] 10:blkio:/system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope
        I0502 12:51:11.377190       1 sti.go:583] 9:devices:/system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope
        I0502 12:51:11.377196       1 sti.go:583] 8:perf_event:/system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope
        I0502 12:51:11.377202       1 sti.go:583] 7:cpuacct,cpu:/system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope
        I0502 12:51:11.377208       1 sti.go:583] 6:hugetlb:/system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope
        I0502 12:51:11.377214       1 sti.go:583] 5:net_cls:/system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope
        I0502 12:51:11.377219       1 sti.go:583] 4:memory:/system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope
        I0502 12:51:11.377225       1 sti.go:583] 3:freezer:/system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope
        I0502 12:51:11.377232       1 sti.go:583] 2:cpuset:/system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope
        I0502 12:51:11.377238       1 sti.go:583] 1:name=systemd:/system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope
        I0502 12:51:11.378020       1 sti.go:583] 317 98 253:12 /rootfs / rw,relatime - xfs /dev/mapper/docker-202:2-134674389-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a rw,context="system_u:object_r:svirt_sandbox_file_t:s0:c232,c839",nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota
        I0502 12:51:11.378038       1 sti.go:583] 318 317 0:76 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
        I0502 12:51:11.378047       1 sti.go:583] 319 317 0:77 / /dev rw,nosuid - tmpfs tmpfs rw,context="system_u:object_r:svirt_sandbox_file_t:s0:c232,c839",mode=755
        I0502 12:51:11.378054       1 sti.go:583] 320 319 0:78 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,context="system_u:object_r:svirt_sandbox_file_t:s0:c232,c839",gid=5,mode=620,ptmxmode=666
        I0502 12:51:11.378061       1 sti.go:583] 321 317 0:66 / /sys ro,nosuid,nodev,noexec,relatime - sysfs sysfs ro,seclabel
        I0502 12:51:11.378066       1 sti.go:583] 322 321 0:83 / /sys/fs/cgroup ro,nosuid,nodev,noexec,relatime - tmpfs tmpfs ro,context="system_u:object_r:svirt_sandbox_file_t:s0:c232,c839",mode=755
        I0502 12:51:11.378072       1 sti.go:583] 323 322 0:20 /system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope /sys/fs/cgroup/systemd ro,nosuid,nodev,noexec,relatime master:9 - cgroup cgroup rw,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
        I0502 12:51:11.378078       1 sti.go:583] 324 322 0:22 /system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope /sys/fs/cgroup/cpuset ro,nosuid,nodev,noexec,relatime master:10 - cgroup cgroup rw,cpuset
        I0502 12:51:11.378086       1 sti.go:583] 325 322 0:23 /system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope /sys/fs/cgroup/freezer ro,nosuid,nodev,noexec,relatime master:11 - cgroup cgroup rw,freezer
        I0502 12:51:11.378092       1 sti.go:583] 326 322 0:24 /system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope /sys/fs/cgroup/memory ro,nosuid,nodev,noexec,relatime master:12 - cgroup cgroup rw,memory
        I0502 12:51:11.378099       1 sti.go:583] 327 322 0:25 /system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope /sys/fs/cgroup/net_cls ro,nosuid,nodev,noexec,relatime master:13 - cgroup cgroup rw,net_cls
        I0502 12:51:11.378105       1 sti.go:583] 328 322 0:26 /system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope /sys/fs/cgroup/hugetlb ro,nosuid,nodev,noexec,relatime master:14 - cgroup cgroup rw,hugetlb
        I0502 12:51:11.378111       1 sti.go:583] 330 322 0:27 /system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope /sys/fs/cgroup/cpuacct,cpu ro,nosuid,nodev,noexec,relatime master:15 - cgroup cgroup rw,cpuacct,cpu
        I0502 12:51:11.378117       1 sti.go:583] 331 322 0:28 /system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope /sys/fs/cgroup/perf_event ro,nosuid,nodev,noexec,relatime master:16 - cgroup cgroup rw,perf_event
        I0502 12:51:11.378123       1 sti.go:583] 332 322 0:29 /system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope /sys/fs/cgroup/devices ro,nosuid,nodev,noexec,relatime master:17 - cgroup cgroup rw,devices
        I0502 12:51:11.378129       1 sti.go:583] 333 322 0:30 /system.slice/docker-0699667a8663265ded1e64476b50c471dde27d485d7a18499b33fe3ec75f9a0a.scope /sys/fs/cgroup/blkio ro,nosuid,nodev,noexec,relatime master:18 - cgroup cgroup rw,blkio
        I0502 12:51:11.378136       1 sti.go:583] 337 317 0:37 / /run/secrets rw,nosuid,nodev,noexec,relatime - tmpfs tmpfs rw,context="system_u:object_r:svirt_sandbox_file_t:s0:c232,c839"
        I0502 12:51:11.378142       1 sti.go:583] 338 317 202:2 /var/lib/docker/containers/7ee8d2d25f19e827a2dc66d59605895e9b2198e2316f8632690945e61e94112e/resolv.conf /etc/resolv.conf rw,relatime master:1 - xfs /dev/xvda2 rw,seclabel,attr2,inode64,noquota
        I0502 12:51:11.378148       1 sti.go:583] 339 317 202:2 /var/lib/docker/containers/7ee8d2d25f19e827a2dc66d59605895e9b2198e2316f8632690945e61e94112e/hostname /etc/hostname rw,relatime master:1 - xfs /dev/xvda2 rw,seclabel,attr2,inode64,noquota
        I0502 12:51:11.378155       1 sti.go:583] 340 317 253:2 /pods/739222bd-1064-11e6-b81f-0e832540ffef/etc-hosts /etc/hosts rw,relatime master:27 - xfs /dev/mapper/docker--vg-openshift--xfs--vol--dir rw,seclabel,attr2,inode64,grpquota
        I0502 12:51:11.378161       1 sti.go:583] 341 319 0:38 / /dev/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,context="system_u:object_r:svirt_sandbox_file_t:s0:c232,c839",size=65536k
        I0502 12:51:11.378167       1 sti.go:583] 342 319 0:13 / /dev/mqueue rw,nosuid,nodev,noexec,relatime - mqueue mqueue rw,seclabel
        I0502 12:51:11.378172       1 sti.go:583] 162 318 0:76 /bus /proc/bus ro,nosuid,nodev,noexec,relatime - proc proc rw
        I0502 12:51:11.378178       1 sti.go:583] 170 318 0:76 /fs /proc/fs ro,nosuid,nodev,noexec,relatime - proc proc rw
        I0502 12:51:11.378184       1 sti.go:583] 186 318 0:76 /irq /proc/irq ro,nosuid,nodev,noexec,relatime - proc proc rw
        I0502 12:51:11.378192       1 sti.go:583] 187 318 0:76 /sys /proc/sys ro,nosuid,nodev,noexec,relatime - proc proc rw
        I0502 12:51:11.378198       1 sti.go:583] 199 318 0:76 /sysrq-trigger /proc/sysrq-trigger ro,nosuid,nodev,noexec,relatime - proc proc rw
        I0502 12:51:11.378204       1 sti.go:583] 200 318 0:77 /null /proc/kcore rw,nosuid - tmpfs tmpfs rw,context="system_u:object_r:svirt_sandbox_file_t:s0:c232,c839",mode=755
        I0502 12:51:11.378210       1 sti.go:583] 201 318 0:77 /null /proc/timer_stats rw,nosuid - tmpfs tmpfs rw,context="system_u:object_r:svirt_sandbox_file_t:s0:c232,c839",mode=755

smarterclayton · 2016-05-11T21:50:26Z

Back up to p1 since we're out of the 1.2 triage window.

mrunalp · 2016-05-11T22:11:23Z

I have opened a kernel bug and will update here once I hear back more on it.

bparees · 2016-06-22T02:08:59Z

another occurance: https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/2229/consoleFull

bparees · 2016-06-30T21:57:01Z

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/2835/consoleFull

mrunalp · 2016-07-06T18:50:51Z

I just heard back on the kernel bug. There was in fact a race in the kernel around writing these values. Can you check what kernel version do we have right now?

mrunalp · 2016-08-11T14:49:56Z

This is being backported right now. Will updated once new packages are available with the fixes.

0xmichalis · 2016-08-24T10:39:54Z

Tagging as a flake so I can link it to PRs with failed merge jobs and have the bot happy.

ncdc · 2016-11-07T18:46:46Z

This was fixed in the 7.3 kernel, right?

mrunalp · 2016-11-07T18:48:35Z

Yes the fix is in the 7.3 kernel. If we saw the latest instance on 7.3 then I will follow up with kernel team.

On Nov 7, 2016, at 10:46 AM, Andy Goldstein [email protected] wrote:

This was fixed in the 7.3 kernel, right?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

ncdc · 2016-11-07T18:49:12Z

I was more asking in hopes of being able to close this now :-)

mrunalp · 2016-11-07T18:50:35Z

Oh okay :) maybe close now and we could reopen if necessary?

On Nov 7, 2016, at 10:49 AM, Andy Goldstein [email protected] wrote:

I was more asking in hopes of being able to close this now :-)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

ncdc · 2016-11-07T18:51:29Z

SGTM.

smarterclayton · 2017-04-13T20:33:35Z

It's baaaaack: https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin/739/testReport/junit/(root)/Extended/_builds__Conformance__s2i_build_with_a_quota_Building_from_a_template_should_create_an_s2i_build_with_a_quota_and_run_it/

mrunalp · 2017-04-13T23:41:30Z

Will check if the kernel is missing the patch they added earlier.

…

On Apr 13, 2017, at 1:33 PM, Clayton Coleman ***@***.***> wrote: It's baaaaack: https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin/739/testReport/junit/(root)/Extended/_builds__Conformance__s2i_build_with_a_quota_Building_from_a_template_should_create_an_s2i_build_with_a_quota_and_run_it/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

openshift-bot · 2018-02-10T04:25:45Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot · 2018-03-14T11:12:12Z

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

openshift-bot · 2018-04-13T17:02:45Z

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

bparees added kind/bug Categorizes issue or PR as related to a bug. priority/P2 labels Apr 8, 2016

bparees assigned mrunalp Apr 8, 2016

mrunalp mentioned this issue Apr 26, 2016

Add mount debug information for cpu quota test flake #8635

Merged

bparees mentioned this issue May 2, 2016

s2i build with a quota extended test flake #8707

Closed

bparees mentioned this issue May 11, 2016

migrate use of s.DefaultConvert to autoConvert #8840

Merged

smarterclayton added priority/P1 and removed priority/P2 labels May 11, 2016

This was referenced Jun 15, 2016

[builds][Conformance] s2i build with a quota #9338

Closed

[builds][Conformance] s2i build with a quota Building from a template [It] should create an s2i build with a quota and run it #9222

Closed

danwinship mentioned this issue Aug 17, 2016

Fix validation of pkg/sdn/api object updates #10466

Merged

ncdc mentioned this issue Aug 18, 2016

UPSTREAM: 30796: Quota usage checking ignores unrelated resources #10493

Merged

0xmichalis mentioned this issue Aug 23, 2016

deploy: update validation for deploymentconfigs #9193

Merged

0xmichalis added the kind/test-flake Categorizes issue or PR as related to test flakes. label Aug 24, 2016

miminar mentioned this issue Aug 31, 2016

[DO NOT MERGE] registry: Suppress unnecessary unauthorized errors #10715

Closed

smarterclayton mentioned this issue Sep 6, 2016

oc should not fail during version negotiation against kube server #10824

Merged

danmcp added the component/containers label Sep 7, 2016

bparees mentioned this issue Sep 26, 2016

start-build from-webhook: use canonical hostport to compare with config #10836

Merged

enj mentioned this issue Oct 19, 2016

Remove -a flag from hack/test-integration.sh #11341

Merged

ncdc closed this as completed Nov 7, 2016

smarterclayton reopened this Apr 13, 2017

knobunc mentioned this issue Apr 14, 2017

Prevent the router from deadlocking itself when calling Commit() #13717

Merged

0xmichalis mentioned this issue Apr 18, 2017

Update logging in our deployment controllers #13762

Merged

sdodson mentioned this issue Apr 25, 2017

Update swap disable tasks openshift/openshift-ansible#3976

Merged

openshift-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 10, 2018

openshift-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Mar 14, 2018

openshift-ci-robot closed this as completed Apr 13, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cgroup quota shows up as -1 #8427

cgroup quota shows up as -1 #8427

bparees commented Apr 8, 2016 •

edited by stevekuznetsov

Loading

bparees commented Apr 8, 2016

smarterclayton commented Apr 8, 2016

bparees commented Apr 8, 2016

mrunalp commented Apr 8, 2016

mrunalp commented Apr 8, 2016

bparees commented Apr 8, 2016

mrunalp commented Apr 8, 2016

bparees commented Apr 8, 2016

mrunalp commented Apr 8, 2016

bparees commented Apr 13, 2016

mrunalp commented Apr 13, 2016

mrunalp commented Apr 19, 2016

mrunalp commented Apr 26, 2016

mrunalp commented Apr 26, 2016

mrunalp commented May 2, 2016

smarterclayton commented May 11, 2016

mrunalp commented May 11, 2016

bparees commented Jun 22, 2016

bparees commented Jun 30, 2016

mrunalp commented Jul 6, 2016

mrunalp commented Aug 11, 2016

0xmichalis commented Aug 24, 2016

ncdc commented Nov 7, 2016

mrunalp commented Nov 7, 2016

ncdc commented Nov 7, 2016

mrunalp commented Nov 7, 2016

ncdc commented Nov 7, 2016

smarterclayton commented Apr 13, 2017

mrunalp commented Apr 13, 2017 via email

openshift-bot commented Feb 10, 2018

openshift-bot commented Mar 14, 2018

openshift-bot commented Apr 13, 2018

cgroup quota shows up as -1 #8427

cgroup quota shows up as -1 #8427

Comments

bparees commented Apr 8, 2016 • edited by stevekuznetsov Loading

bparees commented Apr 8, 2016

smarterclayton commented Apr 8, 2016

bparees commented Apr 8, 2016

mrunalp commented Apr 8, 2016

mrunalp commented Apr 8, 2016

bparees commented Apr 8, 2016

mrunalp commented Apr 8, 2016

bparees commented Apr 8, 2016

mrunalp commented Apr 8, 2016

bparees commented Apr 13, 2016

mrunalp commented Apr 13, 2016

mrunalp commented Apr 19, 2016

mrunalp commented Apr 26, 2016

mrunalp commented Apr 26, 2016

mrunalp commented May 2, 2016

smarterclayton commented May 11, 2016

mrunalp commented May 11, 2016

bparees commented Jun 22, 2016

bparees commented Jun 30, 2016

mrunalp commented Jul 6, 2016

mrunalp commented Aug 11, 2016

0xmichalis commented Aug 24, 2016

ncdc commented Nov 7, 2016

mrunalp commented Nov 7, 2016

ncdc commented Nov 7, 2016

mrunalp commented Nov 7, 2016

ncdc commented Nov 7, 2016

smarterclayton commented Apr 13, 2017

mrunalp commented Apr 13, 2017 via email

openshift-bot commented Feb 10, 2018

openshift-bot commented Mar 14, 2018

openshift-bot commented Apr 13, 2018

bparees commented Apr 8, 2016 •

edited by stevekuznetsov

Loading