#77 autogomaxprocs

[jochen42]

adding the blank import "go.uber.org/automaxprocs" to app.go ensures the correct gomaxprocs settings in container-environments with cgroups.

fixes #77

[danielpoe]

Hey jochen - thanks looks good.
Would be perfect if we can measure some differences to see if the change has impact..

[jochen42]

hey @danielpoe,

it would take quite a long time to create an test/messurement with an real-world-impact as an result.
Some other guy's already did some experiments. e.g:

• Add benchmarks to README? uber-go/automaxprocs#12 (comment)
• resources.cpu.requests + GOMAXPROCS prometheus-operator/prometheus-operator#501 (comment)

But since the results are depending on many different factors (total number of host-cores, configured quoata, count of containers on the host) they don't help us to estimate a real-world impact.

But i will try to explain the technical background and i think i will get clear, that the positive impact will grow, with count of cpu's and the containers on the host-machine.

If we using docker as the container engine with [cpu quota's] (https://docs.docker.com/config/containers/resource_constraints/#cpu) (kubernetes resource limits are resulting in cpu quota's). The docker runtime (on posix-system's) translates them into cgroup-mounts per container. With these cgroups the linux [cfs-scheduler] (https://developer.ibm.com/tutorials/l-completely-fair-scheduler/) will limit the cpu-time for the docker-container-process and it's childs.
If the container requests to much cpu-time, cfs will throttle those requests -> the process has to wait for computing time -> the process stucks.

If i understand the go scheduler correctly, it distributes the requested go-routines accross os-threads. the maximum amount of threads is limited by GOMAXPROCS. Since one thread can utilize exactly one logical cpu-core and go-routines handles io-wait realy smart, it will utilize each created thread nearly 100%. Now the cpu-quota reduces the amount of available cpu-time via cfs-scheduler and the most cpu-time requests by scheduled go-routines will be throttled.

The included library read's the cfs values of the started process, calculates a virtual amount of cpu-cores and floors it.
Something like this: GOMAXPROCS=Math.floor(cpu.cfs_quota_us / cpu.cfs_period_us). Now the scheduled go-routines schould'nt be throttled anymore.
It will help to reduce cfs throtteling on the host machine.

here a small example which shows the increased cpu-throtteling, if GOMAXPROCS is higher than cpu-limit:

some more informations

• articles about the go scheduler
  • https://rakyll.org/scheduler/
  • https://medium.com/@riteeksrivastava/a-complete-journey-with-goroutines-8472630c7f5c
• A nice talk about kubernetes resource-management with many details of cgroups and cfs-scheduler:
  https://www.youtube.com/watch?v=8-apJyr2gi0

[jochen42]

ok. i did an small demo with two tests inside the same docker-environment (cpus=2) on the same macbook (with physical 4-cores (8 logical)).

the first test is with the current stable flamingo core and the second with the branch of this pr.
both with ab -n 2048 -c 32 "http://localhost:3322/"

the automaxrproc-version was around 2 seconds faster and reached 19 requests/sec more.
additionally the cpu throttle time was reduced.

have a detailed look here:

used docker-compose.yaml

version: '2.4'

services:
  flamingo:
    image: golang:alpine
    working_dir: /app/flamingo-eample-helloworld
    cpus: '2.0'
    environment:
      GOPATH: /app/flamingo-eample-helloworld/docker/gopath
    ports:
      - 3322:3322
      - 12310:12310
    volumes:
      - ../:/app:delegated
    command:
      - go
      - run
      - main.go
      - serve

with current stable (flamingo.me/flamingo/v3 v3.0.3)

> ab -n 2048 -c 32 "http://localhost:3322/"
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 204 requests
Completed 408 requests
Completed 612 requests
Completed 816 requests
Completed 1020 requests
Completed 1224 requests
Completed 1428 requests
Completed 1632 requests
Completed 1836 requests
Completed 2040 requests
Finished 2048 requests


Server Software:
Server Hostname:        localhost
Server Port:            3322

Document Path:          /
Document Length:        156 bytes

Concurrency Level:      32
Time taken for tests:   15.590 seconds
Complete requests:      2048
Failed requests:        1
   (Connect: 0, Receive: 0, Length: 1, Exceptions: 0)
Non-2xx responses:      1
Total transferred:      560493 bytes
HTML transferred:       320857 bytes
Requests per second:    131.37 [#/sec] (mean)
Time per request:       243.589 [ms] (mean)
Time per request:       7.612 [ms] (mean, across all concurrent requests)
Transfer rate:          35.11 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:    23  241  19.5    243     492
Waiting:       23  241  19.5    243     492
Total:         24  241  19.5    243     492

Percentage of the requests served within a certain time (ms)
  50%    243
  66%    247
  75%    250
  80%    252
  90%    256
  95%    261
  98%    265
  99%    270
 100%    492 (longest request)

cpu stats

> docker-compose exec flamingo cat /sys/fs/cgroup/cpu/cpu.stat          (master|✚5…)
nr_periods 316
nr_throttled 118
throttled_time 21388297980

with integrated automaxprocs (https://github.com/jochen42/flamingo/commit/572148dc5616dc1ffa52258e777d43df34c1de87)

first line of stdout

> docker-compose exec flamingo cat /sys/fs/cgroup/cpu/cpu.stat
flamingo_1  | 2019/10/12 21:54:04 maxprocs: Updating GOMAXPROCS=2: determined from CPU quota

test result

ab -n 2048 -c 32 "http://localhost:3322/" 
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 204 requests
Completed 408 requests
Completed 612 requests
Completed 816 requests
Completed 1020 requests
Completed 1224 requests
Completed 1428 requests
Completed 1632 requests
Completed 1836 requests
Completed 2040 requests
Finished 2048 requests


Server Software:
Server Hostname:        localhost
Server Port:            3322

Document Path:          /
Document Length:        156 bytes

Concurrency Level:      32
Time taken for tests:   13.683 seconds
Complete requests:      2048
Failed requests:        2
   (Connect: 0, Receive: 0, Length: 2, Exceptions: 0)
Non-2xx responses:      2
Total transferred:      561030 bytes
HTML transferred:       321374 bytes
Requests per second:    149.67 [#/sec] (mean)
Time per request:       213.801 [ms] (mean)
Time per request:       6.681 [ms] (mean, across all concurrent requests)
Transfer rate:          40.04 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       2
Processing:    14  210  20.1    211     414
Waiting:       14  210  20.1    211     414
Total:         16  210  20.0    211     415

Percentage of the requests served within a certain time (ms)
  50%    211
  66%    216
  75%    220
  80%    222
  90%    228
  95%    234
  98%    240
  99%    247
 100%    415 (longest request)

cpu stats

> docker-compose exec flamingo cat /sys/fs/cgroup/cpu/cpu.stat
nr_periods 288
nr_throttled 116
throttled_time 20860882093

[danielpoe]

Thanks for the effort and the adjustments.
Appart from the Logcomment looks fine to me

[danielpoe]

Fine for me - @thebod any comments?