Skip to content

Latest commit

 

History

History
594 lines (577 loc) · 46.9 KB

BENCHMARK.md

File metadata and controls

594 lines (577 loc) · 46.9 KB

Mojito benchmarks

These benchmarks were conducted using gamache/httpc_bench.

GET requests lasting 10 ms are performed.

Clients ran on AWS m5.4xlarge (16 vCPUs, 64 GB RAM) running Ubuntu 18.04.

The server ran on a separate m5.4xlarge, configured identically.

Clients:

  • Mojito 0.3.0 (Mint 0.2.1)
  • Buoy b65c06f
  • MachineGun 0.1.5 (Gun 1.3.0)
  • Dlhttpc 1072652
  • Hackney 1.15.1
  • Ibrowse 4.4.1
  • Httpc (OTP 21.2)
  • Mint 0.2.1, without pooling or keepalive, as a reference client.

Current results (2019-05-15)

This benchmark was run after optimizing the servers for high-concurrency tasks by executing ulimit -n 12000000 to increase the allowable number of open files. This configuration is very different from the default ulimit -n 1024 on Ubuntu systems.

The enhanced configuration showed that Buoy and Gun really shine when they are allowed to stretch their legs. Mojito performs well but has catch-up work to do! Likely areas of improvement are in pipelining (we use none, currently) and concurrency-friendly connection checkout in a similar manner to Shackle or Dispcount.

ClientPool CountPool SizeConcurrencyReq/secError %
ClientPool CountPool SizeConcurrencyReq/secError %
Buoy11024409610128079.6
Buoy1102481929702979.7
Buoy1102420489627279.5
Buoy11024163848567279.6
Buoy151240968434589.8
Buoy151220487875789.8
Buoy151281927794590
MachineGun110242048729500
Buoy1512163847224790
MachineGun110244096721080
MachineGun110241024712190
Buoy125640966233294.9
Mojito42564096620680.8
MachineGun110248192612051
Mojito81284096593540.8
Mojito42568192593530.8
Buoy1102410245927379.4
Buoy125620485906994.9
Mojito81288192587070.8
Mojito82568192580660.8
Mojito812816384576900.8
Mojito16642048574680.8
Mojito166416384570980.8
Mojito161288192567120.8
Mojito425616384566900.8
Mojito16648192566450.8
Mojito82564096563250.9
Mojito825616384561980.8
Buoy125681925599395.1
Mojito165124096556330.6
Mojito1610242048552720.7
Mojito45124096551310.9
MachineGun1102416384550552.9
Mojito32321024549930.8
Mojito32328192549430.8
Mojito323216384545000.8
Mojito1610244096543810.8
Mojito32648192542400.8
Mojito1612816384538820.9
Mojito110241024538481.2
Mojito45122048538140.8
Mojito16644096537311
Mojito161284096537020.9
Mojito325124096536220.6
Mojito161282048536100.9
Mojito85124096534880.7
Mojito32322048534510.8
Mojito810242048532830.9
Mojito110242048530911.5
Mojito321282048530060.7
Mojito162564096528440.9
Mojito410244096527890.8
Mojito162562048526500.8
Mojito326416384525110.9
Mojito110244096522970.9
Mojito32642048522470.8
Mojito410241024522351.2
Mojito32644096521690.9
Mojito321284096520160.9
Buoy1256163845178595.1
Mojito810244096517800.7
Mojito82562048517410.9
Mojito410242048516270.8
Mojito45128192513880.9
Mojito325122048513870.8
Mojito85122048513780.9
Mojito85128192510080.8
Mojito322562048508340.8
Mojito165122048506300.8
Mojito810241024502041.3
Mojito42562048501081.3
Mojito325121024500341.1
Mojito32641024499680.9
Mojito3225616384499661.8
Buoy151210244970989.8
Mojito42561024496351.5
Mojito162561024496131
Mojito1625616384493301.1
Mojito85121024492641
Mojito32324096492090.9
Mojito322564096491640.8
Mojito81282048491521.2
Mojito321288192487320.9
Mojito162568192483950.9
Mojito81281024483491.5
Mojito45121024481001.2
Mojito3212816384480671.1
Mojito161281024478591.3
Mojito165121024475761.3
Mojito110248192475730.8
Mojito82561024472181.2
Mojito322568192468451.7
Mojito165128192468111.4
Mojito1610241024467211.3
Mojito410248192461951.3
Mojito1651216384461901.6
Mojito321281024453861.2
Mojito322561024451151.2
Mojito16641024450821.4
Mojito451216384443981.2
Mojito8102416384438902.1
MachineGun15121024435290
MachineGun15122048434910
Mojito41284096434760.9
Mojito8648192433270.9
Mojito86416384433000.9
Mojito41288192432400.9
Mojito41282048430670.9
Mojito16324096429780.9
Mojito16328192429360.9
Mojito412816384428920.9
Mojito163216384428580.9
Mojito325128192428171.4
Mojito8642048427180.9
Mojito8644096426620.9
Mojito41281024425560.9
Mojito15122048424970.9
Mojito16322048423670.9
MachineGun15124096423620
Mojito15124096421870.9
Mojito8641024419790.9
Mojito16321024418760.9
Buoy112840964049297.5
Buoy125610244014894.9
MachineGun15128192394881.1
Mojito15121024393621
Mojito15128192377460.9
MachineGun151216384376133.1
Buoy112820483750797.5
Mojito3210248192369270.7
Buoy112881923647697.5
Mojito1610248192339140.8
Ibrowse1328192338474.4
Mojito3251216384335542.2
Ibrowse1324096330023.9
Ibrowse1322048319582.9
Buoy1128163843195697.6
Mojito3210244096312893.1
Buoy112810243095697.4
Mojito810248192309552.4
Mojito151216384302965.2
Ibrowse13216384300914.9
Mojito1102416384285468.6
Dlhttpc1102410242844370.5
Mojito16102416384279332.4
Mojito851216384277612.2
Ibrowse1321024275890.1
Ibrowse1644096269420.2
Ibrowse1642048263860.4
Ibrowse1648192261330.7
Mojito3210242048259694.4
Mojito32102416384249212.8
Ibrowse16416384246180.4
Ibrowse11284096235370
Dlhttpc1102481922352895.2
Ibrowse1641024233080
MachineGun12561024231340.1
MachineGun12562048230580.1
Mojito4102416384229084.3
MachineGun12564096228160.1
Buoy16440962278298.7
Mojito4644096227591
Mojito12561024227021
Mojito46416384226661
Mojito12562048226371
Mojito8328192226201
Mojito12564096226041
Mojito4648192225981
Mojito83216384224691
Mojito8324096224691
Ibrowse11282048224690.1
Mojito4642048224631
Mojito8322048223841
Ibrowse11288192223300
Hackney12561024223160
Mojito4641024222961
Mojito8321024222391
MachineGun12568192220631
Buoy16420482194198.7
Hackney12562048216940
MachineGun125616384214793.3
Ibrowse112816384214370.1
Mojito12568192212201.1
Buoy16481922112498.7
Hackney12564096210310
Ibrowse11281024203330
Buoy164163842024498.8
Dlhttpc151220481996494.9
Buoy16410241959798.7
Hackney12568192190850
Mojito125616384190675.8
Hackney15121024186430
Ibrowse12564096186260
Dlhttpc11024163841851396.5
Hackney15124096181330
Ibrowse12562048179550
Ibrowse12568192179250
Hackney125616384176330
Dlhttpc1102420481757187.6
Hackney15122048175380
Ibrowse125616384171260
Dlhttpc1102440961695792.6
Ibrowse12561024169480
Mojito43216384158056.2
Mojito3210241024155118.5
Hackney15128192152650
Dlhttpc151281921369997.6
Ibrowse15124096136980
Mojito1128163841328514.7
Ibrowse15128192131450
Ibrowse15122048130780
Ibrowse151216384127050
Hackney151216384125790
Buoy13220481249299.4
Ibrowse15121024123690
Buoy13281921165799.4
MachineGun11281024116150.1
MachineGun11282048116060.1
Mojito11281024116031
Mojito11282048115961
Hackney110242048115730
Mojito4324096115711
MachineGun11284096115700.1
Mojito11284096115681
Hackney110241024115660
Mojito4322048115561
Mojito4328192115501
Mojito4321024115171
Hackney11281024114650
Buoy13240961146399.4
Buoy132163841144399.4
Hackney11282048114380
MachineGun11288192114320.9
Hackney11284096114300
Mojito11288192112491.3
Hackney11288192111520
Buoy13210241095699.4
Hackney110244096107250
Hackney112816384105440
Dlhttpc125620481027097.7
Hackney11024819296620
Dlhttpc12561024911496.8
Hackney110241638485510
Dlhttpc15121024839389.9
Ibrowse11024409683650
Dlhttpc15124096831995.9
Ibrowse11024204881490
Ibrowse11024819281210
Mojito1648192799315.3
Dlhttpc12568192796698.7
Ibrowse110241638479620
Ibrowse11024102478280
Dlhttpc151216384696898.2
Dlhttpc125616384672299
MachineGun164102458100.1
MachineGun164409658070.1
MachineGun164204858070.1
Mojito164204858061
Mojito164102458061
Mojito164409658051
Hackney164102457910
Hackney164204857870
Hackney164409657810
Dlhttpc11281024575398.6
Hackney164819257460
Httpc132102457420
Hackney1641638456820
Dlhttpc11282048568099
Httpc11024204855710
Httpc1256102455680
Mojito1324096556217.4
Httpc164102455600
Httpc1128102455430
Httpc1256409655320
Httpc1512204854880
Httpc1512102454670
Httpc1128409654080
Httpc11024409653970
Httpc11024102453400
Httpc1256204852630
Httpc132204852310
Httpc164204851860
Dlhttpc11284096518499.2
Httpc1128204851430
Httpc132819250820
Httpc132409650310
Httpc1256819250130
Httpc164819250010
Httpc1512409649980
Mojito1328192499131.8
Httpc1512819249730
Httpc164409649710
Httpc11024819247180
Httpc1641638445900
Httpc1128819245400
Httpc12561638445250
Httpc1321638444270
Httpc110241638444080
Dlhttpc11288192436499.4
Httpc11281638442980
Httpc15121638442680
Dlhttpc112816384369399.5
Mint11204834742
Mint11102434402
Dlhttpc12564096337498.4
Mint11819233266.6
Dlhttpc1641024317899.4
Dlhttpc1642048317099.5
Mint11409631148.7
MachineGun132204829060.1
Mojito132204829031
Mojito132102429031
Hackney132102428980
Hackney132409628970
Hackney132204828970
Hackney132819228960
MachineGun132102428910
Hackney1321638428850
Dlhttpc1644096262299.6
MachineGun1324096236242.2
Dlhttpc1648192215599.7
Dlhttpc16416384209199.7
MachineGun1648192187177.1
Dlhttpc1322048168199.8
Mint111638416580
Dlhttpc1321024160699.7
Dlhttpc1324096141199.8
Dlhttpc1328192120999.9
Dlhttpc13216384102299.9
MachineGun11281638446797.1
MachineGun1641638419698.8
Mojito164163849698.8
MachineGun132163849699.4
MachineGun13281924899.4
Mojito132163844799.4

The original benchmark

With server specs as above, except using Ubuntu's default ulimit -n 1024. This benchmark rightly showed that Mojito is fast, but wrongly showed it being faster than Buoy. It does tell us that Mojito excels in resource-constrained environments, or where configuring custom ulimits is not possible.

ClientPool CountPool SizeConcurrencyReq/secError %
ClientPool CountPool SizeConcurrencyReq/secError %
Mojito42562048525191.2
Mojito81284096521051.1
Mojito81282048516211
Mojito812816384515380.9
Mojito81288192512871
Mojito42564096511161.2
Mojito16642048510641.1
Mojito32324096509300.9
Mojito16648192508481
Mojito16644096503360.9
Mojito166416384502941
Mojito42568192491341.1
Mojito323216384484701
Mojito32328192481030.9
Mojito32322048479141
Mojito425616384468661.2
Mojito8648192435310.9
Mojito15122048434370.9
Mojito41288192432240.9
Mojito41282048431830.9
Mojito412816384431280.9
Mojito41284096428450.9
Mojito86416384425370.9
Mojito15124096422550.9
Mojito8642048420200.9
Mojito8644096419000.9
Mojito16324096418420.9
Mojito163216384414800.9
Mojito16328192413180.9
Mojito16322048407020.9
Mojito15128192374071
Buoy151281923683389.8
Buoy1512163843494289.6
Buoy151240963441789.8
Buoy151220483188989.8
Mojito151216384308705.6
Mojito45122048269242
Mojito161282048244941.7
Mojito32642048242101.4
Mojito46416384229231
Mojito12562048228431
Mojito8328192227451
Mojito4642048227271
Mojito4644096226981
Mojito4648192226771
Mojito8324096226771
Mojito83216384226441
Mojito12564096225881
Mojito8322048222821
Hackney12562048218850
Mojito12568192213361.1
Hackney12564096212260
Mojito326416384210548.8
Mojito16128163841961811.6
Dlhttpc151220481949395.1
Mojito125616384193105.4
Hackney12568192190330
Buoy125640961869294.9
Buoy125681921853194.9
Mojito825616384184097.2
Buoy1256163841805294.8
Ibrowse12562048176550
Ibrowse12564096176260
Hackney15122048176190
Hackney125616384172720
Mojito45128192171579.5
Mojito82564096170224.8
Ibrowse12568192170070
Buoy125620481697394.9
Hackney15124096168210
Ibrowse125616384165270
Mojito161284096156363.9
Mojito82562048154591.6
Mojito85122048153852
Mojito165122048153341.8
Mojito82568192153329.7
Mojito45124096153185.1
Mojito32644096151796.2
Hackney15128192149540
Dlhttpc151281921488497.3
Mojito32648192148828.5
Mojito162562048147941.9
Mojito325122048147801.9
Mojito1612881921467310.6
Mojito4512163841463814.8
Ibrowse15122048133180
Ibrowse15124096132910
Hackney151216384130610
Ibrowse15128192127870
Mojito321282048123152.6
Ibrowse151216384122710
Mojito322562048120502.6
Mojito321284096116344.6
Mojito11282048116021
Mojito4322048115831
Ibrowse11282048115670.1
Mojito11284096115521
Mojito4324096115401
Dlhttpc1512163841149398.1
Hackney11282048114910
Ibrowse11284096114880
Mojito32128163841146219.3
Hackney11284096114180
Mojito3212881921136110.8
Mojito1625681921127611.8
Mojito11288192112601.3
Ibrowse11288192112320
Hackney11288192111510
Mojito432163841101548.2
Mojito4328192108551.7
Dlhttpc125620481073797.7
Ibrowse112816384106553.8
Hackney112816384105900
Dlhttpc125640961018198.3
Mojito162564096101624.3
Mojito8512163841015314.6
Mojito32256409698524.9
Mojito16512409696735.7
Buoy11284096962697.4
Mojito8512409696177.9
Mojito85128192951811.3
Mojito112816384942547.4
Buoy11288192940397.4
Buoy11282048903697.4
Buoy112816384862497.6
Mojito1625616384852431.6
Dlhttpc12568192811098.7
Dlhttpc15124096799896.3
Mojito325124096753816.2
Dlhttpc125616384715298.8
Mojito165128192669016.5
Mojito322568192654712.7
Mojito1651216384646717.6
Mojito3225616384623421.8
Mojito164204858101
Mojito164409658091
Ibrowse164409658050
Hackney164204857980
Hackney164409657910
Ibrowse164204857910
Ibrowse164819257883.4
Hackney164819257580
Ibrowse16416384575630
Mojito325128192571524.4
Httpc164204856980
Httpc1128204856910
Hackney1641638456730
Httpc1256204855060
Httpc132409654640
Httpc1512204854380
Httpc1512409654360
Httpc1128409653700
Dlhttpc11282048531999
Httpc164409653000
Httpc132204852650
Httpc1128819251730
Dlhttpc11284096517099.2
Httpc1256409650260
Httpc164819248850
Httpc1512819248840
Mojito1648192487849.6
Httpc11281638448280
Buoy1642048479398.7
Buoy1644096477498.7
Httpc1321638445640
Mojito3251216384452519
Httpc15121638445140
Httpc1256819244800
Buoy1648192442798.8
Httpc132819244210
Httpc12561638443670
Buoy16416384432198.8
Dlhttpc11288192412299.4
Httpc1641638440870
Dlhttpc112816384393499.4
Mojito132204829051
Hackney132204829030
Hackney132409629020
Ibrowse13216384290155.1
Ibrowse132409629003.8
Hackney132819228990
Ibrowse1328192289231.5
Ibrowse132204828870.1
Hackney1321638428830
Dlhttpc1642048286099.5
Dlhttpc1644096274599.6
Mojito1324096247759.5
Buoy1322048238699.4
Dlhttpc1648192228299.7
Buoy1324096220399.4
Buoy1328192216099.4
Buoy13216384214499.4
Dlhttpc16416384207299.7
Mojito1328192189774.2
Dlhttpc1322048159299.8
Dlhttpc1324096150099.8
Dlhttpc1328192119599.8
Dlhttpc13216384111999.8
Mint111638425479
Mint11819222751.9
Mint11409620635
Mojito164163849798.8
Mojito132163844799.4