Code

WIP: Smart Node Drain#4005

Code

Closed

dadgar wants to merge 221 commits intomasterhashicorp/nomad:masterfrom f-drain-rebasedhashicorp/nomad:f-drain-rebasedCopy head branch name to clipboard

+6,217-631

Commits

Commits on Feb 21, 2018

• wip - struct updates

  schmichael

  committed
  1a7d557

• wip - deadline checking

  schmichael

  committed
  af6ad54

• wip - node drainer

  schmichael

  committed
  910598a

• wip - refine deadline timer

  schmichael

  committed
  8525673

• wip - adding migration strategy

  schmichael

  committed
  a0094b6

• wip - adding migrate to jobspec

  schmichael

  committed
  fe27fa2

• wip - drainer

  schmichael

  committed
  086ea32

• wip - add migrate support to allow health watcher

  schmichael

  committed
  255df6c

• wip - refactor node drainer state initialization

  schmichael

  committed
  34af62f

• wip - implement node watcher

  schmichael

  committed
  a539782

• wip - mainloop impl up to alloc stopping

  schmichael

  committed
  d627246

• wip - almost finish drainer mainloop

  schmichael

  committed
  ecf773f

• wip - finish initial pass at node drainer **

  schmichael

  committed
  96f3465

• wip - lots more comments from review with alex

  schmichael

  committed
  ef0165a

• wip - add more comments

  schmichael

  committed
  3d65c26

• wip - fix dumb compile errors

  schmichael

  committed
  e6faa75

• wip - one more compiler error

  schmichael

  committed
  dec1b73

• wip - a test that at least runs!

  schmichael

  committed
  9fa07bf

• wip - fix nextDeadline initialization

  schmichael

  committed
  840d993

• wip - fix node watcher initialization

  schmichael

  committed
  cdea7bf

• wip - ignore canceled errors

  schmichael

  committed
  f1feb8d

• wip - always trigger an initial run if there's a node draining

  schmichael

  committed
  538a7c8

• wip - logs logs logs logs logs

  schmichael

  committed
  7719dc3

• wip - test: watch for alloc replacements

  schmichael

  committed
  8565821

• wip - converting test to use test clients

  schmichael

  committed
  694adda

• wip - testlog env var support

  schmichael

  committed
  e3891f2

• wip - stop triggering drains on every node update

  schmichael

  committed
  3fd2422

• wip - start using rpcs in drain test

  schmichael

  committed
  373762d

• wip - fix panic in alloc watcher log line

  schmichael

  committed
  e1693e6

• wip - stop injecting Node because that breaks everything

  schmichael

  committed
  b85ad98

• improve mock driver Kill() logging

  schmichael

  committed
  0e3c445

• wip - drainer: trace log every alloc drain

  schmichael

  committed
  7b3d15f

• server: default to scheduling eligible in Node.Register

  schmichael

  committed
  ef3b326

• server: consider scheduling eligibility in scheduling

  schmichael

  committed
  8fd8a87

• wip - hack default drain strategy into server

  schmichael

  committed
  e6934e5

• wip - trigger drain properly in drain test

  schmichael

  committed
  cc561a4

• wip - log prev alloc in test

  schmichael

  committed
  7ffa70c

Commits on Feb 22, 2018

• Allocation struct

  

  dadgar

  authored and

  schmichael

  committed
  8c46783

• RPC, FSM, State Store for marking DesiredTransistion

  

  dadgar

  authored and

  schmichael

  committed
  9b9a9a9

• Fix tests

  

  dadgar

  authored and

  schmichael

  committed
  3023429

• fix build tag

  

  dadgar

  authored and

  schmichael

  committed
  ad4a415

Commits on Feb 26, 2018

• wip - switch to job based watching

  schmichael

  committed
  01dc0a4

• Test is ugly but it's passing \o/

  schmichael

  committed
  a437737

• transistion -> transition

  schmichael

  committed
  d483f0b

• wip - switch to new raft DesiredTransition message

  schmichael

  committed
  5a00fda

• wip - drain system jobs as well

  schmichael

  committed
  ef8753c

• filterByTainted node should always migrate non-terminal migrating allocs

  schmichael

  committed
  9467e38

• server: unsure we want to enforce having an eval right now

  schmichael

  committed
  a98ed65

• improve drain fsm/statestore tests

  schmichael

  committed
  d446414

Commits on Feb 27, 2018

• Node.Drain takes strategy

  dadgar

  committed
  dfe7dab

• Drain cli, api, http

  dadgar

  committed
  8569c4b

• Fix tests

  dadgar

  committed
  51cff3e

• Remove update time

  dadgar

  committed
  b2a0be6

• Upgrade path

  dadgar

  committed
  8a90a47

• flag comment

  dadgar

  committed
  9947bfd

• Merge pull request #3899 from hashicorp/f-drain-api

  Show description for 5e415b3
  dadgar

  authored
  5e415b3

• RPC/FSM/State Store for Eligibility

  dadgar

  committed
  cdecd22

• HTTP and API

  dadgar

  committed
  185e66b

• node eligibility command

  dadgar

  committed
  e05fb9b

• Add eligibility to node view

  dadgar

  committed
  f40ffa1

• Unblock evals once eligible

  dadgar

  committed
  e17bd34

• Fix retaining the drain

  dadgar

  committed
  e9328c7

• Small refactor and cleanups

  dadgar

  committed
  c2dc67f

• code review

  dadgar

  committed
  ffa6f4c

• Merge pull request #3919 from hashicorp/f-eligibility

  Show description for 8e9bf6b
  dadgar

  authored
  8e9bf6b

• refactor drainer into a subpkg

  schmichael

  committed
  5ffd310

• drainer: drainer should shutdown with server

  schmichael

  committed
  35dc5d4

• Remove unused context

  schmichael

  committed
  7f2d8fa

• Restart every time SetEnabled(true) is called

  schmichael

  committed
  6e5b7fc

• drainer: factor job & node watchers out of drainer.go

  schmichael

  committed
  fc13aa7

• drainer: convert fsm errors to go errors

  schmichael

  committed
  984acf9

Commits on Mar 1, 2018

• Improve DeadlineTime helper

  dadgar

  committed
  bda7ec7

• Merge pull request #3922 from hashicorp/f-deadline

  Show description for aaf392a
  dadgar

  authored
  aaf392a

• client: don't monitor health of non-service jobs

  schmichael

  committed
  95b9183

• wip - fix system job draining; won't work without deadline fixes

  schmichael

  committed
  4ed2a5a

• drainer: factor out newStopAllocs and applyMigrations

  schmichael

  committed
  37f5869

• small cleanups

  schmichael

  committed
  f5d57f8

• refactor main drainloop into 2 more methods

  schmichael

  committed
  80ba859

• Correct defaulting

  dadgar

  committed
  d822367

• Merge pull request #3925 from hashicorp/b-defaulting

  Show description for 1a96c25
  dadgar

  authored
  1a96c25

• Fix file names

  dadgar

  committed
  2318e1e

Commits on Mar 2, 2018

• System test runs on mac

  dadgar

  committed
  0df7acc

• Initial design

  dadgar

  committed
  2938045

Commits on Mar 3, 2018

• drain heap

  dadgar

  committed
  f3ac2b4

• node watcher

  dadgar

  committed
  44f1cc5

• Node's being untracked or having updated deadlines, updates the deadliner

  dadgar

  committed
  90da01e

Commits on Mar 6, 2018

• job watcher

  dadgar

  committed
  ea4df19

• Drainer

  dadgar

  committed
  1e7186f

Commits on Mar 7, 2018

• integration test and basic fixes

  dadgar

  committed
  eebca69

• handle empty node case

  dadgar

  committed
  83d3555

Commits on Mar 8, 2018

• Comments

  dadgar

  committed
  5abffa0

• spelling fixes

  dadgar

  committed
  3e83591

• code review

  dadgar

  committed
  ef6a003

• Merge pull request #3951 from hashicorp/f-drainer

  Show description for 0db7e8d
  dadgar

  authored
  0db7e8d

• Toggle Drain allows resetting eligibility

  Show description for 309bcf8
  dadgar

  committed
  309bcf8

• Merge pull request #3955 from hashicorp/f-manual-undrain

  Show description for 5efde3e
  dadgar

  authored
  5efde3e

• Switch to drainerv2 impl

  schmichael

  committed
  88c9d70

Commits on Mar 9, 2018

• Batch drain update

  dadgar

  committed
  8f4ce6e

Commits on Mar 10, 2018

• sharding

  dadgar

  committed
  ffde29d

Commits on Mar 12, 2018

• Fix deadline handling

  dadgar

  committed
  0e86233

• Merge pull request #3967 from hashicorp/b-deadline

  Show description for 51bd58a
  dadgar

  authored
  51bd58a

Commits on Mar 14, 2018

• Fix node eligibility test

  schmichael

  committed
  a77c452

• Add DesiredTransition.ShouldMigrate to api pkg

  schmichael

  committed
  8b7b42c

• Monitor node drains until completion in CLI

  Show description for 14d76fc
  schmichael

  committed
  14d76fc

• fix comment

  dadgar

  committed
  8506049

• Merge pull request #3960 from hashicorp/f-sharding

  Show description for f545e6a
  dadgar

  authored
  f545e6a

Commits on Mar 16, 2018

• Improve drain log messages

  Show description for 5c4ffd4
  schmichael

  committed
  5c4ffd4

• Merge pull request #3948 from hashicorp/f-drainv2-cli

  Show description for b2d682b
  schmichael

  authored
  b2d682b

• Fix deadline heap triggering

  Show description for 6c5d442
  schmichael

  committed
  6c5d442

• Fix job watcher index handling

  schmichael

  committed
  3429b7d

• wip test for job watcher

  schmichael

  committed
  c1ee664

• Revert making drains async

  Show description for 5733c7c
  schmichael

  committed
  5733c7c

• Test job watcher

  schmichael

  committed
  d18d2d7

Commits on Mar 19, 2018

• wip - struct updates

  

  schmichael

  authored and

  dadgar

  committed
  adc979e

• wip - deadline checking

  

  schmichael

  authored and

  dadgar

  committed
  84ab471

• wip - node drainer

  

  schmichael

  authored and

  dadgar

  committed
  c67be08

• wip - refine deadline timer

  

  schmichael

  authored and

  dadgar

  committed
  85eb9a1

• wip - adding migration strategy

  

  schmichael

  authored and

  dadgar

  committed
  4533b03

• wip - adding migrate to jobspec

  

  schmichael

  authored and

  dadgar

  committed
  07225eb

• wip - drainer

  

  schmichael

  authored and

  dadgar

  committed
  540a696

• wip - add migrate support to allow health watcher

  

  schmichael

  authored and

  dadgar

  committed
  b57fb51

• wip - refactor node drainer state initialization

  

  schmichael

  authored and

  dadgar

  committed
  e9c63b9

• wip - implement node watcher

  

  schmichael

  authored and

  dadgar

  committed
  590f25a

• wip - mainloop impl up to alloc stopping

  

  schmichael

  authored and

  dadgar

  committed
  b4b71a4

• wip - almost finish drainer mainloop

  

  schmichael

  authored and

  dadgar

  committed
  a6cc4dc

• wip - finish initial pass at node drainer **

  

  schmichael

  authored and

  dadgar

  committed
  6057ed5

• wip - lots more comments from review with alex

  

  schmichael

  authored and

  dadgar

  committed
  518aa5f

• wip - add more comments

  

  schmichael

  authored and

  dadgar

  committed
  0662421

• wip - fix dumb compile errors

  

  schmichael

  authored and

  dadgar

  committed
  4a9c6cb

• wip - one more compiler error

  

  schmichael

  authored and

  dadgar

  committed
  d061d89

• wip - a test that at least runs!

  

  schmichael

  authored and

  dadgar

  committed
  47f7d70

• wip - fix nextDeadline initialization

  

  schmichael

  authored and

  dadgar

  committed
  151f2dc

• wip - fix node watcher initialization

  

  schmichael

  authored and

  dadgar

  committed
  e0643d4

• wip - ignore canceled errors

  

  schmichael

  authored and

  dadgar

  committed
  4533b67

• wip - always trigger an initial run if there's a node draining

  

  schmichael

  authored and

  dadgar

  committed
  152f456

• wip - logs logs logs logs logs

  

  schmichael

  authored and

  dadgar

  committed
  1663428

• wip - test: watch for alloc replacements

  

  schmichael

  authored and

  dadgar

  committed
  f493d52

• wip - converting test to use test clients

  

  schmichael

  authored and

  dadgar

  committed
  910e7c8

• wip - testlog env var support

  

  schmichael

  authored and

  dadgar

  committed
  c97c2e4

• wip - stop triggering drains on every node update

  

  schmichael

  authored and

  dadgar

  committed
  8ff951c

• wip - start using rpcs in drain test

  

  schmichael

  authored and

  dadgar

  committed
  bd3bfa5

• wip - fix panic in alloc watcher log line

  

  schmichael

  authored and

  dadgar

  committed
  0ea4f9f

• wip - stop injecting Node because that breaks everything

  

  schmichael

  authored and

  dadgar

  committed
  db419ec

• improve mock driver Kill() logging

  

  schmichael

  authored and

  dadgar

  committed
  1d25f33

• wip - drainer: trace log every alloc drain

  

  schmichael

  authored and

  dadgar

  committed
  3914180

• server: default to scheduling eligible in Node.Register

  

  schmichael

  authored and

  dadgar

  committed
  d8615e7

• server: consider scheduling eligibility in scheduling

  

  schmichael

  authored and

  dadgar

  committed
  9a36570

• wip - hack default drain strategy into server

  

  schmichael

  authored and

  dadgar

  committed
  9e0f0eb

• wip - trigger drain properly in drain test

  

  schmichael

  authored and

  dadgar

  committed
  ad13507

• wip - log prev alloc in test

  

  schmichael

  authored and

  dadgar

  committed
  6e7227b

• Allocation struct

  dadgar

  committed
  965c57b

• RPC, FSM, State Store for marking DesiredTransistion

  dadgar

  committed
  632fbd7

• Fix tests

  dadgar

  committed
  75bf315

• fix build tag

  dadgar

  committed
  1a0bfbc

• wip - switch to job based watching

  

  schmichael

  authored and

  dadgar

  committed
  95aa121

• Test is ugly but it's passing \o/

  

  schmichael

  authored and

  dadgar

  committed
  ab9df29

• wip - switch to new raft DesiredTransition message

  

  schmichael

  authored and

  dadgar

  committed
  db048bd

• wip - drain system jobs as well

  

  schmichael

  authored and

  dadgar

  committed
  fc217d7

• filterByTainted node should always migrate non-terminal migrating allocs

  

  schmichael

  authored and

  dadgar

  committed
  cde47ff

• server: unsure we want to enforce having an eval right now

  

  schmichael

  authored and

  dadgar

  committed
  7cea5ee

• improve drain fsm/statestore tests

  

  schmichael

  authored and

  dadgar

  committed
  6b80cd3

• Node.Drain takes strategy

  dadgar

  committed
  240d50b

• Drain cli, api, http

  dadgar

  committed
  a1466cf

• Fix tests

  dadgar

  committed
  7e0317d

• Remove update time

  dadgar

  committed
  e675257

• Upgrade path

  dadgar

  committed
  e18cea5

• flag comment

  dadgar

  committed
  c901b21

• JobNs -> NamespacedID

  Show description for b02fd1e
  schmichael

  committed
  b02fd1e

• Deregister garbage collected jobs

  schmichael

  committed
  f3a1752

• Remove debug prints

  schmichael

  committed
  1739290

• RPC/FSM/State Store for Eligibility

  dadgar

  committed
  43d5246

• HTTP and API

  dadgar

  committed
  1c0aaa7

• node eligibility command

  dadgar

  committed
  a73398d

• Add eligibility to node view

  dadgar

  committed
  4915b76

• Unblock evals once eligible

  dadgar

  committed
  45db5ca

• Fix retaining the drain

  dadgar

  committed
  9db4f7d

• Small refactor and cleanups

  dadgar

  committed
  ab3fc5b

• code review

  dadgar

  committed
  b807fdc

• refactor drainer into a subpkg

  

  schmichael

  authored and

  dadgar

  committed
  3f2d7f1

• drainer: drainer should shutdown with server

  

  schmichael

  authored and

  dadgar

  committed
  b93142c

• Remove unused context

  

  schmichael

  authored and

  dadgar

  committed
  10a3d38

• Restart every time SetEnabled(true) is called

  

  schmichael

  authored and

  dadgar

  committed
  d78da97

• drainer: factor job & node watchers out of drainer.go

  

  schmichael

  authored and

  dadgar

  committed
  1753afa

• drainer: convert fsm errors to go errors

  

  schmichael

  authored and

  dadgar

  committed
  7da9b61

• Improve DeadlineTime helper

  dadgar

  committed
  e3dff57

• client: don't monitor health of non-service jobs

  

  schmichael

  authored and

  dadgar

  committed
  d58a50b

• wip - fix system job draining; won't work without deadline fixes

  

  schmichael

  authored and

  dadgar

  committed
  4d54ac1

• drainer: factor out newStopAllocs and applyMigrations

  

  schmichael

  authored and

  dadgar

  committed
  6c84ecd

• small cleanups

  

  schmichael

  authored and

  dadgar

  committed
  c61db41

• refactor main drainloop into 2 more methods

  

  schmichael

  authored and

  dadgar

  committed
  3a7d297

• Correct defaulting

  dadgar

  committed
  cd80e91

• Fix file names

  dadgar

  committed
  1304311

• System test runs on mac

  dadgar

  committed
  7fde810

• Initial design

  dadgar

  committed
  e376f0b

• drain heap

  dadgar

  committed
  4f88361

• node watcher

  dadgar

  committed
  65edbb8

• Node's being untracked or having updated deadlines, updates the deadliner

  dadgar

  committed
  58e3b35

• job watcher

  dadgar

  committed
  2ed8896

• Drainer

  dadgar

  committed
  3a82359

• integration test and basic fixes

  dadgar

  committed
  ba19415

• handle empty node case

  dadgar

  committed
  4e96ac4

• Comments

  dadgar

  committed
  2536ba9

• spelling fixes

  dadgar

  committed
  7fba651

• code review

  dadgar

  committed
  0e7b37d

• Toggle Drain allows resetting eligibility

  Show description for 6321a27
  dadgar

  committed
  6321a27

• Switch to drainerv2 impl

  

  schmichael

  authored and

  dadgar

  committed
  6eef768

• Batch drain update

  dadgar

  committed
  da06b87

• sharding

  dadgar

  committed
  3085f58

• Refactor assertOps into a helper func

  schmichael

  committed
  8b1fb9f

• Merge pull request #3961 from hashicorp/f-drainv2-testing

  Show description for b4e5979
  schmichael

  authored
  b4e5979

• Fix deadline handling

  dadgar

  committed
  5ed3c7b

• Fix node eligibility test

  

  schmichael

  authored and

  dadgar

  committed
  c4158c0

• Add DesiredTransition.ShouldMigrate to api pkg

  

  schmichael

  authored and

  dadgar

  committed
  91b73cc

• Monitor node drains until completion in CLI

  Show description for 1c5aa73
  

  schmichael

  authored and

  dadgar

  committed
  1c5aa73

• fix comment

  dadgar

  committed
  e987350

• Improve drain log messages

  Show description for 4639dc7
  

  schmichael

  authored and

  dadgar

  committed
  4639dc7

• Merge branch 'f-drainv2-node-drainer' of github.com:hashicorp/nomad into f-drainv2-node-drainer

  dadgar

  committed
  46de654

• merge cleanup

  dadgar

  committed
  f902c88

• Merge branch 'master' into f-drain-rebased

  dadgar

  authored
  6b732eb

• fix race in drain integration tests

  dadgar

  committed
  0482280

• Merge branch 'master' into f-drain-rebased

  dadgar

  committed
  59f3df9