manual backport docker test fixes to 1.2.x #13117

…ix/hardly-selected-jaguar This pull request was automerged via backport-assistant

…ions/admittedly-merry-hermit This pull request was automerged via backport-assistant

…apidly-clever-fowl This pull request was automerged via backport-assistant

…ications/internally-rapid-mole This pull request was automerged via backport-assistant

…ted-client-options/urgently-topical-shiner This pull request was automerged via backport-assistant

…ewrite/poorly-bright-pug This pull request was automerged via backport-assistant

…mplate-detail/firstly-glorious-panther This pull request was automerged via backport-assistant

…1-website/plainly-trusty-foxhound This pull request was automerged via backport-assistant

…st-beagle This pull request was automerged via backport-assistant

…-1.2.6 (#12377)

Co-authored-by: Sarah <[email protected]> Co-authored-by: Luiz Aoqui <[email protected]>

…p-canary-deploys/routinely-able-pangolin This pull request was automerged via backport-assistant

…-good-frog This pull request was automerged via backport-assistant

…-resources/truly-fond-walleye This pull request was automerged via backport-assistant

…eserved-cores/amazingly-valued-lacewing This pull request was automerged via backport-assistant

…ely-coherent-calf This pull request was automerged via backport-assistant

…s/optionally-just-pup This pull request was automerged via backport-assistant

…p-reservedMemory-unit/curiously-calm-snake This pull request was automerged via backport-assistant

…#11956)

…cate-check/admittedly-precise-anteater This pull request was automerged via backport-assistant

…-counts/currently-beloved-terrier This pull request was automerged via backport-assistant

…-rodent This pull request was automerged via backport-assistant

…/initially-evolved-mongrel This pull request was automerged via backport-assistant

…hing/presently-sacred-snail This pull request was automerged via backport-assistant

…estarts/shortly-open-lamprey This pull request was automerged via backport-assistant

…4/strangely-intimate-sponge This pull request was automerged via backport-assistant

…sly-suited-rat This pull request was automerged via backport-assistant

…kindly-touching-thrush This pull request was automerged via backport-assistant

…/frankly-sweet-guinea This pull request was automerged via backport-assistant

…lly-viable-newt This pull request was automerged via backport-assistant

…t-claim-time/specially-simple-swine This pull request was automerged via backport-assistant

…rally-real-orca This pull request was automerged via backport-assistant

…ially-exciting-kodiak This pull request was automerged via backport-assistant

…ry-on-claim/firmly-lucky-shark This pull request was automerged via backport-assistant

…ugin-on-job-delete/grossly-driving-dog This pull request was automerged via backport-assistant

…-responses/positively-divine-cowbird This pull request was automerged via backport-assistant

The Prestart hook for task runner hooks doesn't get called when we restore a task, because the task is already running. The Postrun hook for CSI plugin supervisors needs the socket path to have been populated so that the client has a valid path.

…fix/nicely-wealthy-monarch This pull request was automerged via backport-assistant

…-prefix/deeply-endless-wolf This pull request was automerged via backport-assistant

…tering/finally-diverse-squid This pull request was automerged via backport-assistant

…for-snapshot/commonly-living-cockatoo This pull request was automerged via backport-assistant

This reverts commit 1ffdda0.

The RPC for listing volume snapshots requires a plugin ID. Update the `volume snapshot list` command to find the specific plugin from the provided prefix.

…ible/firstly-still-ray This pull request was automerged via backport-assistant

Co-authored-by: Thomas Lefebvre <[email protected]>

A volume that has single-use access mode is feasibility checked during scheduling to ensure that only a single reader or writer claim exists. However, because feasibility checking is done one alloc at a time before the plan is written, a job that's misconfigured to have count > 1 that mounts one of these volumes will pass feasibility checking. Enforce the check at validation time instead to prevent us from even trying to evaluation a job that's misconfigured this way.

When a node is garbage collected, we assume that the volume is no longer attached to it and ignore the `ErrUnknownNode` error. But we used `errors.Is` to check for a wrapped error, and RPC flattens the errors during serialization. This results in an error check that works in automated testing but not in real clusters. Use a string contains check instead.

Listing snapshots was incorrectly returning nanoseconds instead of seconds, and formatting of timestamps both list and create snapshot was treating the timestamp as though it were nanoseconds instead of seconds. This resulted in create timestamps always being displayed as zero values. Fix the unit conversion error in the command line and the incorrect extraction in the CSI plugin client code. Beef up the unit tests to make sure this code is actually exercised.

…-header/officially-tidy-lamb This pull request was automerged via backport-assistant

…ted-client-options/officially-enough-lemming This pull request was automerged via backport-assistant

…-order/evenly-cosmic-eft This pull request was automerged via backport-assistant

The volume watcher design was based on deploymentwatcher and drainer, but has an important difference: we don't want to maintain a goroutine for the lifetime of the volume. So we stop the volumewatcher goroutine for a volume when that volume has no more claims to free. But the shutdown races with updates on the parent goroutine, and it's possible to drop updates. Fortunately these updates are picked up on the next core GC job, but we're most likely to hit this race when we're replacing an allocation and that's the time we least want to wait. Wait until the volume has "settled" before stopping this goroutine so that the race between shutdown and the parent goroutine sending on `<-updateCh` is pushed to after the window we most care about quick freeing of claims. * Fixes a resource leak when volumewatchers are no longer needed. The volume is nil and can't ever be started again, so the volume's `watcher` should be removed from the top-level `Watcher`. * De-flakes the GC job test: the test throws an error because the claimed node doesn't exist and is unreachable. This flaked instead of failed because we didn't correctly wait for the first pass through the volumewatcher. Make the GC job wait for the volumewatcher to reach the quiescent timeout window state before running the GC eval under test, so that we're sure the GC job's work isn't being picked up by processing one of the earlier claims. Update the claims used so that we're sure the GC pass won't hit a node unpublish error. * Adds trace logging to unpublish operations

The plugin manager for CSI hands out instances of a plugin for callers that need to mount a volume. The `MounterForPlugin` method accesses the internal instances map without a lock, and can be called concurrently from outside the plugin manager's main run-loop. The original commit for the instances map included a warning that it needed to be accessed only from the main loop but that comment was unfortunately ignored shortly thereafter, so this bug has existed in the code for a couple years without being detected until we ran tests with `-race` in #12098. Lesson learned here: comments make for lousy enforcement of invariants!

…tus/partly-modern-mastiff This pull request was automerged via backport-assistant

…4/actually-elegant-tick This pull request was automerged via backport-assistant

…11927/primarily-robust-sawfly This pull request was automerged via backport-assistant

…lid-marmoset This pull request was automerged via backport-assistant

The Prestart hook for task runner hooks doesn't get called when we restore a task, because the task is already running. The Postrun hook for CSI plugin supervisors needs the socket path to have been populated so that the client has a valid path. Co-authored-by: Tim Gross <[email protected]>

…unds-link/yearly-valued-gecko This pull request was automerged via backport-assistant

…client-restart/amazingly-hot-owl This pull request was automerged via backport-assistant

…estructive/forcibly-noted-mayfly This pull request was automerged via backport-assistant

…cerely-handy-sheep This pull request was automerged via backport-assistant

…12813)

* Merge pull request #11889 from hashicorp/build-update-circle build: upgrade circleci configuration * Merge pull request #12299 from hashicorp/ci-parallel ci: trade test parallelization for unconstrained gomaxprocs * Merge pull request #12321 from hashicorp/ci-less-logging ci: limit gotestsum to circle ci * CI: build binaries for UI branches (#12594) Build binaries for every code change, not just backend code changes. This means that we'll have up-to-date compiled assets for every commit available in CircleCI artifacts. * Merge pull request #12736 from hashicorp/build-update-go-1.17.9 build: update golang to 1.17.9 * cgutil test: reserve only a single CPU for AddAlloc test Reserving the entire machine's worth of CPUs for the `AddAlloc` test triggers a condition where the empty shared CPUs defaults to the parent CPU set on some system configurations. This was done in `main` as part of cgroups v2 work, but we need to backport this to earlier branches in order to use the same machines across branches. Co-authored-by: Seth Hoenig <[email protected]> Co-authored-by: Tim Gross <[email protected]>

…acity/positively-concise-goblin This pull request was automerged via backport-assistant

…kefile/hardly-eternal-bee This pull request was automerged via backport-assistant

…ral-fowl This pull request was automerged via backport-assistant

…worthy-catfish This pull request was automerged via backport-assistant

…/vaguely-chief-polliwog This pull request was automerged via backport-assistant

…/badly-settled-gibbon This pull request was automerged via backport-assistant

…ing/centrally-main-monkfish This pull request was automerged via backport-assistant

…igorously-living-liger This pull request was automerged via backport-assistant

…node-plan-rejected-log/probably-whole-rooster This pull request was automerged via backport-assistant

…ny/newly-known-hyena This pull request was automerged via backport-assistant

* remove website source code and related circle jobs * remove data files * updates platform-cli * update local instructions * updates package-lock

During the release there are several files that need to be modified: - .release/ci.hcl: the notification channel needs to be updated to a channel with greater team visibility during the release. - version/version.go: the Version and VersionPrerelease variables need to be set so they match the release version. After the release these files need to be reverted. For GA releases the following additional changes also need to happen: - version/version.go: the Version variable needs to be bumped to the next version number. - GNUMakefile: the LAST_RELEASE variable needs to be set to the version that was just released. Since the release process will commit file changes to the branch being used for the release, it should _never_ run on main, so the first step is now to protect against that. It also adds a validation to make the user input version is correct. After looking at the different release options and steps I noticed that automatic CHANGELOG generation is actually the exception, so it would be better to have the default to be false.

add service acct to codeowners for backport merging

This pull request was automerged via backport-assistant

…12948) This pull request was automerged via backport-assistant

This pull request was automerged via backport-assistant

* no-op commit due to failed cherry-picking * docs: update json jobs docs (#12766) * docs: update json jobs docs Did you know that Nomad has not 1 but 2 JSON formats for jobs? 2½ if you want to acknowledge that sometimes our JSON job representations have a Job top-level wrapper and sometimes do not. The 2½ formats are: ``` 1. HCL JSON 2. Input API JSON (top-level Job field) 2.5. Output API JSON (lacks top-level Job field) ``` `#2` is what our docs consider our API JSON. `#2.5` seems to be an accident of history we can't fix with breaking API compatibility. `#1` is an even more interesting accident of history: the `jobspec2` package automatically detects if the input to Parse is JSON and switches to a JSON parser. This behavior is undocumented, the format is unspecified, and there is no official HashiCorp tooling to produce this JSON from HCL. The plot thickens when you discover popular third party tools like hcl2json.com and https://github.com/tmccombs/hcl2json seem to produce JSON that `nomad run` accepts! Since we have no telemetry around whether or not anyone passes HCL JSON to `nomad run`, and people don't file bugs around features that Just Work, I'm choosing to leave that code path in place and *acknowledged but not suggested* in documentation. See hashicorp/hcl#498 for a more comprehensive discussion of what officially supporting HCL JSON in Nomad would look like. (I also added some of the missing fields to the (Input API flavor) JSON Job documentation, but it still needs a lot of work to be comprehensive.) Co-authored-by: Tim Gross <[email protected]> Co-authored-by: temp <[email protected]> Co-authored-by: Michael Schurter <[email protected]> Co-authored-by: Tim Gross <[email protected]>

This pull request was automerged via backport-assistant

Fix numerous go-getter security issues: - Add timeouts to http, git, and hg operations to prevent DoS - Add size limit to http to prevent resource exhaustion - Disable following symlinks in both artifacts and `job run` - Stop performing initial HEAD request to avoid file corruption on retries and DoS opportunities. **Approach** Since Nomad has no ability to differentiate a DoS-via-large-artifact vs a legitimate workload, all of the new limits are configurable at the client agent level. The max size of HTTP downloads is also exposed as a node attribute so that if some workloads have large artifacts they can specify a high limit in their jobspecs. In the future all of this plumbing could be extended to enable/disable specific getters or artifact downloading entirely on a per-node basis.

This pull request was automerged via backport-assistant

(cherry-pick and fix conflict)

Commits on Jan 19, 2022

backport of commit b8db8f3

kainoaseto committed Jan 19, 2022

Configuration menu

View commit details

Copy full SHA for ab5c8eb

Browse repository at this point

Copy the full SHA

ab5c8eb View commit details

Browse the repository at this point in the history

Commits on Jan 21, 2022

backport of commit 3222ede

NomAnor committed Jan 21, 2022

Configuration menu

View commit details

Copy full SHA for 3f9b8f6

Browse repository at this point

Copy the full SHA

3f9b8f6 View commit details

Browse the repository at this point in the history

Commits on Jan 25, 2022

backport of commit 15442b3

shoenig committed Jan 25, 2022

Configuration menu

View commit details

Copy full SHA for 66c8160

Browse repository at this point

Copy the full SHA

66c8160 View commit details

Browse the repository at this point in the history

Commits on Jan 29, 2022

backport of commit 5da5985

beautifulentropy committed Jan 29, 2022

Configuration menu

View commit details

Copy full SHA for a685993

Browse repository at this point

Copy the full SHA

a685993 View commit details

Browse the repository at this point in the history

Commits on Feb 11, 2022

backport of commit 787acdf

tgross committed Feb 11, 2022

Configuration menu

View commit details

Copy full SHA for b1f0628

Browse repository at this point

Copy the full SHA

b1f0628 View commit details

Browse the repository at this point in the history

Commits on Mar 23, 2022

backport of commit 0fe621d

tgross committed Mar 23, 2022

Configuration menu

View commit details

Copy full SHA for a95b2b8

Browse repository at this point

Copy the full SHA

a95b2b8 View commit details

Browse the repository at this point in the history

Commits on Mar 30, 2022

backport of commit 18f1055

Bryce Kalow committed Mar 30, 2022

Configuration menu

View commit details

Copy full SHA for 6e4fd84

Browse repository at this point

Copy the full SHA

6e4fd84 View commit details

Browse the repository at this point in the history

Commits on Apr 15, 2022

backport of commit d17f537

tgross committed Apr 15, 2022

Configuration menu

View commit details

Copy full SHA for a6dd43a

Browse repository at this point

Copy the full SHA

a6dd43a View commit details

Browse the repository at this point in the history

Commits on Apr 17, 2022

backport of commit a099660

chavacava authored Apr 17, 2022

Configuration menu

View commit details

Copy full SHA for 18f10b3

Browse repository at this point

Copy the full SHA

18f10b3 View commit details

Browse the repository at this point in the history

Commits on Apr 18, 2022

backport of commit 761c6c3

tgross committed Apr 18, 2022

Configuration menu

View commit details

Copy full SHA for 7874ce0

Browse repository at this point

Copy the full SHA

7874ce0 View commit details

Browse the repository at this point in the history

Commits on May 3, 2022

backport of commit d551cda

mdeggies committed May 3, 2022

Configuration menu

View commit details

Copy full SHA for 71367e3

Browse repository at this point

Copy the full SHA

71367e3 View commit details

Browse the repository at this point in the history

manual backport docker test fixes to 1.2.x #13117

manual backport docker test fixes to 1.2.x #13117

Commits on Jan 19, 2022

Commits on Jan 21, 2022

Commits on Jan 25, 2022

Commits on Jan 26, 2022

Commits on Jan 27, 2022

Commits on Jan 29, 2022

Commits on Jan 31, 2022

Commits on Feb 2, 2022

Commits on Feb 3, 2022

Commits on Feb 8, 2022

Commits on Feb 10, 2022

Commits on Feb 11, 2022

Commits on Feb 15, 2022

Commits on Feb 22, 2022

Commits on Feb 23, 2022

Commits on Feb 24, 2022

Commits on Mar 1, 2022

Commits on Mar 3, 2022

Commits on Mar 4, 2022

Commits on Mar 8, 2022

Commits on Mar 10, 2022

Commits on Mar 14, 2022

Commits on Mar 23, 2022

Commits on Mar 28, 2022

Commits on Mar 30, 2022

Commits on Mar 31, 2022

Commits on Apr 6, 2022

Commits on Apr 8, 2022

Commits on Apr 11, 2022

Commits on Apr 12, 2022

Commits on Apr 14, 2022

Commits on Apr 15, 2022

Commits on Apr 17, 2022

Commits on Apr 18, 2022

Commits on Apr 19, 2022

Commits on Apr 20, 2022

Commits on Apr 21, 2022

Commits on Apr 22, 2022

Commits on Apr 25, 2022

Commits on Apr 26, 2022

Commits on Apr 28, 2022

Commits on Apr 29, 2022

Commits on May 1, 2022

Commits on May 2, 2022

Commits on May 3, 2022

Commits on May 4, 2022

Commits on May 5, 2022

Commits on May 6, 2022

Commits on May 10, 2022

Commits on May 11, 2022

Commits on May 12, 2022

Commits on May 13, 2022

Commits on May 16, 2022

Commits on May 17, 2022

Commits on May 19, 2022

Commits on May 20, 2022

Commits on May 24, 2022

Commits on May 25, 2022