Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CWS] improve speed of btfhub archive constant builder script #11467

Merged
merged 2 commits into from
Mar 30, 2022
Merged

[CWS] improve speed of btfhub archive constant builder script #11467

merged 2 commits into from
Mar 30, 2022

Conversation

paulcacheux
Copy link
Contributor

@paulcacheux paulcacheux commented Mar 28, 2022 •
edited
Loading

What does this PR do?

This PR allows the script to extract multiple constant sets in parallel.
The concurrency is limited to 2 * NCPUs because you can easily go over the limit in opened file descriptor.

Before:

Executed in  208.30 secs    fish           external
   usr time  196.47 secs  805.00 micros  196.47 secs
   sys time   63.45 secs    0.00 micros   63.45 secs

After:

Executed in   74.87 secs    fish           external
   usr time  172.61 secs    0.00 micros  172.61 secs
   sys time   11.13 secs  376.00 micros   11.13 secs

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

Reviewer's Checklist

  • If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
  • Use the major_change label if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.
  • A release note has been added or the changelog/no-changelog label has been applied.
  • Changed code has automated tests for its functionality.
  • Adequate QA/testing plan information is provided if the qa/skip-qa label is not applied.
  • At least one team/.. label has been applied, indicating the team(s) that should QA this change.
  • If applicable, docs team has been notified or an issue has been opened on the documentation repo.
  • If applicable, the need-change/operator and need-change/helm labels have been applied.
  • If applicable, the config template has been updated.

@paulcacheux paulcacheux added this to the 7.36.0 milestone Mar 28, 2022
@paulcacheux paulcacheux marked this pull request as ready for review March 28, 2022 13:36
@paulcacheux paulcacheux requested a review from a team as a code owner March 28, 2022 13:36
@paulcacheux paulcacheux merged commit 09b2187 into main Mar 30, 2022
@paulcacheux paulcacheux deleted the paulcacheux/speed-btfhub branch March 30, 2022 08:16
thedavlee pushed a commit that referenced this pull request Apr 13, 2022
@thedavl
Squashed commit of the following:
1470efd 
commit 5aab196
Author: Nicolas Hinsch <[email protected]>
Date:   Tue Apr 12 18:49:52 2022 -0400

    [serverless] Share snapshots between integration tests (#11630)

commit 3cb281b
Author: Olivier Vielpeau <[email protected]>
Date:   Tue Apr 12 12:00:58 2022 +0200

    [changelog] Fix reST formatting in 7.35.0 entry (#11623)

    * [changelog] Fix reST formatting in 7.35.0 entry

    And mention for clarity that some changes are related to CWS

    * [changelog] More formatting fixes

    * [changelog] Fix one more reST format thing

    * [changelog] Unfix

    * [changelog] Last try

commit 6e2ec40
Author: Carlos <[email protected]>
Date:   Tue Apr 12 08:59:09 2022 +0100

    [fix] check_logger now correctly logs next run message (#11617)

commit a32607a
Author: Kacper <[email protected]>
Date:   Mon Apr 11 23:44:19 2022 +0200

    Update release.json and Go modules for 6/7.36.0-rc.2 (#11628)

commit 973e94d
Author: Paul Cacheux <[email protected]>
Date:   Mon Apr 11 17:19:24 2022 +0200

    [CSPM] bump security-agent-policies to v0.25.2 (#11613)

    * [CSPM] bump security-agent-policies to v0.25.1

    * [CSPM] bump security-agent-policies to v0.25.2

commit 99e41c0
Author: Sylvain Afchain <[email protected]>
Date:   Mon Apr 11 11:57:19 2022 +0200

    [CWS] do not copy args envs during the snapshot (#11571)

    * Do not copy args envs during the snapshot

    * Fix ancestors during the snapshot

commit cb51f14
Author: Julio Greff <[email protected]>
Date:   Thu Apr 7 15:28:18 2022 +0200

    [tagger] Add static tags to container/pod entities

    These were removed in #11293, and then lived just in GlobalEntityID,
    with the assumption that all data coming out of the agent would call
    `tagger.EnrichTags`, where we add GlobalEntityID's tags to another set
    of tags. That assumption is wrong, as it turns out that process and
    container collection (and possibly other components) *do not* call that
    func, and therefore would not have the static tags added.

commit 8fd9f07
Author: Ivan Ilichev <[email protected]>
Date:   Fri Apr 8 07:06:11 2022 -0400

    [process-agent] Fix nil derefef on MacOS and suppress RT check banner (#11601)

    - Fix nil dereference in the `check` command on Mac (flag not configured).
    - Suppress the check results banner when JSON output is requested.

commit f452a7d
Author: Kacper <[email protected]>
Date:   Thu Apr 7 21:39:34 2022 +0200

    last_stable set to 6/7.35.0 (#11598)

commit a9c94ae
Author: Kacper <[email protected]>
Date:   Thu Apr 7 21:39:22 2022 +0200

    Changelog for 7.35.0 (#11592) (#11597)

    * Changelog for 7.35.0

    * Update CHANGELOG.rst

    * Update CHANGELOG.rst

    Co-authored-by: Olivier Vielpeau <[email protected]>

    Co-authored-by: Olivier Vielpeau <[email protected]>

    Co-authored-by: Olivier Vielpeau <[email protected]>

commit 1d7c1dc
Author: Slavek Kabrda <[email protected]>
Date:   Thu Apr 7 20:29:18 2022 +0200

    Fix dependency of deb packages on datadog-signing-keys (#11588)

commit ceaf503
Author: Julio Greff <[email protected]>
Date:   Tue Mar 22 12:31:35 2022 +0100

    [workloadmeta] Clear list of subscribers when unsubscribing all

    This prevents a subscriber's channel from being closed twice (generating
    a panic), since it's possible for store.Unsubscribe to be called after
    the workloadmeta has been asked to shut down, depending on the order
    other components are shut down relative to workloadmeta.

commit ef4421b
Author: Bryce Kahle <[email protected]>
Date:   Wed Apr 6 13:00:08 2022 -0700

    Fix conflict with probe names between NPM/OOM/TCPQ (#11586)

    * Fix conflict with probe names between NPM/OOM/TCPQ

    * Add changelog

commit 268da67
Author: Lénaïc Huard <[email protected]>
Date:   Wed Apr 6 18:52:39 2022 +0200

    Fix E2E tests `failed to create containerd task` (#11577)

    Fix
    ```
    StartError (exit code 128): failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "sh": executable file not found in $PATH: unknown
    ```
    caused by the removal of the `busybox` shell from the `argoproj/argoexec` image.

commit 2fb5e82
Author: Paul Cacheux <[email protected]>
Date:   Wed Apr 6 18:11:10 2022 +0200

    [CWS] bump security agent policies to v0.25.0 (#11582)

commit 58d202d
Author: Hasan Mahmood <[email protected]>
Date:   Wed Apr 6 10:22:50 2022 -0500

    Add missing call to manager Start for dns filter (#11564)

commit 21df6ba
Author: Paul Cacheux <[email protected]>
Date:   Wed Apr 6 15:57:25 2022 +0200

    [CWS] other fixes to constant tests (#11545)

    * [CWS] improve kernel version display on failed constant test

    * [CWS] bump debian 11 version (for kernel headers)

    * Fix `pipe_inode_info_bufs_offset` for debian 11

    * Skip DNS and NetworkDevice tests on SLES (but keep OpenSUSE)

    * Exclude TC programs for offset guesser

    * Add escape hatch to ignore missing constant if we know it may be missing

commit 1bb4ed2
Author: Kylian Serrania <[email protected]>
Date:   Wed Apr 6 11:17:02 2022 +0200

    [gitlab] Set Windows NPM / system-probe tests as allowed to fail (#11573)

    Sets kitchen_test_system_probe_windows_x64, kitchen_windows_installer_npm_install_scenarios-a7 and kitchen_windows_installer_npm_driver-a7 as allowed to fail.

commit 0228dc0
Author: Ahmed Mezghani <[email protected]>
Date:   Wed Apr 6 10:09:53 2022 +0200

    [Cluster Agent] Support internal profiling (#11568)

    * [Cluster Agent] Support internal profiling

    * add common.SetupInternalProfiling()

commit a239866
Author: Vincent Boulineau <[email protected]>
Date:   Wed Apr 6 10:09:29 2022 +0200

    Allow fast custom image publication (#11519)

commit 2d26c2f
Author: Kacper <[email protected]>
Date:   Tue Apr 5 10:55:27 2022 +0200

    Update release.json and Go modules for 6/7.36.0-rc.1 (#11567)

commit d097a57
Author: Vincent Boulineau <[email protected]>
Date:   Tue Apr 5 10:02:34 2022 +0200

    Create a new dedicated call to get count of openfiles for a set of PIDs (#11435)

commit 416a923
Author: Kylian Serrania <[email protected]>
Date:   Mon Apr 4 18:39:01 2022 -0400

    [gitlab] Fix junit upload job when no owners are specified (#11563)

    Follow-up of #11562.
    Makes the junit upload task not fail when no codeowners are assigned, as having no codeowners is a valid codeowners state, according to the codeowners syntax.

commit 1687f5c
Author: Corrina Sivak <[email protected]>
Date:   Mon Apr 4 17:11:39 2022 -0400

    [Process Agent] Disable console logging when running `check --json` (#11560)

    * [Process Agent] Disable console logging when running `check --json`

    * [Process Agent] Disable console logging when running `check --json`

    * Add release note

    * Update process-agent-fix-json-output-bc58ab39de316bc4.yaml

commit 4f4f02d
Author: Dustin J. Mitchell <[email protected]>
Date:   Mon Apr 4 16:02:58 2022 -0400

    Dustin.mitchell/fix codeowners errors (#11562)

    * replace do-not-notify team with empty team list
    * fix spelling of team name

commit 20bd3ed
Author: Celene <[email protected]>
Date:   Mon Apr 4 15:40:10 2022 -0400

    Update permission of secret backend scripts for cluster check runners (#11246)

    * Update permission of secret backend scripts for cluster check runners

commit fbc44d2
Author: Dustin J. Mitchell <[email protected]>
Date:   Mon Apr 4 15:34:41 2022 -0400

    Factor containers-or-pods choice into a package (#11289)

    The current logic for deciding which launcher to run is complex, but
    changing it will result in undesirable behavior changes.  This commit
    isolates that logic in a dedicated containerorpods package and uses it
    in the two launchers.

    This prepares for breaking those launchers up into smaller components,
    all of which will depend on the new package.

commit b2c5b17
Author: Olivier G <[email protected]>
Date:   Mon Apr 4 14:35:43 2022 -0400

    Remove SendSeries (#11517)

    * Duplicate Series.MarshalJSON and Series.SplitPayload to make next commit easier to review
    * Implement JSONMarshaler for IterableSeries
    * Call IterationStopped() after all kinds of series serialization.
    * Remove `aggregator_flush_metrics_and_serialize_in_parallel` (always use parallel serialization)
    * Remove flushedSeries from flushTrigger
    * As SentIterableSerie supports JSONMarshaler, IterableSeries is always supported.
    * Use SendIterableSeries in serializerConsumer
    * Remove SendSeries
    * Fix an issue when serializing 0 series (Was {"series":nil} instead of `{"series":[]}`
    * Remove unused methods from metrics/series.go
    * Move functions from series.go to iterable_series.go (no code change)
    * Fix race conditions in unit tests.
    - replace serializer.MockSerializer by MockSerializerIterableSerie
    - fix a race condition when the aggregator is running and the line `demux.aggregator.serializer = s` is called.

commit 45c01f4
Author: Arthur Bellal <[email protected]>
Date:   Mon Apr 4 14:16:59 2022 -0400

    cmd/trace-agent: remove remote rates and config endpoint APM feature flag (#11395)

    * (rcm) remove remote rates APM feature flag

    * add an option to disable APM

    * Update run.go

commit c729181
Author: Arthur Bellal <[email protected]>
Date:   Mon Apr 4 14:16:25 2022 -0400

    (rcm) modularise and refactor the remote-config client (#11528)

    * (rcm) extract and partially rewrite secondary RC client

    * modularize

    * first working version

    * migrate all partial client tests

    * fix build

    * fix all tests

    * fix trace agent build

    * add missing comments

    * fix typo

    * add missing module entry

    * unused

    * failing test

    * update CODEOWNERS

commit bdbfa76
Author: Hasan Mahmood <[email protected]>
Date:   Mon Apr 4 12:25:32 2022 -0500

    Fix system-probe functional tests (#11537)

commit 4371715
Author: Dustin J. Mitchell <[email protected]>
Date:   Mon Apr 4 13:24:03 2022 -0400

    generalize logs-agent AD Listener (#11416)

    This generalizes the AD listener, allowing multiple subscribers to config's from AD.

    The AD listener handles the details of waiting until AD is available. It is currently only used in the AD logs scheduler.

commit 221c333
Author: David Ortiz <[email protected]>
Date:   Mon Apr 4 18:32:38 2022 +0200

    [clusteragent/admission/webhook] Add selectors needed for AKS (#11367)

    This fixes a problem that happens when enabling the DCA admission
    controller on Azure AKS.

    AKS automatically updates our webhooks adding a selector that excludes
    the "control-plane" namespace. This creates conflicts when we reconcile
    the webhooks in the controller and spams the logs with messages like:

    `Couldn't reconcile Webhook datadog-webhook: Operation cannot be
    fulfilled on mutatingwebhookconfigurations.admissionregistration.k8s.io
    "datadog-webhook": the object has been modified; please apply your
    changes to the latest version and try again`.

    References:
    - https://docs.microsoft.com/en-us/azure/aks/faq#can-i-use-admission-controller-webhooks-on-aks
    - Azure/AKS#1771

commit 681c509
Author: Lénaïc Huard <[email protected]>
Date:   Mon Apr 4 18:31:46 2022 +0200

    Switch default value of `external_metrics_provider.port` to 8443 (#11408)

commit 9d5c9c6
Author: Arthur Bellal <[email protected]>
Date:   Mon Apr 4 12:28:33 2022 -0400

    Update CODEOWNERS (#11559)

    Adds @DataDog/remote-config as codeowners of /pkg/remoteconfig/

commit 8f8994e
Author: Guillaume Fournier <[email protected]>
Date:   Mon Apr 4 18:09:45 2022 +0200

    [CWS] Add IP and CIDR secl types (#11460)

    * add IP and CIDR secl types

    * fix CI

    * Rebase fixes

    Co-authored-by: Sylvain Afchain <[email protected]>

commit d2b9d5c
Author: Cédric Van Rompay <[email protected]>
Date:   Mon Apr 4 15:21:02 2022 +0200

    [SINT-408] Test Security of Uptane Partial Verification (#11449)

    * list TODO as comment

    * test rejection of unsigned targets

    * test rejects invalid signature

    * test rejection of revoked keys

    also test valid root key rotation

    * minor: adding comments

    * fix linter error (comment exported values)

    * minor: add some documentation

    * factorize similar error handling

    * (minor) use require.New

    * use "config" as filename

    resolves #11449 (comment)

commit 90a4a63
Author: Albert Vaca Cintora <[email protected]>
Date:   Mon Apr 4 08:45:46 2022 -0400

    Remove team/agent-platform default label from dependabot PR (#11549)

    If we don't know the team that owns a dependency beforehand, add
    team/triage. This makes it easier to identify and assign them.

commit 783d097
Author: Ahmed Mezghani <[email protected]>
Date:   Mon Apr 4 08:41:03 2022 -0400

    [DCA][Admission Controller] Use ReinvocationPolicy `IfNeeded` + make it configurable (#11440)

commit 62f2859
Author: Paul Cacheux <[email protected]>
Date:   Mon Apr 4 13:26:55 2022 +0200

    [CWS] sync BTFHub constants (#11539)

    * [CWS] sync BTFHub constants

    * [CWS] add constants.json as generated files in gitattributes

commit d203439
Author: Arthur Bellal <[email protected]>
Date:   Mon Apr 4 06:54:42 2022 -0400

    (rcm) bypass orgID check for employee sourced configs (#11536)

    * (rcm) bypass orgID check for employee sourced configs

    * missing comment

    * typo

commit a05103c
Author: Vincent Boulineau <[email protected]>
Date:   Mon Apr 4 12:15:30 2022 +0200

    Fix PID retrieval with cgroupv2 without host cgroup namespace (#11506)

commit b911529
Author: Paul Cacheux <[email protected]>
Date:   Mon Apr 4 11:15:47 2022 +0200

    [CWS] fix docker functional test arm64 (#11502)

    * [CWS] bump nikos

    * [CWS] remove x86 libc install and add ca-certificates in docker local functests

    * [CWS] set DOCKER_DD_AGENT=yes in docker functional tests

    * [CWS] small improvements

    * [CWS] `DOCKER_DD_AGENT=yes` in kitchen tests docker container

    * [CWS] fix python linting issues

    * [CWS] fix volumes

    * [CWS] improve volumes and env for kitchen tests in container

    * Fix typo

commit da92310
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Apr 4 11:15:17 2022 +0200

    Bump github.com/vektra/mockery/v2 from 2.10.0 to 2.10.2 in /internal/tools (#11530)

    * Bump github.com/vektra/mockery/v2 in /internal/tools

    Bumps [github.com/vektra/mockery/v2](https://github.com/vektra/mockery) from 2.10.0 to 2.10.2.
    - [Release notes](https://github.com/vektra/mockery/releases)
    - [Changelog](https://github.com/vektra/mockery/blob/master/.goreleaser.yml)
    - [Commits](vektra/mockery@v2.10.0...v2.10.2)

    ---
    updated-dependencies:
    - dependency-name: github.com/vektra/mockery/v2
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    * Run `invoke -e security-agent.gen-mocks`

    * Run `inv -e tidy-all`

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Paul Cacheux <[email protected]>

commit cb9f293
Author: Ahmed Mezghani <[email protected]>
Date:   Mon Apr 4 04:42:15 2022 -0400

    [Cluster Agent] Add info metric cluster_checks_configs_info (#11518)

commit 6c03a0f
Author: Pablo Baeyens <[email protected]>
Date:   Mon Apr 4 10:42:04 2022 +0200

    [internal/tools] Run `inv -e tidy-all` (#11542)

commit b43de5a
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Apr 4 10:40:54 2022 +0200

    Bump black from 22.1.0 to 22.3.0 in /.circleci (#11534)

    Bumps [black](https://github.com/psf/black) from 22.1.0 to 22.3.0.
    - [Release notes](https://github.com/psf/black/releases)
    - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
    - [Commits](psf/black@22.1.0...22.3.0)

    ---
    updated-dependencies:
    - dependency-name: black
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 96dce3b
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Apr 4 10:13:09 2022 +0200

    Bump github.com/fzipp/gocyclo from 0.4.0 to 0.5.0 in /internal/tools (#11532)

    * Bump github.com/fzipp/gocyclo from 0.4.0 to 0.5.0 in /internal/tools

    Bumps [github.com/fzipp/gocyclo](https://github.com/fzipp/gocyclo) from 0.4.0 to 0.5.0.
    - [Release notes](https://github.com/fzipp/gocyclo/releases)
    - [Changelog](https://github.com/fzipp/gocyclo/blob/main/CHANGELOG.md)
    - [Commits](fzipp/gocyclo@v0.4.0...v0.5.0)

    ---
    updated-dependencies:
    - dependency-name: github.com/fzipp/gocyclo
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    * Auto-generate go.sum and LICENSE-3rdparty.csv changes

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

commit acd4557
Author: Pablo Baeyens <[email protected]>
Date:   Mon Apr 4 10:11:28 2022 +0200

    [pkg/otlp] Add unit tests for literal `null` on config (#11374)

commit a941a03
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Apr 4 10:10:21 2022 +0200

    Bump github.com/golangci/golangci-lint from 1.44.2 to 1.45.2 in /internal/tools (#11533)

    * Bump github.com/golangci/golangci-lint in /internal/tools

    Bumps [github.com/golangci/golangci-lint](https://github.com/golangci/golangci-lint) from 1.44.2 to 1.45.2.
    - [Release notes](https://github.com/golangci/golangci-lint/releases)
    - [Changelog](https://github.com/golangci/golangci-lint/blob/master/CHANGELOG.md)
    - [Commits](golangci/golangci-lint@v1.44.2...v1.45.2)

    ---
    updated-dependencies:
    - dependency-name: github.com/golangci/golangci-lint
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    * Auto-generate go.sum and LICENSE-3rdparty.csv changes

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

commit 6850ee7
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Apr 4 10:09:48 2022 +0200

    Bump flake8-bugbear from 22.1.11 to 22.3.23 in /.circleci (#11535)

    Bumps [flake8-bugbear](https://github.com/PyCQA/flake8-bugbear) from 22.1.11 to 22.3.23.
    - [Release notes](https://github.com/PyCQA/flake8-bugbear/releases)
    - [Commits](PyCQA/flake8-bugbear@22.1.11...22.3.23)

    ---
    updated-dependencies:
    - dependency-name: flake8-bugbear
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit e29f26a
Author: Raphael Gavache <[email protected]>
Date:   Mon Apr 4 08:36:12 2022 +0200

    pkg/trace: fix deadlock in sampler (#11538)

    I introduced a dead lock on agent 7.35.RC2 with #11069. Every 5s muRates.Lock is called while muSeen is locked (`countWeightedSig` calls `updateRates`).

    #### Deadlock example

    1. muSeen.Lock() aquired by goroutine A `countWeightedSample`
    2. muRate.Rlock() aquired by goroutine B `getSignatureSampleRate`
    3. goroutine B tries to aquire muSee.Rlock() and waits (new signature triggers a defaultRate() call)
    4. goroutine A tries to aquire muRate.Lock() and waits (5s rotation of the rates)

    #### PR changes
    - fixes the deadlock by releasing muRate as soon as possible on getSignature functions
    - ensures TestSamplerAccessRace catch this deadlock and future ones by forcing bucket rotations
    - move sampler size metric to Gauge type

commit af25c37
Author: Paul <[email protected]>
Date:   Fri Apr 1 23:15:29 2022 +0200

    Add integration trace flag (#11520)

    * Add integration trace flag

    * Address review

    * Add hidden config

commit 3b29134
Author: Derek Brown <[email protected]>
Date:   Fri Apr 1 11:37:24 2022 -0700

    Windows/NPM Update release.json for 1.3.2 driver (#11417)

    * add looping test for npm driver

    * add testsigned build

    * add timeout to test loop

    * Remove new tests (they didn't work)
    Update to tagged driver for signing

    * update to release-signed driver; remove accidentally added code

    * Update releasenotes/notes/npm132driver-f59bb5ae5effecce.yaml

    Co-authored-by: May Lee <[email protected]>

    * Update `release` section too

    Co-authored-by: May Lee <[email protected]>

commit 4746bce
Author: Kylian Serrania <[email protected]>
Date:   Fri Apr 1 12:04:29 2022 -0400

    [gitlab] Silence check_serverless_size_arm64 (#11527)

    This check is currently failing, and the server less team is currently investigating it.
    This PR sets the check as allowed to fail, to not block the upcoming release.

commit 116fe13
Author: Brian Floersch <[email protected]>
Date:   Fri Apr 1 11:29:37 2022 -0400

    adding additional telemetry to logs sender (#11514)

    * adding additional telemetry

    * formatting

commit de56f0c
Author: Sylvain Afchain <[email protected]>
Date:   Fri Apr 1 17:17:46 2022 +0200

    Refactor the GetNextAncestorNoFork functions to remove the limit (#11493)

commit 192b941
Author: Sylvain Baubeau <[email protected]>
Date:   Fri Apr 1 17:16:16 2022 +0200

    Fix isxdigit symbol conflict with kernel 5.16 (#11524)

commit ce79c7d
Author: Sylvain Baubeau <[email protected]>
Date:   Fri Apr 1 17:16:01 2022 +0200

    Bump security agent policies to v0.24.0 (#11503)

commit 1866d3d
Author: Sylvain Afchain <[email protected]>
Date:   Fri Apr 1 17:14:05 2022 +0200

    Reduce complexity of the approver discovery (#11508)

commit 327dd54
Author: Sylvain Afchain <[email protected]>
Date:   Fri Apr 1 16:05:19 2022 +0200

    [CWS] introduce case insensitive string matchers and use it for dns name (#11478)

    * Move matcher compilation within comparison functions

    * Introduce string compare options

    * Replace go string equality with a scalar StringMatcher

    * Rename IsScalar to IsStatic which more accurate

    * Implement case insensitive for all matchers

    * Introduce an operator override to leverage case insensitive

    * Apply to DNS

    * Fix lint of DNS

commit 855e837
Author: Mackenzie <[email protected]>
Date:   Fri Apr 1 15:38:27 2022 +0200

    pkg/trace/api: OTLP: fix race with merge attributes (#11525)

    This change fixes a race which resulted in the `rattr` map being modified as a side effect of passing it to `span.Meta` instead of making a copy of it.

    Co-authored-by: Gabriel Aszalos <[email protected]>

commit ab6a9da
Author: David Ortiz <[email protected]>
Date:   Fri Apr 1 15:18:35 2022 +0200

    [workloadmeta, tagger] Add QOS class to kube pods (#11515)

    * [workloadmeta/kubelet] Add QOS class to pods

    * [tagger/collectors/workloadmeta] Add kube_qos tag to pods

    * Add release note

commit 7042171
Author: Sylvain Afchain <[email protected]>
Date:   Fri Apr 1 14:28:40 2022 +0200

    Do not release all the argv cache entry (#10775)

commit 22c5e45
Author: Paul Cacheux <[email protected]>
Date:   Fri Apr 1 12:31:16 2022 +0200

    [CWS] improve cleanup of BTF spec in constant fetcher (#11523)

    * [CWS] improve cleanup of BTF spec in constant fetcher

    * [CWS] run GC after constant fetching

commit 7268106
Author: Gabriel Aszalos <[email protected]>
Date:   Fri Apr 1 12:49:24 2022 +0300

    pkg/trace/api: OTLP: set error.msg from status message (#11522)

    This change ensures that the "error.msg" is set from the span status message as a fallback, when no "exception" type span event is found.

commit 4b00c18
Author: Ivan Ilichev <[email protected]>
Date:   Thu Mar 31 18:31:01 2022 -0400

    [process-agent] Implement human readable check output (#11480)

    * [process-agent] Implement human readable check output

    * [WIP][process-agent] Add human readable output to real time checks (#11487)

    * add human readable output for rtcontainer check

    * add human readable output to rtprocess check

    * add process state

    * fix exported var/function comments

    * Reformat some templates for brevity

    * Add process state and switches info

    * Human readable timestamps

    * Container addresses

    * Humanize bytes

    * Add basic host info output

    * Add release note

    * Fix timestamps (in ms since epoch) and improve io stats output

    * Uppercase RSS and VMS

    * Address review feedback - error on incompatible type and crlf fix for Windows

    * Reorganize host info output in the templates

    * Fix container timestamps (in seconds) and improve cpupct helper

    * Remove Image and Name from output - they are deprecated fields in the proto

    Co-authored-by: Moisés Botarro <[email protected]>

commit 3c081fa
Author: Slavek Kabrda <[email protected]>
Date:   Thu Mar 31 21:18:27 2022 +0200

    [kitchen] Workaround dnf bug preventing installation on RHEL/CentOS 8.1 (#11516)

    When DNF runs in non-interactive mode on RHEL 8.1, it will run into https://bugzilla.redhat.com/show_bug.cgi?id=1792506 - essentially, if there are multiple gpgkey entries configured, only the first one will be respected. If the repodata is signed with key that's not the first entry, it will lead to yum makecache -y failing.

    This PR makes us turn off repodata signature verification on RHEL 8.1 thus avoiding this issue. (I did test on RHEL 8.5 and it works fine there with repodata signature verification turned off). Note that install_script.sh and all other config mgmt tools already do this, so this is the last place affected.

    Co-authored-by: Kylian Serrania <[email protected]>

commit a27c16c
Author: Vincent Boulineau <[email protected]>
Date:   Thu Mar 31 10:52:39 2022 -0400

    Change Cluster Agent API to be served only with TLS 1.3 by default (#11443)

commit fabb98b
Author: Paul Cacheux <[email protected]>
Date:   Thu Mar 31 10:37:40 2022 +0200

    [CWS] sync btfhub constants (#11504)

    * [CWS] sync btfhub constants

    * [CWS] sync btfhub constants

commit 4e682bb
Author: Julien Lebot <[email protected]>
Date:   Wed Mar 30 16:24:04 2022 -0400

    [Windows][Installer] Don't consider NT AUTHORITY as a domain. (#10926)

commit e61f892
Author: Florian Veaux <[email protected]>
Date:   Wed Mar 30 15:14:27 2022 -0400

    Bump gosnmp to v1.34.1-0.20220306115220-ca8397b73095 (#11494)

    * Bump gosnmp to v1.34.1-0.20220306115220-ca8397b73095

    * Update releasenotes/notes/Bump-gosnmp-to-v1.34.1-9d5fda248a1cbba9.yaml

    Co-authored-by: Kaylyn <[email protected]>

    Co-authored-by: Kaylyn <[email protected]>

commit 3ba5998
Author: Derek Brown <[email protected]>
Date:   Wed Mar 30 11:49:22 2022 -0700

    [windows] fix npm upgrade bug (#11446)

    * fix npm upgrade bug

    * add helpful comment

    * Update tools/windows/install-help/cal/customactiondata.cpp

    Co-authored-by: Julien Lebot <[email protected]>

    * Update tools/windows/install-help/cal/customactiondata.cpp

    Co-authored-by: Julien Lebot <[email protected]>

    Co-authored-by: Julien Lebot <[email protected]>

commit 91012a8
Author: Dustin J. Mitchell <[email protected]>
Date:   Wed Mar 30 12:44:06 2022 -0400

    Create a new LogSource when changing serverless tags (#11270)

    * Create a new LogSource when changing serverless tags

    The "channel" LogSource is only used by the serverless agent, but could
    be useful for other purposes, so this also abstracts the
    serverless-specific bits (name, source) out.

    More importantly, when the tags are updated, this now drops the old
    source and creates a new source.  This avoids race conditions with
    modifying a slice pointer while it might be in use by another gorotuine.

    Since the tailers for both sources are reading from the same channel, no
    messages will be lost -- a given message will either be handled by the
    old tailer or, if it stops before the message is handled, will wait for
    the new tailer to start.

    * Protect channel tailer tags with mutex

    The channel tailer closes its input channel, and the channel scheduler
    does not support removing sources, so this implements the async update
    of the tags applied to the channel with a new field (ChannelTags)
    protected by a new mutex (ChannelTagsMutex).  The new field avoids the
    requirement to guard _every_ access to LogConfig.Tags with a mutex.

    * Do not multiply append tags to serverless messages.

    * deduplicate ddtags in serverless snapshots

    * Fix Java integration tests

    * update comment

    Co-authored-by: Nick Hinsch <[email protected]>

commit 821c100
Author: Maxime mouial <[email protected]>
Date:   Mon Mar 28 11:35:46 2022 +0200

    Bump Gohai version

commit 598a395
Author: Paul Cacheux <[email protected]>
Date:   Wed Mar 30 17:11:46 2022 +0200

    [CWS] octogon improvements and fix offset guesser (#11500)

    * [CWS] add missing helper call

    * [CWS] improve output from failed `assertConstantContains`

    * [CWS] improve output from failed `assertConstantContains`

    * [CWS] fix offset guesser constant fetchr

commit 26978e4
Author: Paul Cacheux <[email protected]>
Date:   Wed Mar 30 15:27:12 2022 +0200

    [CWS] improve sles/suse kernel detection, enable `TestFallbackConstants` on OpenSUSE (#11423)

    * [CWS] improve sles/suse kernel detection, enable `TestFallbackConstants` on OpenSUSE

    * [CWS] bump opensuse urn version

    * Add debug prints

    * Fix debug print

    * More debug print

    * More debug

    * Fix rebase fail

    * Fix check compatibility in DNS and NetDevice tests

    * Fix some offsets on OpenSUSE 15

    * Revert debug utilities

    * [CWS] add missing helper call

    * [CWS] run dns and network device test on OpenSUSE

commit 0182e75
Author: Kylian Serrania <[email protected]>
Date:   Wed Mar 30 08:27:26 2022 -0400

    [gitlab] Increase datadog-(heroku)-agent size delta threshold to 25MB (#11499)

    Increases the check_pkg_size threshold from 20MB to 25MB for the datadog-agent and datadog-heroku-agent packages.

commit 1e2845b
Author: Gabriel Aszalos <[email protected]>
Date:   Wed Mar 30 15:26:48 2022 +0300

    pkg/trace/api: get rid of pb/otlppb (#11456)

    This change removes the copy of the proto structures and services made from the OpenTelemetry repositories in order to allow the module to be used in conjunction with the datadogexporter, sharing the very same data structures.

commit 42e72dd
Author: Andrea Marziali <[email protected]>
Date:   Wed Mar 30 13:44:25 2022 +0200

    pkg/trace: do not compute stats for long running spans (#11497)

commit cefa369
Author: Paul Cacheux <[email protected]>
Date:   Wed Mar 30 12:10:16 2022 +0200

    [CWS] constant fetcher octogon tests improvements (#11491)

    * [CWS] improve error display when an constant fetcher test fails

    * [CWS] add missing kernel compatibility checks

    * [CWS] add missing config definitions

    * [CWS] fix kernel compatibility checks

commit 47f71b3
Author: Slavek Kabrda <[email protected]>
Date:   Wed Mar 30 12:01:59 2022 +0200

    Rotate package signing keys for DEB and RPM packages (#11489)

commit d7e2c44
Author: Paul Cacheux <[email protected]>
Date:   Wed Mar 30 11:26:48 2022 +0200

    [CWS] add ubuntu 18.04 with kernel 4.15 to CI + fix constants (#11010)

    * [CWS] add ubuntu 18.04 with kernel 4.15 to CI

    * [CWS] fix constants on 4.15 kernel

    * [CWS] fix `pid_numbers_offset` on Ubuntu kernel 5.0

    * [CWS] compact CI functional tests

    * [CWS] revert sizeof inode changes

    * [CWS] add ubuntu kernel version specific parsing

    * [CWS] fix net_ns_offset constant on 4.15 kernels

    * [CWS] add missing documentation comment

    * [CWS] minimal abi versions

    * [CWS] remove upload from ubuntu kernel version

    * [CWS] fix sizeof inode constant on ubuntu 4.15 kernels

    * Fix sizeof inode on amazon linux

commit 09b2187
Author: Paul Cacheux <[email protected]>
Date:   Wed Mar 30 10:16:12 2022 +0200

    [CWS] improve speed of btfhub archive constant builder script (#11467)

    * [CWS] run multiple tar/btf extractions in parallel

    * [CWS] increase concurrent goroutines

commit 2a12a6d
Author: Paul Cacheux <[email protected]>
Date:   Wed Mar 30 10:07:41 2022 +0200

    [CWS] fix `pid_numbers_offset` fallback constant on Debian kernel 4.19 (#11490)

    * [CWS] fix `pid_numbers_offset` fallback constant on Debian kernel 4.19

    * Fix doc comment

commit 60c59cb
Author: Dustin J. Mitchell <[email protected]>
Date:   Tue Mar 29 16:36:54 2022 -0400

    use different max payload sizes for metric series (#11496)

    * supply max payload sizes to Compressor instead of fetching config directly
    * use different max payload sizes for metric series

commit 33c141d
Author: Bryce Kahle <[email protected]>
Date:   Tue Mar 29 15:37:32 2022 -0400

    Update debian 10 image (#11495)

    Kernel headers are no longer available for the existing version

commit a8ffc96
Author: Nathaniel Graham <[email protected]>
Date:   Tue Mar 29 13:16:47 2022 -0400

    admission controller: Set failure policy (#11432)

    * add admission controller failure policy config option

commit bcfa9e1
Author: Florian Veaux <[email protected]>
Date:   Tue Mar 29 13:05:55 2022 -0400

    Ship SNMP Traps database (#10937)

    * Ship SNMP Traps database

    * Update releasenotes/notes/SNMP---include-traps-db-7c44bd129daf7667.yaml

    Co-authored-by: Alexandre Yang <[email protected]>

    * Move snmp-traps dep to agent.rb

    Co-authored-by: Alexandre Yang <[email protected]>

commit cc93137
Author: Guillaume Fournier <[email protected]>
Date:   Tue Mar 29 17:57:57 2022 +0200

    [CWS] Add flowi offsets (#11452)

    * fix flowi offsets

    * fix family offset

commit 96c7183
Author: Lars Lehtonen <[email protected]>
Date:   Tue Mar 29 07:27:04 2022 -0700

    cmd/serverless: deprecation of aws session.New (#11477)

commit 8c4a85d
Author: Sylvain Afchain <[email protected]>
Date:   Tue Mar 29 15:24:32 2022 +0200

    [CWS] Introduce offset guesser as constant fetcher (#11343)

    * Introduce offset guesser

    * Use dedicated runtime compilation config parameter

    * Make constant tester not using test module

    * Exclude network and write user functions

commit 15973a9
Author: Sylvain Afchain <[email protected]>
Date:   Tue Mar 29 14:12:13 2022 +0200

    Use statsd client for ruleset_loaded test (#11486)

commit d52a6f6
Author: Len Gamburg <[email protected]>
Date:   Tue Mar 29 07:45:06 2022 -0400

    Switched to github.com/shirou/gopsutil/v3 (#11315)

    Switched to github.com/shirou/gopsutil/v3

commit f4b1539
Author: Kylian Serrania <[email protected]>
Date:   Tue Mar 29 07:38:57 2022 -0400

    [cmd/security-agent] Fix unused packages (#11481)

    Follow-up of #11023: remove unused imported packages to fix unit tests & builds in the pipeline.

commit ecf45b4
Author: Sylvain Afchain <[email protected]>
Date:   Tue Mar 29 11:11:16 2022 +0200

    [CWS] make use of statsd client interface (#11386)

    * Make use of statsd client interface

    * Add dentry erpc vs map test

commit c2c8c20
Author: Albert Vaca Cintora <[email protected]>
Date:   Mon Mar 28 16:41:28 2022 -0400

    Run Windows unit tests on PRs from forks (#11482)

commit 92d7c61
Author: Keisuke Umegaki <[email protected]>
Date:   Tue Mar 29 02:04:18 2022 +0900

    Graceful shutdown the Agent expvar server (#11023)

commit 85e83a1
Author: David Ortiz <[email protected]>
Date:   Mon Mar 28 17:29:28 2022 +0200

    [corechecks/helm] Add "release_state" service check (#11428)

    * [corechecks/helm/store] Keep track of latest revisions

    * [corechecks/helm] Add service check

    * Add release note

commit d0929b0
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Mar 28 10:42:15 2022 -0400

    Bump github.com/itchyny/gojq from 0.12.6 to 0.12.7 (#11461)

    * Bump github.com/itchyny/gojq from 0.12.6 to 0.12.7

    Bumps [github.com/itchyny/gojq](https://github.com/itchyny/gojq) from 0.12.6 to 0.12.7.
    - [Release notes](https://github.com/itchyny/gojq/releases)
    - [Changelog](https://github.com/itchyny/gojq/blob/main/CHANGELOG.md)
    - [Commits](itchyny/gojq@v0.12.6...v0.12.7)

    ---
    updated-dependencies:
    - dependency-name: github.com/itchyny/gojq
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    * Auto-generate go.sum and LICENSE-3rdparty.csv changes

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

commit 25812f3
Author: Keisuke Umegaki <[email protected]>
Date:   Mon Mar 28 23:28:56 2022 +0900

    Add `--service` to `stream-logs` (#11291)

    * add `--service` to `stream-logs`

    * add releasenote

commit fe563af
Author: Paul Cacheux <[email protected]>
Date:   Mon Mar 28 16:21:10 2022 +0200

    [CWS] fix position of sanity checks in relation to kernel version detection (#11474)

commit 4822fc4
Author: Vincent Boulineau <[email protected]>
Date:   Mon Mar 28 16:04:49 2022 +0200

    Fix AD port name interpolation by properly merging slices in workloadmeta (#11453)

commit 615b391
Author: Vickenty Fesunov <[email protected]>
Date:   Mon Mar 28 15:51:54 2022 +0200

    Reenable tags store (#11472)

    Follow-up for #11362.

commit 4743d24
Author: Ofek Lev <[email protected]>
Date:   Mon Mar 28 09:51:05 2022 -0400

    Upgrade `hatchling` Python build dependency (#11465)

commit fa780da
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Mar 28 09:50:11 2022 -0400

    Bump github.com/stretchr/testify from 1.7.0 to 1.7.1 in /pkg/util/log (#11344)

    * Bump github.com/stretchr/testify from 1.7.0 to 1.7.1 in /pkg/util/log

    Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.7.0 to 1.7.1.
    - [Release notes](https://github.com/stretchr/testify/releases)
    - [Commits](stretchr/testify@v1.7.0...v1.7.1)

    ---
    updated-dependencies:
    - dependency-name: github.com/stretchr/testify
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    * inv tidy-all

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Kylian Serrania <[email protected]>

commit ed15aed
Author: Kylian Serrania <[email protected]>
Date:   Mon Mar 28 09:45:32 2022 -0400

    [gitlab] Increase datadog-agent size delta threshold to 20MB (#11471)

    Increases the threshold of the check_pkg_size jobs to 20MB for the Datadog Agent release.

commit 705c42e
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Mar 28 09:10:47 2022 -0400

    Bump github.com/stretchr/testify from 1.7.0 to 1.7.1 in /pkg/util/scrubber (#11346)

    * Bump github.com/stretchr/testify in /pkg/util/scrubber

    Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.7.0 to 1.7.1.
    - [Release notes](https://github.com/stretchr/testify/releases)
    - [Commits](stretchr/testify@v1.7.0...v1.7.1)

    Fixes #11354, #11344

    ---
    updated-dependencies:
    - dependency-name: github.com/stretchr/testify
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    * inv tidy-all

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Kylian Serrania <[email protected]>

commit 3485422
Author: Pablo Baeyens <[email protected]>
Date:   Mon Mar 28 14:22:17 2022 +0200

    [pkg/otlp] Deprecate `metrics.report_quantiles` in favor of `metrics.summaries.mode` (#11451)

    * [pkg/otlp] Deprecate `metrics.report_quantiles` in favor of `metrics.summaries.mode`

    * Apply suggestions from code review

    Co-authored-by: maxime mouial <[email protected]>
    Co-authored-by: Kylian Serrania <[email protected]>

    Co-authored-by: maxime mouial <[email protected]>
    Co-authored-by: Kylian Serrania <[email protected]>

commit cb7be61
Author: Sylvain Baubeau <[email protected]>
Date:   Mon Mar 28 14:20:43 2022 +0200

    Check environment when run in container and export in status (#11434)

commit b42cd0a
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Mar 28 07:29:34 2022 -0400

    Bump github.com/stretchr/testify from 1.7.0 to 1.7.1 in /pkg/otlp/model (#11347)

    Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.7.0 to 1.7.1.
    - [Release notes](https://github.com/stretchr/testify/releases)
    - [Commits](stretchr/testify@v1.7.0...v1.7.1)

    ---
    updated-dependencies:
    - dependency-name: github.com/stretchr/testify
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Kylian Serrania <[email protected]>

commit 683f8a2
Author: Guillaume Fournier <[email protected]>
Date:   Mon Mar 28 13:13:30 2022 +0200

    prevent panic (#11469)

commit 52ebb9f
Author: Maxime mouial <[email protected]>
Date:   Fri Mar 25 13:48:33 2022 +0100

    Fix flaky test for IBM cloud
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lebauce lebauce lebauce approved these changes

Assignees
No one assigned
Labels
changelog/no-changelog team/agent-security
Projects
None yet
Milestone
7.36.0
Development

Successfully merging this pull request may close these issues.
2 participants
@paulcacheux @lebauce