Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Label action #2

Closed
wants to merge 625 commits into from
Closed

Label action #2

wants to merge 625 commits into from

Conversation

axw
Copy link
Owner

@axw axw commented Sep 16, 2021

Motivation/summary

Checklist

I have considered changes for:

How to test these changes

Related issues

apmmachine and others added 30 commits May 31, 2021 06:49
* model/modeldecoder: add metric type and unit

* systemtest: test histogram metrics

* Update changelog

* systemtest: fix min docs expectation in test
tlscommon captures a new closure that assert.Equals can not compare
* Tidy up beater/config

- remove some "setup" methods, set default values before unpacking
- unexport DefaultKibanaAgentConfig
- de-pointer various config fields to avoid the need for nil-pointer
  checks, and rely on default values; remove "IsEnabled" methods in
  favour of setting default values for Enabled fields
- move various config types into their own files
- remove dead validator code

* beater/config: remove now-pointless test
* Update to elastic/beats@27e76c567711

- Update to Go 1.16.4
- Adapt to elastic/beats#25696

* golint fixes

* tests: chown go.sum files

go mod download will download all modules, even
if they are not used (e.g. dependencies of parts
of beats that are not dependencies of apm-server).
As such the command may need to update go.sum,
so make sure it is owned by the running user.

* Add gotestsum to go.mod
* disable kibana if running in managed mode

* remove agent_config service validation

selecting "all" for service name or environment in
the kibana ui marshals service.name and
service.environment as empty strings, which is
valid

* remove kibana api key

this was used temporarily to support central
config via kibana, which is not necessary for 7.14
* add units to metric duration fields

* Apply suggestions from code review

Co-authored-by: Andrew Wilkins <[email protected]>

* make update

* Add changelog

Co-authored-by: Andrew Wilkins <[email protected]>
* apmpackage: remove version directory

The version directory should be added when copying
to package-storage, but it is not needed in this repo.
The version is maintained in manifest.yml.

* apmpackage: update package version to 0.3.0

* Update apmpackage instructions and script

* Makefile: update *-package targets

* add docker entrypoint to copy package to am/n.n.n

* apmpackage/cmd/gen-package: no version in path

* make fmt
…ic#5351)

* Add additional logging when starting with data_streams.enabled

But no active libbeat management mode.

* make fmt update

* add assertion that the error being logged during startup is the one we expect

* make fmt

* Update comment and reorder imports

Co-authored-by: Andrew Wilkins <[email protected]>
Generate and use a cert/key pair for fleet-server,
and supply the CA certificate to the elastic-agent
container in tests.
…stic#5422)

* beater/authorization: introduce Resource type

Introduce a Resource type, which describes a specific
resource for which authorization is being queried. This
can later be used to restrict access to specific agents
and services.

If the supplied resource is the zero value, then the
query is interpreted as checking if the requester has
any access at all. If the resource is non-zero, then
the query is interpreted as checking if the requester
has access to that specific resource (agent/service).

* beater/authorization: add context functions

* beater/authorization: introduce AnonymousAuth

* beater: check authorization for agent+service
Check that rate limiting is wired up, rely on
unit tests to cover more specific scenarios.
Due to some changes in elastic-agent
(elastic/beats#24817), injection
of the apm-server binary became ineffective and we have
been running system tests with the published artifacts.

Artifacts (such as the apm-server) are now unpacked into
state/data/install/<artifact>. The state/data/install
directory is expected to be owned by the elastic-agent
user, so we can no longer bind mount the apm-server binary.
Instead, we now create a custom Docker image and copy in
the apm-server and apm-server.yml files.
Refactor processor/stream and beater/api/intake to
handle stream-level errors in the handler code. This
is a precursor to moving rate-limiting and "allowed
service" handling out of the processor/stream package.

The processor is now only responsible for recording
per-event errors. All stream-level errors will cause
the HandleStream method to return, and it is up to
the caller to handle them.

In addition to the above all JSON encoding of results
is now performed in the HTTP handler package, and
stream-level error metrics are no longer recorded in
processor/stream as they are redundant with other
`apm-server.server.response.*` metrics.
… metrics (elastic#5451)

* Remove the service from the dataset for all events except application metrics

* code review

* add changelog
Add support for a custom resource attribute,
`telemetry.sdk.elastic_export_timestamp`,
representing the time at which the agent sent
the event payload to the server. The server
uses this to adjust the timestamps of events,
to cater for end-user devices having incorrect
system clock settings.
Set compression level to 5 on cloud.
* Update to elastic/beats@2871d29be93a

* upgrade go version to 1.15.6

related to elastic#5374
* Introduce `apm-server.auth.*` config

Introduce the new AgentAuth config structure, which
holds API Key and secret token auth. Later we will
add "anonymous" auth here too.

We also introduce a new YAML naming scheme for the
config, `apm-server.auth.*`. The old config is
deprecated and copied across to the new config fields.

* docs: update config names

* apmpackage: update auth config keys
axw and others added 15 commits September 9, 2021 17:23
Migrate TLS system tests to Go. Only essentials are migrated,
some less important configurations (e.g. optional client certs)
are no longer tested.

We now capture net/http server error logs with logp. Previously
they were sent to the standard library's log package, which logp
disables by default.

Client certificate/key pair generated with following program:

---

package main

import (
        "crypto/rand"
        "crypto/rsa"
        "crypto/x509"
        "crypto/x509/pkix"
        "encoding/pem"
        "log"
        "math/big"
        "os"
        "time"
)

func main() {
        serialNumberLimit := new(big.Int).Lsh(big.NewInt(1), 128)
        serialNumber, err := rand.Int(rand.Reader, serialNumberLimit)
        if err != nil {
                log.Fatalf("Failed to generate serial number: %v", err)
        }
        notBefore := time.Now()
        notAfter := notBefore.Add(10 * 365 * 24 * time.Hour)
        template := x509.Certificate{
                SerialNumber: serialNumber,
                Subject: pkix.Name{
                        Organization: []string{"Acme Co"},
                },
                NotBefore: notBefore,
                NotAfter:  notAfter,

                KeyUsage:              x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment,
                ExtKeyUsage:           []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth},
                BasicConstraintsValid: true,
        }

        clientKey, err := rsa.GenerateKey(rand.Reader, 2048)
        if err != nil {
                log.Fatal(err)
        }
        derBytes, err := x509.CreateCertificate(rand.Reader, &template, &template, clientKey.Public(), clientKey)
        if err != nil {
                log.Fatal("failed to create certificate: %s", err)
        }
        certOut, err := os.Create("client_cert.pem")
        if err != nil {
                log.Fatalf("Failed to open client_cert.pem for writing: %v", err)
        }
        if err := pem.Encode(certOut, &pem.Block{Type: "CERTIFICATE", Bytes: derBytes}); err != nil {
                log.Fatalf("Failed to write data to client_cert.pem: %v", err)
        }
        if err := certOut.Close(); err != nil {
                log.Fatalf("Error closing client_cert.pem: %v", err)
        }
        log.Print("wrote client_cert.pem\n")

        keyOut, err := os.OpenFile("client_key.pem", os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)
        if err != nil {
                log.Fatalf("Failed to open client_key.pem for writing: %v", err)
                return
        }
        privBytes, err := x509.MarshalPKCS8PrivateKey(clientKey)
        if err != nil {
                log.Fatalf("Unable to marshal private key: %v", err)
        }
        if err := pem.Encode(keyOut, &pem.Block{Type: "PRIVATE KEY", Bytes: privBytes}); err != nil {
                log.Fatalf("Failed to write data to client_key.pem: %v", err)
        }
        if err := keyOut.Close(); err != nil {
                log.Fatalf("Error closing client_key.pem: %v", err)
        }
        log.Print("wrote client_key.pem\n")
}
* Add deprecation warnings about ES setup

Log warnings when index management config is explicitly
configured in standalone mode, and print a warning message
when users run `apm-server setup`.

Also, remove old check for `setup.dashboards` config.

* docs: deprecation notices for standalone indices

* Update getting-started-apm-server.asciidoc

* Update docs/getting-started-apm-server.asciidoc

Co-authored-by: Brandon Morelli <[email protected]>

* Update docs/getting-started-apm-server.asciidoc

Co-authored-by: Brandon Morelli <[email protected]>

* Update docs/getting-started-apm-server.asciidoc

Co-authored-by: Brandon Morelli <[email protected]>

* docs: mention apm integration in set up section

* Add qualifier to setup recommendation

* add deprecation notice to ingest pipeline page

Co-authored-by: Brandon Morelli <[email protected]>
…lastic#6142)

* [Automation] Update elastic stack version to 8.0.0-c0c740b6 for testing

* Adapt Kibana healthcheck to /api/status change

Co-authored-by: apmmachine <[email protected]>
Co-authored-by: Andrew Wilkins <[email protected]>
Move HTTP out of Transaction and Span, and into model.APMEvent.
We now set the same top-level `http.*` fields for both transactions
and spans. Some fields are copied for spans in addition, for
backwards compatibility.
To improve the onboarding experience a dedicated architecture markdown is added.
This adds the first diagram, outlining the ingest flow.
* Remove `http.request.socket` fields

Stop recording `http.request.socket.encrypted`, with no replacement.
This field is not used in the UI, and is redundant; the URL scheme
can be used instead.

Stop recording `http.request.socket.remote_address`. Instead, record
the direct network peer IP and port in `source.ip` and `source.port`.

* beater/request: reset all fields in Context.Reset
Remove the Parent field from model.Exception, and set it
only during transformation to beats events.
@axw axw force-pushed the label-action branch 2 times, most recently from ee8a16e to b7a9c7a Compare September 16, 2021 08:34
@axw axw added backport-foo bug Something isn't working and removed backport-foo labels Sep 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.