-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(github): migrate the gitproviderreceiver githubscraper to github…
…receiver (#523)
- Loading branch information
Showing
44 changed files
with
32,909 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| include ../../Makefile.Common |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| # GitHub Receiver | ||
|
|
||
| <!-- status autogenerated section --> | ||
| | Status | | | ||
| | ------------- |-----------| | ||
| | Stability | [development]: metrics | | ||
| | Distributions | [liatrio] | | ||
| | Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Areceiver%2Fgithub) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Areceiver%2Fgithub) | | ||
|
|
||
| [development]: https://github.com/open-telemetry/opentelemetry-collector#development | ||
| [liatrio]: | ||
| <!-- end autogenerated section --> | ||
|
|
||
| The GitHub receiver receives data from [GitHub](https://github.com). As a | ||
| starting point it scrapes metrics from repositories but will be extended to | ||
| include traces and logs. | ||
|
|
||
| The current default set of metrics can be found in | ||
| [documentation.md](./documentation.md). | ||
|
|
||
| These metrics can be used as leading indicators ([capabilities][doracap]) | ||
| to the [DORA][dorafour] metrics; helping provide insight into modern-day | ||
| engineering practices. | ||
|
|
||
| [doracap]: https://dora.dev/capabilities/ | ||
| [dorafour]: https://dora.dev/guides/dora-metrics-four-keys/ | ||
|
|
||
| ## Getting Started | ||
|
|
||
| The collection interval is common to all scrapers and is set to 30 seconds by default. | ||
|
|
||
| > Note: Generally speaking, if the vendor allows for anonymous API calls, then you | ||
| > won't have to configure any authentication, but you may only see public repositories | ||
| > and organizations. You may run into significantly more rate limiting. | ||
| ```yaml | ||
| github: | ||
| collection_interval: <duration> #default = 30s recommended 300s | ||
| scrapers: | ||
| <scraper1>: | ||
| <scraper2>: | ||
| ... | ||
| ``` | ||
|
|
||
| A more complete example using the GitHub scrapers with authentication is as follows: | ||
|
|
||
| ```yaml | ||
| extensions: | ||
| bearertokenauth/github: | ||
| token: ${env:GH_PAT} | ||
|
|
||
| receivers: | ||
| github: | ||
| initial_delay: 1s | ||
| collection_interval: 60s | ||
| scrapers: | ||
| github: | ||
| metrics: | ||
| vcs.repository.contributor.count: | ||
| enabled: true | ||
| github_org: myfancyorg | ||
| search_query: "org:myfancyorg topic:o11yalltheway" #Recommended optional query override, defaults to "{org,user}:<github_org>" | ||
| endpoint: "https://selfmanagedenterpriseserver.com" | ||
| auth: | ||
| authenticator: bearertokenauth/github | ||
| service: | ||
| extensions: [bearertokenauth/github] | ||
| pipelines: | ||
| metrics: | ||
| receivers: [..., github] | ||
| processors: [] | ||
| exporters: [...] | ||
| ``` | ||
| A Grafana Dashboard exists on the marketplace for metrics from this receiver | ||
| and can be found | ||
| [here](https://grafana.com/grafana/dashboards/20976-engineering-effectiveness-metrics/). | ||
| ## Scraping | ||
| > Important: | ||
| > * The GitHub scraper does not emit metrics for branches that have not had | ||
| > changes since creation from the default branch (trunk). | ||
| > * Due to GitHub API limitations, it is possible for the branch time metric to | ||
| > change when rebases occur, recreating the commits with new timestamps. | ||
| <!-- TODO: Combine this documentation once the scraper code is restructured due scope change --> | ||
| For additional context on GitHub scraper limitations and inner workings please | ||
| see the [Scraping README][ghsread]. | ||
| [ghsread]: internal/scraper/githubscraper/README.md#github-limitations |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| // Copyright The OpenTelemetry Authors | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| package githubreceiver // import "github.com/liatrio/liatrio-otel-collector/receiver/githubreceiver" | ||
|
|
||
| import ( | ||
| "errors" | ||
| "fmt" | ||
|
|
||
| "go.opentelemetry.io/collector/component" | ||
| "go.opentelemetry.io/collector/confmap" | ||
| "go.opentelemetry.io/collector/receiver/scraperhelper" | ||
|
|
||
| "github.com/liatrio/liatrio-otel-collector/receiver/githubreceiver/internal" | ||
| "github.com/liatrio/liatrio-otel-collector/receiver/githubreceiver/internal/metadata" | ||
| ) | ||
|
|
||
| const ( | ||
| scrapersKey = "scrapers" | ||
| ) | ||
|
|
||
| // Config that is exposed to this github receiver through the OTEL config.yaml | ||
| type Config struct { | ||
| scraperhelper.ControllerConfig `mapstructure:",squash"` | ||
| Scrapers map[string]internal.Config `mapstructure:"scrapers"` | ||
| metadata.MetricsBuilderConfig `mapstructure:",squash"` | ||
| } | ||
|
|
||
| var _ component.Config = (*Config)(nil) | ||
| var _ confmap.Unmarshaler = (*Config)(nil) | ||
|
|
||
| // Validate the configuration passed through the OTEL config.yaml | ||
| func (cfg *Config) Validate() error { | ||
| if len(cfg.Scrapers) == 0 { | ||
| return errors.New("must specify at least one scraper") | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| // Unmarshal a config.Parser into the config struct. | ||
| func (cfg *Config) Unmarshal(componentParser *confmap.Conf) error { | ||
| if componentParser == nil { | ||
| return nil | ||
| } | ||
|
|
||
| // load the non-dynamic config normally | ||
| err := componentParser.Unmarshal(cfg, confmap.WithIgnoreUnused()) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| // dynamically load the individual collector configs based on the key name | ||
|
|
||
| cfg.Scrapers = map[string]internal.Config{} | ||
|
|
||
| scrapersSection, err := componentParser.Sub(scrapersKey) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| for key := range scrapersSection.ToStringMap() { | ||
| factory, ok := getScraperFactory(key) | ||
| if !ok { | ||
| return fmt.Errorf("invalid scraper key: %q", key) | ||
| } | ||
|
|
||
| collectorCfg := factory.CreateDefaultConfig() | ||
| collectorSection, err := scrapersSection.Sub(key) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| err = collectorSection.Unmarshal(collectorCfg) | ||
| if err != nil { | ||
| return fmt.Errorf("error reading settings for scraper type %q: %w", key, err) | ||
| } | ||
|
|
||
| cfg.Scrapers[key] = collectorCfg | ||
| } | ||
|
|
||
| return nil | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| // Copyright The OpenTelemetry Authors | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| package githubreceiver | ||
|
|
||
| import ( | ||
| "path/filepath" | ||
| "testing" | ||
| "time" | ||
|
|
||
| "github.com/stretchr/testify/assert" | ||
| "github.com/stretchr/testify/require" | ||
| "go.opentelemetry.io/collector/component" | ||
| "go.opentelemetry.io/collector/confmap" | ||
| "go.opentelemetry.io/collector/otelcol/otelcoltest" | ||
| "go.opentelemetry.io/collector/receiver/scraperhelper" | ||
|
|
||
| "github.com/liatrio/liatrio-otel-collector/receiver/githubreceiver/internal" | ||
| "github.com/liatrio/liatrio-otel-collector/receiver/githubreceiver/internal/metadata" | ||
| "github.com/liatrio/liatrio-otel-collector/receiver/githubreceiver/internal/scraper/githubscraper" | ||
| ) | ||
|
|
||
| func TestLoadConfig(t *testing.T) { | ||
| factories, err := otelcoltest.NopFactories() | ||
| require.NoError(t, err) | ||
|
|
||
| factory := NewFactory() | ||
| factories.Receivers[metadata.Type] = factory | ||
| // https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/33594 | ||
| // nolint:staticcheck | ||
| cfg, err := otelcoltest.LoadConfigAndValidate(filepath.Join("testdata", "config.yaml"), factories) | ||
|
|
||
| require.NoError(t, err) | ||
| require.NotNil(t, cfg) | ||
|
|
||
| assert.Len(t, cfg.Receivers, 2) | ||
|
|
||
| r0 := cfg.Receivers[component.NewID(metadata.Type)] | ||
| defaultConfigGitHubScraper := factory.CreateDefaultConfig() | ||
| defaultConfigGitHubScraper.(*Config).Scrapers = map[string]internal.Config{ | ||
| githubscraper.TypeStr: (&githubscraper.Factory{}).CreateDefaultConfig(), | ||
| } | ||
|
|
||
| assert.Equal(t, defaultConfigGitHubScraper, r0) | ||
|
|
||
| r1 := cfg.Receivers[component.NewIDWithName(metadata.Type, "customname")].(*Config) | ||
| expectedConfig := &Config{ | ||
| ControllerConfig: scraperhelper.ControllerConfig{ | ||
| CollectionInterval: 30 * time.Second, | ||
| InitialDelay: 1 * time.Second, | ||
| }, | ||
| Scrapers: map[string]internal.Config{ | ||
| githubscraper.TypeStr: (&githubscraper.Factory{}).CreateDefaultConfig(), | ||
| }, | ||
| } | ||
|
|
||
| assert.Equal(t, expectedConfig, r1) | ||
| } | ||
|
|
||
| func TestLoadInvalidConfig_NoScrapers(t *testing.T) { | ||
| factories, err := otelcoltest.NopFactories() | ||
| require.NoError(t, err) | ||
|
|
||
| factory := NewFactory() | ||
| factories.Receivers[metadata.Type] = factory | ||
| // https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/33594 | ||
| // nolint:staticcheck | ||
| _, err = otelcoltest.LoadConfigAndValidate(filepath.Join("testdata", "config-noscrapers.yaml"), factories) | ||
|
|
||
| require.Contains(t, err.Error(), "must specify at least one scraper") | ||
| } | ||
|
|
||
| func TestLoadInvalidConfig_InvalidScraperKey(t *testing.T) { | ||
| factories, err := otelcoltest.NopFactories() | ||
| require.NoError(t, err) | ||
|
|
||
| factory := NewFactory() | ||
| factories.Receivers[metadata.Type] = factory | ||
| // https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/33594 | ||
| // nolint:staticcheck | ||
| _, err = otelcoltest.LoadConfigAndValidate(filepath.Join("testdata", "config-invalidscraperkey.yaml"), factories) | ||
|
|
||
| require.Contains(t, err.Error(), "error reading configuration for \"github\": invalid scraper key: \"invalidscraperkey\"") | ||
| } | ||
|
|
||
| func TestConfig_Unmarshal(t *testing.T) { | ||
| type fields struct { | ||
| ControllerConfig scraperhelper.ControllerConfig | ||
| Scrapers map[string]internal.Config | ||
| MetricsBuilderConfig metadata.MetricsBuilderConfig | ||
| } | ||
|
|
||
| type args struct { | ||
| componentParser *confmap.Conf | ||
| } | ||
|
|
||
| tests := []struct { | ||
| name string | ||
| fields fields | ||
| args args | ||
| wantErr bool | ||
| }{ | ||
| { | ||
| name: "Empty Component Parser", | ||
| fields: fields{}, | ||
| args: args{componentParser: nil}, | ||
| wantErr: false, | ||
| }, | ||
| } | ||
| for _, test := range tests { | ||
| t.Run(test.name, func(t *testing.T) { | ||
| cfg := &Config{ | ||
| ControllerConfig: test.fields.ControllerConfig, | ||
| Scrapers: test.fields.Scrapers, | ||
| MetricsBuilderConfig: test.fields.MetricsBuilderConfig, | ||
| } | ||
| if err := cfg.Unmarshal(test.args.componentParser); (err != nil) != test.wantErr { | ||
| t.Errorf("Config.Unmarshal() error = %v, wantErr %v", err, test.wantErr) | ||
| } | ||
| }) | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| // Copyright The OpenTelemetry Authors | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| //go:generate ../../.tools/mdatagen metadata.yaml | ||
|
|
||
| package githubreceiver // import "github.com/liatrio/liatrio-otel-collector/receiver/githubreceiver" |
Oops, something went wrong.