Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure feedback #297

Merged
merged 4 commits into from
May 7, 2021
Merged

Azure feedback #297

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
295ebae
rename event_data --> message per observiq platform feedback
jsirianni May 7, 2021
a4d1298
update docs to reflect final field names and promotions
jsirianni May 7, 2021
671495f
add changelog for 0.13.20 and 0.13.21
jsirianni May 7, 2021
c13cf48
tag v0.13.21
jsirianni May 7, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,12 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## Unreleased
## [0.13.21] - 2021-05-07

### Changed
- Renamed Azure Event Hub event_data field to message [PR297](https://github.com/observIQ/stanza/pull/297)

## [0.13.20] - 2021-05-06

### Added
- Added flatten Operator [PR 286](https://github.com/observIQ/stanza/pull/286)
Expand Down
2 changes: 1 addition & 1 deletion cmd/stanza/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ go 1.14

require (
github.com/kardianos/service v1.2.0
github.com/observiq/stanza v0.13.20
github.com/observiq/stanza v0.13.21
github.com/observiq/stanza/operator/builtin/input/k8sevent v0.1.0
github.com/observiq/stanza/operator/builtin/input/windows v0.1.1
github.com/observiq/stanza/operator/builtin/output/elastic v0.1.0
Expand Down
8 changes: 5 additions & 3 deletions docs/operators/azure_event_hub_input.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,14 +40,16 @@ A list of potential keys and their purpose can be found [here](https://github.co
{
"timestamp": "2021-04-19T18:44:34.619Z",
"severity": 0,
"resource": {
"event_id": "fea3c182-00a6-4951-8f6f-9331031f978f"
},
"record": {
"event_data": "hello, world!",
"id": "28447813-ebac-4908-9bd4-c1a79f9f98ad",
"message": "hello, world!",
"system_properties": {
"x-opt-enqueued-time": "2021-04-19T18:44:34.619Z",
"x-opt-offset": 6120,
"x-opt-sequence-number": 51
}
}
}
```
```
41 changes: 20 additions & 21 deletions docs/operators/azure_log_analytics_input.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

The `azure_log_analytics_input` operator reads Azure Log Analytics logs from Azure Event Hub using.

The `azure_log_analytics_input` operator will use the `timegenerated` field as the parsed entry's timestamp. The label `azure_log_analytics_type` is derived from the log's `type` field. All other fields are added to the entry's record.
The `azure_log_analytics_input` operator will use the `timegenerated` field as the parsed entry's timestamp. The label `azure_log_analytics_table` is derived from the log's `type` field.

## Prerequisites

Expand Down Expand Up @@ -42,28 +42,27 @@ A list of potential fields for each Azure Log Analytics table can be found [here

```json
{
"timestamp": "2021-04-26T18:19:31.358Z",
"timestamp": "2021-05-07T14:01:26.105Z",
"severity": 0,
"labels": {
"azure_log_analytics_type": "ContainerLog",
},
"record": {
"_internal_workspaceresourceid": "/subscriptions/09373b6b-bc8b-4093-925d-eb87334c7d56/resourcegroups/bindplane-integration/providers/microsoft.operationalinsights/workspaces/bp-integration1",
"_resourceid": "/subscriptions/09373b6b-bc8b-4093-925d-eb87334c7d56/resourceGroups/devops/providers/Microsoft.ContainerService/managedClusters/log-analytics",
"computer": "aks-agentpool-39365618-vmss000001",
"containerid": "93f4537223ae81d1c39e12e684de25c65207549d1003d153356055a6137f82b0",
"logentry": "[SpanData(name='Recv.grpc.health.v1.Health.Check', context=SpanContext(trace_id=9d186b35325a4a9093242435948ada22, span_id=None, trace_options=TraceOptions(enabled=True), tracestate=None), span_id='bc4877b54bc8407b', parent_span_id=None, attributes={'component': 'grpc'}, start_time='2021-04-26T18:19:31.358155Z', end_time='2021-04-26T18:19:31.358229Z', child_span_count=0, stack_trace=None, time_events=[<opencensus.trace.time_event.TimeEvent object at 0x7f5d9fc53190>, <opencensus.trace.time_event.TimeEvent object at 0x7f5d9fc537d0>], links=[], status=None, same_process_as_parent_span=None, span_kind=1)]",
"logentrysource": "stdout",
"mg": "00000000-0000-0000-0000-000000000002",
"sourcesystem": "Containers",
"system_properties": {
"x-opt-enqueued-time": "2021-04-26T18:19:50.361Z",
"x-opt-offset": 14480072,
"x-opt-sequence-number": 1548
"containerlog": {
"_internal_workspaceresourceid": "/subscriptions/000-000/resourcegroups/integration/providers/microsoft.operationalinsights/workspaces/stanza",
"_resourceid": "/subscriptions/0000-000/resourceGroups/devops/providers/Microsoft.ContainerService/managedClusters/log-analytics",
"computer": "aks-agentpool-39365618-vmss000001",
"containerid": "f5376c6972ac19630113736e7d3bf359fe67065fde3831b0502cfee33470e68f",
"logentry": "request to api failed"
"logentrysource": "stdout",
"mg": "00000000-0000-0000-0000-000000000002",
"sourcesystem": "Containers",
"tenantid": "ae0db88b-40bb-40b7-b056-57980214436c",
"timegenerated": "2021-05-07T14:01:26.1050000Z",
"timeofcommand": "2021-05-07T14:01:29.0000000Z"
},
"tenantid": "ae0db88b-40bb-40b7-b056-57980214436c",
"timegenerated": "2021-04-26T18:19:31.3580000Z",
"timeofcommand": "2021-04-26T18:19:44.0000000Z"
"system_properties": {
"x-opt-enqueued-time": "2021-05-07T14:01:37.789Z",
"x-opt-offset": 150347296000,
"x-opt-sequence-number": 125576
}
}
}
```
```
2 changes: 1 addition & 1 deletion operator/builtin/input/azure/event_hub_parse.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ func parse(event azhub.Event, e *entry.Entry) error {
m := make(map[string]interface{})

if len(event.Data) != 0 {
m["event_data"] = string(event.Data)
m["message"] = string(event.Data)
}

if event.PartitionKey != nil {
Expand Down
12 changes: 6 additions & 6 deletions operator/builtin/input/azure/event_hub_parse_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ func TestParseEvent(t *testing.T) {
&entry.Entry{
Timestamp: testTime,
Record: map[string]interface{}{
"event_data": "event hub entry",
"message": "event hub entry",
"system_properties": map[string]interface{}{
"x-opt-sequence-number": &testSequenceNum,
"x-opt-enqueued-time": &testTime,
Expand Down Expand Up @@ -77,7 +77,7 @@ func TestParseEvent(t *testing.T) {
&entry.Entry{
Timestamp: testTime,
Record: map[string]interface{}{
"event_data": "hello world",
"message": "hello world",
"partition_key": &testPartitionKey,
"properties": map[string]interface{}{
"user": "stanza",
Expand Down Expand Up @@ -135,7 +135,7 @@ func TestPromoteTime(t *testing.T) {
&entry.Entry{
Timestamp: enqueuedTime,
Record: map[string]interface{}{
"event_data": "event hub entry",
"message": "event hub entry",
"system_properties": map[string]interface{}{
"x-opt-enqueued-time": &enqueuedTime,
},
Expand All @@ -157,7 +157,7 @@ func TestPromoteTime(t *testing.T) {
&entry.Entry{
Timestamp: ioTHubEnqueuedTime,
Record: map[string]interface{}{
"event_data": "event hub entry",
"message": "event hub entry",
"system_properties": map[string]interface{}{
"iothub-enqueuedtime": &ioTHubEnqueuedTime,
},
Expand All @@ -180,7 +180,7 @@ func TestPromoteTime(t *testing.T) {
&entry.Entry{
Timestamp: enqueuedTime,
Record: map[string]interface{}{
"event_data": "event hub entry",
"message": "event hub entry",
"system_properties": map[string]interface{}{
"x-opt-enqueued-time": &enqueuedTime,
"iothub-enqueuedtime": &ioTHubEnqueuedTime,
Expand All @@ -204,7 +204,7 @@ func TestPromoteTime(t *testing.T) {
&entry.Entry{
Timestamp: ioTHubEnqueuedTime,
Record: map[string]interface{}{
"event_data": "event hub entry",
"message": "event hub entry",
"system_properties": map[string]interface{}{
"x-opt-enqueued-time": &time.Time{},
"iothub-enqueuedtime": &ioTHubEnqueuedTime,
Expand Down