Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure Monitor Logs ingestion #29517

Merged
merged 16 commits into from
Jun 17, 2022
Merged
5 changes: 5 additions & 0 deletions eng/jacoco-test-coverage/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,11 @@
<artifactId>azure-messaging-webpubsub</artifactId>
<version>1.2.0-beta.1</version> <!-- {x-version-update;com.azure:azure-messaging-webpubsub;current} -->
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-monitor-ingestion</artifactId>
<version>1.0.0-beta.1</version> <!-- {x-version-update;com.azure:azure-monitor-ingestion;current} -->
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-monitor-opentelemetry-exporter</artifactId>
Expand Down
1 change: 1 addition & 0 deletions eng/versioning/version_client.txt
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ com.azure:azure-messaging-webpubsub;1.1.3;1.2.0-beta.1
com.azure:azure-mixedreality-authentication;1.2.3;1.3.0-beta.1
com.azure:azure-mixedreality-remoterendering;1.1.7;1.2.0-beta.1
com.azure:azure-monitor-opentelemetry-exporter;1.0.0-beta.5;1.0.0-beta.6
com.azure:azure-monitor-ingestion;1.0.0-beta.1;1.0.0-beta.1
com.azure:azure-monitor-query;1.0.7;1.1.0-beta.1
com.azure:azure-monitor-query-perf;1.0.0-beta.1;1.0.0-beta.1
com.azure:azure-perf-test-parent;1.0.0-beta.1;1.0.0-beta.1
Expand Down
10 changes: 10 additions & 0 deletions sdk/monitor/azure-monitor-ingestion/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Release History

## 1.0.0-beta.1 (Unreleased)
Version 1.0.0-beta.1 is a preview of our efforts in creating a client library for Azure Logs Ingestion that is
developer-friendly, idiomatic to the Java ecosystem, and as consistent across different languages and platforms as
possible. The principles that guide our efforts can be found in the
[Azure SDK Design Guidelines for Java](https://azure.github.io/azure-sdk/java_introduction.html).

## Features Added
- Initial release. Please see the README and wiki for information on using the new library.
179 changes: 179 additions & 0 deletions sdk/monitor/azure-monitor-ingestion/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
# Azure Monitor Ingestion client library for Java

The Azure Monitor Ingestion client library is used to send custom logs to [Azure Monitor][azure_monitor_overview].

This library allows you to send data from virtually any source to supported built-in tables or to custom tables
that you create in Log Analytics workspace. You can even extend the schema of built-in tables with custom columns.

## Getting started

### Prerequisites
- A [Java Development Kit (JDK)][jdk_link], version 8 or later.
- [Azure Subscription][azure_subscription]
- A [Data Collection Endpoint][data_collection_endpoint]
- A [Data Collection Rule][data_collection_rule]
- A [Log Analytics workspace][log_analytics_workspace]

### Include the package

[//]: # ({x-version-update-start;com.azure:azure-monitor-ingestion;current})
```xml
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-monitor-ingestion</artifactId>
<version>1.0.0-beta.1</version>
</dependency>
```

### Create the client

An authenticated client is required to upload logs to Azure Monitor. The library includes both synchronous and asynchronous forms of the clients. To authenticate,
the following examples use `DefaultAzureCredentialBuilder` from the [com.azure:azure-identity](https://search.maven.org/artifact/com.azure/azure-identity) package.

#### Authenticating using Azure Active Directory
You can authenticate with Azure Active Directory using the [Azure Identity library][azure_identity].
To use the [DefaultAzureCredential][DefaultAzureCredential] provider shown below, or other credential providers provided with the Azure SDK, please include the `azure-identity` package:
[//]: # ({x-version-update-start;com.azure:azure-identity;dependency})
```xml
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-identity</artifactId>
<version>1.5.2</version>
</dependency>
```
Set the values of the client ID, tenant ID, and client secret of the AAD application as environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_CLIENT_SECRET.

#### Synchronous Logs Ingestion client

```java readme-sample-createLogsIngestionClient
DefaultAzureCredential tokenCredential = new DefaultAzureCredentialBuilder().build();

LogsIngestionClient client = new LogsIngestionClientBuilder()
.endpoint("<data-collection-endpoint>")
.credential(tokenCredential)
.buildClient();
```

#### Asynchronous Logs Ingestion client

```java readme-sample-createLogsIngestionAsyncClient
DefaultAzureCredential tokenCredential = new DefaultAzureCredentialBuilder().build();

LogsIngestionAsyncClient asyncClient = new LogsIngestionClientBuilder()
.endpoint("<data-collection-endpoint>")
.credential(tokenCredential)
.buildAsyncClient();
```
### Uploading logs

For examples on how to upload logs, see the [Examples](#examples) section.
srnagar marked this conversation as resolved.
Show resolved Hide resolved

## Key concepts

### Data Collection Endpoint

Data Collection Endpoints (DCEs) allow you to uniquely configure ingestion settings for Azure Monitor. [This
article][data_collection_endpoint] provides an overview of data collection endpoints including their contents and structure and how you can create
and work with them.

### Data Collection Rule

Data collection rules define data collected by Azure Monitor and specify how and where that data should be sent or
stored. The REST API call must specify a DCR to use. A single DCE can support multiple DCRs, so you can specify a
lmolkova marked this conversation as resolved.
Show resolved Hide resolved
different DCR for different sources and target tables.

The DCR must understand the structure of the input data and the structure of the target table. If the two don't match,
it can use a transformation to convert the source data to match the target table. You may also use the transform to
filter source data and perform any other calculations or conversions.

For more details, refer to [Data collection rules in Azure Monitor](https://docs.microsoft.com/azure/azure-monitor/essentials/data-collection-rule-overview).

### Log Analytics Workspace Tables

Custom logs can send data to any custom table that you create and to certain built-in tables in your Log Analytics
workspace. The target table must exist before you can send data to it. The following built-in tables are currently supported:

- [CommonSecurityLog](https://docs.microsoft.com/azure/azure-monitor/reference/tables/commonsecuritylog)
- [SecurityEvents](https://docs.microsoft.com/azure/azure-monitor/reference/tables/securityevent)
- [Syslog](https://docs.microsoft.com/azure/azure-monitor/reference/tables/syslog)
- [WindowsEvents](https://docs.microsoft.com/azure/azure-monitor/reference/tables/windowsevent)

## Examples

- [Upload custom logs](#upload-custom-logs)
- [Upload custom logs with max concurrency](#upload-custom-logs-with-max-concurrency)

### Upload custom logs

```java readme-sample-uploadLogs
DefaultAzureCredential tokenCredential = new DefaultAzureCredentialBuilder().build();

LogsIngestionClient client = new LogsIngestionClientBuilder()
.endpoint("<data-collection-endpoint")
.credential(tokenCredential)
.buildClient();

List<Object> logs = getLogs();
UploadLogsResult result = client.upload("<data-collection-rule-id>", "<stream-name>", logs);
System.out.println("Logs upload result status " + result.getStatus());
```

### Upload custom logs with max concurrency
srnagar marked this conversation as resolved.
Show resolved Hide resolved
```java readme-sample-uploadLogsWithMaxConcurrency
DefaultAzureCredential tokenCredential = new DefaultAzureCredentialBuilder().build();

LogsIngestionClient client = new LogsIngestionClientBuilder()
.endpoint("<data-collection-endpoint")
.credential(tokenCredential)
.buildClient();

List<Object> logs = getLogs();
UploadLogsOptions uploadLogsOptions = new UploadLogsOptions()
.setMaxConcurrency(3);
UploadLogsResult result = client.upload("<data-collection-rule-id>", "<stream-name>", logs, uploadLogsOptions,
Context.NONE);
System.out.println("Logs upload result status " + result.getStatus());
```

## Troubleshooting

### Enabling Logging

Azure SDKs for Java offer a consistent logging story to help aid in troubleshooting application errors and expedite
their resolution. The logs produced will capture the flow of an application before reaching the terminal state to help
locate the root issue. View the [logging][logging] wiki for guidance about enabling logging.

## Next steps
More samples can be found [here][samples].

## Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License
Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution.
For details, visit [https://cla.microsoft.com](https://cla.microsoft.com).

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the
PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this
once across all repos using our CLA.

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact
[[email protected]](mailto:[email protected]) with any additional questions or comments.

<!-- LINKS -->
[azure_identity]: https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/identity/azure-identity
[azure_monitor_overview]: https://docs.microsoft.com/azure/azure-monitor/overview
[azure_subscription]: https://azure.microsoft.com/free
[cla]: https://cla.microsoft.com
[coc]: https://opensource.microsoft.com/codeofconduct/
[coc_faq]: https://opensource.microsoft.com/codeofconduct/faq/
[coc_contact]: mailto:[email protected]
[data_collection_endpoint]: https://docs.microsoft.com//azure/azure-monitor/essentials/data-collection-endpoint-overview
[data_collection_rule]: https://docs.microsoft.com/azure/azure-monitor/essentials/data-collection-rule-overview
[DefaultAzureCredential]: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/identity/azure-identity/README.md#defaultazurecredential
[jdk_link]: https://docs.microsoft.com/java/azure/jdk/?view=azure-java-stable
[log_analytics_workspace]: https://docs.microsoft.com//azure/azure-monitor/logs/log-analytics-workspace-overview
[logging]: https://docs.microsoft.com//azure/developer/java/sdk/logging-overview
[samples]: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/monitor/azure-monitor-ingestion/src/samples/java/com/azure/monitor/ingestion

![Impressions](https://azure-sdk-impressions.azurewebsites.net/api/impressions/azure-sdk-for-java%2Fsdk%2Fmonitor%2Fazure-monitor-ingestion%2FREADME.png)
107 changes: 107 additions & 0 deletions sdk/monitor/azure-monitor-ingestion/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
<!-- Copyright (c) Microsoft Corporation. All rights reserved.
Licensed under the MIT License. -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">

<modelVersion>4.0.0</modelVersion>

<groupId>com.azure</groupId>
<artifactId>azure-monitor-ingestion</artifactId>
<version>1.0.0-beta.1</version> <!-- {x-version-update;com.azure:azure-monitor-ingestion;current} -->

<packaging>jar</packaging>
<name>Microsoft Azure SDK for Azure Monitor Data Ingestion</name>
<description>This package contains Microsoft Azure Monitor Data Ingestion SDK.</description>
<parent>
<groupId>com.azure</groupId>
<artifactId>azure-client-sdk-parent</artifactId>
<version>1.7.0</version> <!-- {x-version-update;com.azure:azure-client-sdk-parent;current} -->
<relativePath>../../parents/azure-client-sdk-parent</relativePath>
</parent>
<url>https://github.com/Azure/azure-sdk-for-java</url>

<licenses>
<license>
<name>The MIT License (MIT)</name>
<url>http://opensource.org/licenses/MIT</url>
<distribution>repo</distribution>
</license>
</licenses>

<distributionManagement>
<site>
<id>azure-java-build-docs</id>
<url>${site.url}/site/${project.artifactId}</url>
</site>
</distributionManagement>

<scm>
<url>scm:git:https://github.com/Azure/azure-sdk-for-java</url>
<connection>scm:git:[email protected]:Azure/azure-sdk-for-java.git</connection>
<tag>HEAD</tag>
</scm>

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<legal>
<![CDATA[[INFO] Any downloads listed may be third party software. Microsoft grants you no rights for third party software.]]></legal>
</properties>

<developers>
<developer>
<id>microsoft</id>
<name>Microsoft</name>
</developer>
</developers>

<dependencies>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-core</artifactId>
<version>1.29.1</version> <!-- {x-version-update;com.azure:azure-core;dependency} -->
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-core-http-netty</artifactId>
<version>1.12.2</version> <!-- {x-version-update;com.azure:azure-core-http-netty;dependency} -->
</dependency>

<!-- test -->
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>5.8.2</version> <!-- {x-version-update;org.junit.jupiter:junit-jupiter-api;external_dependency} -->
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>5.8.2</version> <!-- {x-version-update;org.junit.jupiter:junit-jupiter-engine;external_dependency} -->
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-params</artifactId>
<version>5.8.2</version> <!-- {x-version-update;org.junit.jupiter:junit-jupiter-params;external_dependency} -->
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-core-test</artifactId>
<version>1.9.1</version> <!-- {x-version-update;com.azure:azure-core-test;dependency} -->
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-test</artifactId>
<version>3.4.17</version> <!-- {x-version-update;io.projectreactor:reactor-test;external_dependency} -->
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-identity</artifactId>
<version>1.5.2</version> <!-- {x-version-update;com.azure:azure-identity;dependency} -->
<scope>test</scope>
</dependency>
</dependencies>
</project>
Loading