Merge branch 'main' into 5674-DaemonSet-deprecation

docs/serverless/AI-for-security/ai-assistant.mdx

@@ -95,10 +95,11 @@ Use these features to adjust and act on your conversations with AI Assistant:
     * **Copy to clipboard** (<DocIcon type="copyClipboard" title="Copy to clipboard" />): Copy the text to clipboard to paste elsewhere. Also helpful for resubmitting a previous prompt.
     * **Add to timeline** (<DocIcon type="timeline" title="Timeline" />): Add a filter or query to Timeline using the text. This button appears for particular queries in AI Assistant's responses.
 
-        <DocCallOut title="Tip">
-        Be sure to specify which language you'd like AI Assistant to use when writing a query. For example: "Can you generate an Event Query Language query to find four failed logins followed by a successful login?"
-        </DocCallOut>
+   Be sure to specify which language you'd like AI Assistant to use when writing a query. For example: "Can you generate an Event Query Language query to find four failed logins followed by a successful login?"
 
+<DocCallOut title="Tip">
+AI Assistant can remember particular information you tell it to remember. For example, you could tell it: "When anwering any question about srv-win-s1-rsa or an alert that references it, mention that this host is in the New York data center". This will cause it to remember the detail you highlighted.
+</DocCallOut>
 
 <div id="configure-ai-assistant"></div>
 
@@ -124,6 +125,7 @@ The **Settings** menu has the following tabs:
 
 ### Anonymization
 
+
 The **Anonymization** tab of the AI Assistant settings menu allows you to define default data anonymization behavior for events you send to AI Assistant. Fields with **Allowed** toggled on are included in events provided to AI Assistant. **Allowed** fields with **Anonymized** set to **Yes** are included, but with their values obfuscated.
 
 ![AI Assistant's settings menu, open to the Anonymization tab](../images/ai-assistant/-assistant-assistant-anonymization-menu.png)
@@ -138,7 +140,7 @@ When you include a particular event as context, such as an alert from the Alerts
 
 <DocCallOut template="beta" />
 
-The **Knowledge base** tab of the AI Assistant settings menu allows you to enable AI Assistant to answer questions about the Elastic Search Query Language ((esql)), and about alerts in your environment.
+The **Knowledge base** tab of the AI Assistant settings menu allows you to enable AI Assistant to answer questions about the Elastic Search Query Language (((esql))), and about alerts in your environment. To use it, you must <DocLink slug="/serverless/security/ml-requirements" text="enable machine learning"/>,
 
 ### Knowledge base for ((esql))
 
@@ -148,13 +150,11 @@ The **Knowledge base** tab of the AI Assistant settings menu allows you to enabl
 
 When this feature is enabled, AI Assistant can help you write an ((esql)) query for a particular use case, or answer general questions about ((esql)) syntax and usage. To enable AI Assistant to answer questions about ((esql)):
 
-* Enable the Elastic Learned Sparse EncodeR (ELSER). This model provides additional context to the third-party LLM. To learn more, refer to [Configure ELSER](((ml-docs))/ml-nlp-elser.html#download-deploy-elser).
-* Initialize the knowledge base by clicking **Initialize**.
-* Turn on the **Knowledge Base** option.
+* Turn on the knowledge base by clicking **Setup**. If the **Setup** button doesn't appear, knowledge base is already enabled.
 * Click **Save**. The knowledge base is now active. A quick prompt for ((esql)) queries becomes available, which provides a good starting point for your ((esql)) conversations and questions.
 
 <DocCallOut title="Note">
-To update AI Assistant so that it uses the most current ((esql)) documentation to answer your questions, click **Delete** next to **Knowledge Base**, and toggle the **Knowledge Base** slider off and then on.
+AI Assistant's knowledge base gets additional context from [Elastic Learned Sparse EncodeR (ELSER)](((ml-docs))/ml-nlp-elser.html#download-deploy-elser).
 </DocCallOut>
 
 ### Knowledge base for alerts
@@ -163,8 +163,8 @@ When this feature is enabled, AI Assistant will receive multiple alerts as conte
 
 To enable RAG for alerts:
 
-1. Turn on the **Alerts** setting.
-1. Use the slider to select the number of alerts to send to AI Assistant.
+* Turn on the knowledge base by clicking **Setup**. If the **Setup** button doesn't appear, knowledge base is already enabled.
+* Use the slider to select the number of alerts to send to AI Assistant. Click **Save**.
 
 ![AI Assistant's settings menu, open to the Knowledge base tab](../images/ai-assistant/assistant-kb-menu.png)
 

docs/serverless/AI-for-security/llm-performance-matrix.mdx

@@ -8,11 +8,11 @@ status: in review
 
 This table describes the performance of various large language models (LLMs) for different use cases in ((elastic-sec)), based on our internal testing. To learn more about these use cases, refer to <DocLink slug="/serverless/security/attack-discovery" text="Attack discovery"/> or <DocLink slug="/serverless/security/ai-assistant" text="AI Assistant"/>.
 
-|           **Feature**        | **Model**             |                    |                    |            |                 |
-|-------------------------------|-----------------------|--------------------|--------------------|------------|-----------------|
-|                              | **Claude 3: Opus** | **Claude 3.5: Sonnet** | **Claude 3: Haiku** | **GPT-4o** | **GPT-4 Turbo** |
-| **Assistant: general**       | Excellent              | Excellent          | Excellent          | Excellent  | Excellent       | 
-| **Assistant: ((esql)) generation** | Great            | Great              | Poor               | Excellent  | Poor            | 
-| **Assistant: alert questions** | Excellent            | Excellent          | Excellent          | Excellent  | Poor            |
-| **Attack discovery**          | Excellent             | Excellent          | Poor               | Poor       | Good            | 
+|           **Feature**        | **Model**             |                    |                    |            |                 |                   |           |  
+|-------------------------------|-----------------------|--------------------|--------------------|------------|-----------------|------------------|-----------|
+|                              | **Claude 3: Opus** | **Claude 3.5: Sonnet** | **Claude 3: Haiku** | **GPT-4o** | **GPT-4 Turbo** | Gemini 1.5 Pro | Gemini 1.5 Flash |
+| **Assistant: general**       | Excellent              | Excellent          | Excellent          | Excellent  | Excellent       | Excellent        | Excellent |
+| **Assistant: ((esql)) generation** | Great            | Great              | Poor               | Excellent  | Poor            | Good              | Poor    |
+| **Assistant: alert questions** | Excellent            | Excellent          | Excellent          | Excellent  | Poor            | Excellent         | Good    |
+| **Attack discovery**          | Excellent             | Excellent          | Poor               | Poor       | Good            | Great             | Poor     |
 

docs/serverless/AI-for-security/usecase-attack-disc-ai-assistant-incident-reporting.mdx

@@ -22,7 +22,7 @@ Attack discovery can detect a wide range of threats by finding relationships amo
 
 <DocImage url="images/attck-disc-11-alerts-disc.png" alt="An Attack discovery card showing an attack with 11 related alerts"/>
 
-In the example above, Attack discovery found connections between eleven alerts, and used them to identify and describe an attack chain.
+In the example above, Attack discovery found connections between nine alerts, and used them to identify and describe an attack chain.
 
 After Attack discovery outlines your threat landscape, use Elastic AI Assistant to quickly analyze a threat in detail. 
 
@@ -31,8 +31,10 @@ After Attack discovery outlines your threat landscape, use Elastic AI Assistant
 
 From a discovery on the Attack discovery page, click **View in AI Assistant** to start a chat that includes the discovery as context. 
 
-AI Assistant can quickly compile essential data and provide suggestions to help you generate an incident report and plan an effective response. You can ask it to provide relevant data or answer questions, such as “How can I remediate this threat?” or “What ((esql)) query would isolate actions taken by this user?” 
+<DocImage url="images/attck-disc-remediate-sodinokibi.gif" alt="A dialogue with AI Assistant that has the attack discovery as context"/>
+
 
+AI Assistant can quickly compile essential data and provide suggestions to help you generate an incident report and plan an effective response. You can ask it to provide relevant data or answer questions, such as “How can I remediate this threat?” or “What ((esql)) query would isolate actions taken by this user?” 
 
 <DocImage url="images/attck-disc-esql-query-gen-example.png" alt="An AI Assistant dialogue in which the user asks for a purpose-built ES|QL query" />
 
@@ -41,14 +43,18 @@ The image above shows an ((esql)) query generated by AI Assistant in response to
 At any point in a conversation with AI Assistant, you can add data, narrative summaries, and other information from its responses to ((elastic-sec))'s case management system to generate incident reports. 
 
 <div id="use-case-incident-reporting-create-a-case-using-ai-assistant"/>
-## Create a case using AI Assistant
+## Generate reports
 
 From the AI Assistant dialog window, click **Add to case** (<DocIcon type="addDataApp" title="Add data" />) next to a message to add the information in that message to a <DocLink slug="/serverless/security/cases-overview" text="case"/>. Cases help centralize relevant details in one place for easy sharing with stakeholders.
 
 If you add a message that contains a discovery to a case, AI Assistant automatically adds the attack summary and all associated alerts to the case. You can also add AI Assistant messages that contain remediation steps and relevant data to the case. 
 
 <div id="use-case-incident-reporting-translate"/>
 ## Translate incident information to a different human language using AI Assistant
+
+<DocImage url="images/attck-disc-translate-japanese.png" alt="An AI Assistant dialogue in which the assistant translates from English to Japanese" />
+
+
 AI Assistant can translate its findings into other human languages, helping to enable collaboration among global security teams, and making it easier to operate within multilingual organizations. 
 
 After AI Assistant provides information in one language, you can ask it to translate its responses. For example, if it provides remediation steps for an incident, you can instruct it to “Translate these remediation steps into Japanese.” You can then add the translated output to a case. This can help team members receive the same information and insights regardless of their primary language.

docs/serverless/ingest/auto-import.mdx

@@ -0,0 +1,87 @@
+---
+slug: /serverless/security/automatic-import
+title: Automatic Import
+description: Use Automatic Import to quickly normalize and ingest third-party data.
+tags: [ 'serverless', 'security', 'how-to' ]
+status: in review
+---
+
+<DocBadge template="technical preview" />
+
+<DocCallOut title="Technical preview" color="warning">
+This feature is in technical preview. It may change in the future, and you should exercise caution when using it in production environments. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of GA features.
+</DocCallOut>
+
+Automatic Import helps you quickly parse, ingest, and create [ECS mappings](https://www.elastic.co/elasticsearch/common-schema) for data from sources that don't yet have prebuilt Elastic integrations. This can accelerate your migration to ((elastic-sec)), and help you quickly add new data sources to an existing SIEM solution in ((elastic-sec)). Automatic Import uses a large language model (LLM) with specialized instructions to quickly analyze your source data and create a custom integration. 
+
+While Elastic has 400+ [prebuilt data integrations](((integrations-docs))), Automatic Import helps you extend data coverage to other security-relevant technologies and applications. Elastic integrations (including those created by Automatic Import) normalize data to [the Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html), which creates uniformity across dashboards, search, alerts, machine learning, and more. 
+
+
+<DocCallOut title="Tip">
+Click [here](https://elastic.navattic.com/automatic-import) to access an interactive demo that shows the feature in action, before setting it up yourself.
+</DocCallOut>
+
+<DocCallOut title="Requirements">
+
+- A working <DocLink slug="/serverless/security/connect-to-bedrock" text="Amazon Bedrock connector"/>. Automatic Import currently works with all variants of Claude 3. Other models are not supported in this technical preview, but will be supported in future versions.
+- A [Security Analytics Complete](https://www.elastic.co/pricing/serverless-security) subscription.
+- A sample of the data you want to import, in JSON or NDJSON format. 
+
+</DocCallOut>
+
+<DocCallOut color="warning" title="Important">
+Using Automatic Import allows users to create new third-party data integrations through the use of third-party generative AI models (“GAI models”). Any third-party GAI models that you choose to use are owned and operated by their respective providers. Elastic does not own or control these third-party GAI models, nor does it influence their design, training, or data-handling practices. Using third-party GAI models with Elastic solutions, and using your data with third-party GAI models is at your discretion. Elastic bears no responsibility or liability for the content, operation, or use of these third-party GAI models, nor for any potential loss or damage arising from their use. Users are advised to exercise caution when using GAI models with personal, sensitive, or confidential information, as data submitted may be used to train the models or for other purposes. Elastic recommends familiarizing yourself with the development practices and terms of use of any third-party GAI models before use.
+
+You are responsible for ensuring that your use of Automatic Import complies with the terms and conditions of any third-party platform you connect with.
+</DocCallOut>
+
+
+## Create a new custom integration
+
+1. In ((elastic-sec)), click **Add integrations**.
+2. Under **Can't find an integration?** click **Create new integration**.
+
+    <DocImage url="images/auto-import-create-new-integration-button.png" alt="The Integrations page with the Create new integration button highlighted" />
+
+3. Click **Create integration**.
+4. Select an <DocLink slug="/serverless/security/connect-to-bedrock" text="Amazon Bedrock connector"/>. 
+5. Define how your new integration will appear on the Integrations page by providing a **Title**, **Description**, and **Logo**.  Click **Next**.
+6. Define your integration's package name, which will prefix the imported event fields. 
+7. Define your **Data stream title**, **Data stream description**, and **Data stream name**. These fields appear on the integration's configuration page to help identify the data stream it writes to.
+8. Select your [**Data collection method**](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html). This determines how your new integration will ingest the data (for example, from an S3 bucket, an HTTP endpoint, or a file stream).
+9. Upload a sample of your data in JSON or NDJSON format. Make sure to include all the types of events that you want the new integration to handle. 
+<DocCallOut title="Best practices for sample data"> 
+- The file extension (`.JSON` or `.NDJSON`) must match the file format.
+- Only the first 10 events in the sample are analyzed. In this technical preview, additional data is truncated. 
+- Ensure each JSON or NDJSON object represents an event, and avoid deeply nested object structures. 
+- The more variety in your sample, the more accurate the pipeline will be (for example, include 10 unique log entries instead of the same type of entry 10 times).
+- Ideally, each field name should describe what the field does.
+</DocCallOut>
+10. Click **Analyze logs**, then wait for processing to complete. This may take several minutes.
+11. After processing is complete, the pipeline's field mappings appear, including ECS and custom fields.
+
+    <DocImage url="images/auto-import-review-integration-page.png" alt="The Automatic Import Review page showing proposed field mappings" />
+
+12. (Optional) After reviewing the proposed pipeline, you can fine-tune it by clicking **Edit pipeline**. Refer to the [((elastic-sec)) ECS reference](https://www.elastic.co/guide/en/security/current/siem-field-reference.html) to learn more about formatting field mappings. When you're satisfied with your changes, click **Save**. 
+
+    <DocImage url="images/auto-import-edit-pipeline.gif" alt="A gif showing the user clicking the edit pipeline button and viewing the ingest pipeline flyout" />
+
+13. Click **Add to Elastic**. After the **Success** message appears, your new integration will be available on the Integrations page. 
+
+    <DocImage url="images/auto-import-success-message.png" alt="The Automatic Import success message" />
+
+14. Click **Add to an agent** to deploy your new integration and start collecting data, or click **View integration** to view detailed information about your new integration. 
+
+<DocCallOut title="Note">
+Once you've added an integration, you can't edit any details other than the ingest pipeline, which you can edit by going to **Project Settings → Stack Management → Ingest Pipelines**. 
+</DocCallOut>
+
+<DocCallOut title="Tip">
+You can use the <DocLink slug="/serverless/security/data-quality-dash" text="Data Quality dashboard"/> to check the health of your data ingest pipelines and field mappings.
+</DocCallOut>
+
+
+
+
+
+

docs/serverless/serverless-security.docnav.json

@@ -76,6 +76,9 @@
         {
           "slug": "/serverless/security/threat-intelligence",
           "classic-sources": [ "enSecurityEsThreatIntelIntegrations" ]
+        }, 
+        {
+          "slug": "/serverless/security/automatic-import"
         }
       ]
     },