-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DOCS] Updates add data content #81093
Changes from 5 commits
2938e3e
b862531
0457d09
7f06994
ad122d0
3717987
0be0960
1d6e585
b9e7e54
572a496
e845479
ec19d62
a3a1481
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,108 +1,75 @@ | ||
[[connect-to-elasticsearch]] | ||
== Add data to {kib} | ||
++++ | ||
<titleabbrev>Add data</titleabbrev> | ||
++++ | ||
|
||
To start working with your data in {kib}, you can: | ||
The fastest way to add data to {kib} is to use one of the built-in options. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. One comment about "add data to Kibana". This statement is a little technically misleading. Kibana does not store data so you never add data to kibana. You add data to Elasticsearch or the Elastic stack. Sometimes you add data by using kibana like with sample data or file upload. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thoughts on replacing "use one of our built in options" with "one of our many ingest options"? |
||
You can collect data from an app or service | ||
or upload a file that contains your data. If you're not ready to use your own data, | ||
add a sample data set and give {kib} a test drive. | ||
The built-in options are available on the home page. | ||
|
||
* Upload a CSV, JSON, or log file with the File Data Visualizer. | ||
[role="screenshot"] | ||
image::images/add-data-home.png[Built-in options for adding data to Kibana: Add data, Add Elastic Agent, Upload a file] | ||
|
||
* Upload geospatial data with the GeoJSON Upload feature. | ||
[float] | ||
[[add-data-tutorial-kibana]] | ||
=== Add data | ||
|
||
* Index logs, metrics, events, or application data by setting up a Beats module. | ||
Want to ingest logs, metrics, security, or application data? | ||
Install and configure a Beats data shipper or other module to periodically collect the data | ||
and send it to {es}. You can then use the pre-built dashboards to explore and analyze the data. | ||
|
||
* Connect {kib} with existing {es} indices. | ||
[role="screenshot"] | ||
image::images/add-data-tutorials.png[Add Data tutorials] | ||
|
||
If you're not ready to use your own data, you can add a <<get-started, sample data set>> | ||
to see all that you can do in {kib}. | ||
[discrete] | ||
=== Add Elastic Agent | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think the split between add data and add Elastic agent is fine now, because we have them in product. We'll have to work with @mostlyjason on when we can begin to default to agent and remove add data from the docs (similarly to how we did with Lens) and eventually, product There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These two things are very similar, integrations with Agent + Fleet are the next generation of modules. I wonder if we can spin this in a way with "get a sneak peak of where we're going with.." or something along those lines. |
||
|
||
[float] | ||
[[upload-data-kibana]] | ||
=== Upload a CSV, JSON, or log file | ||
beta[] *Elastic Agent* is a centralized way to set up your data integrations. | ||
With the *Fleet* UI, you can add | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should we just say with "Fleet" instead of "Fleet UI"? |
||
and manage integrations for popular services and platforms, providing | ||
an easy way to collect your data. The integrations | ||
ship with dashboards and visualizations, | ||
so you quickly get insights on your data. | ||
gchaps marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
experimental[] | ||
To get started, refer to | ||
{ingest-guide}/ingest-management-getting-started.html[Quick start: Get logs and metrics into the Elastic Stack]. | ||
|
||
To visualize data in a CSV, JSON, or log file, you can upload it using the File | ||
Data Visualizer. On the home page, click *Upload a file*, and | ||
then drag your file onto the *File Data Visualizer*. Alternatively, you can open | ||
it by navigating to *Machine Learning* from the side navigation and selecting | ||
[role="screenshot"] | ||
image::images/add-data-fleet.png[Add data using Fleet] | ||
|
||
*Data Visualizer*. | ||
NOTE: This feature is not intended for use as part of a | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not sure I like this note because it's a little misleading. Eventually it will be the main way to ingest data, but right now, it's not recommended in a production environment. Maybe qualify it by saying, "The beta release of this feature...." |
||
repeated production process, but rather for the initial exploration of your data. | ||
|
||
[role="screenshot"] | ||
image::images/ingest-data.png[File Data Visualizer on the home page] | ||
[discrete] | ||
[[upload-data-kibana]] | ||
=== Upload a file | ||
|
||
You can upload a file up to 100 MB. This value is configurable up to 1 GB in | ||
experimental[] If your data is in a CSV, JSON, or log file, you can upload it using the File | ||
Data Visualizer. You can upload a file up to 100 MB. This value is configurable up to 1 GB in | ||
<<kibana-ml-settings, Advanced Settings>>. | ||
|
||
[role="screenshot"] | ||
image::images/add-data-fv.png[File Data Visualizer] | ||
|
||
The File Data Visualizer uses the {ref}/ml-find-file-structure.html[find_file_structure API] to analyze | ||
the uploaded file and to suggest ingest pipelines and mappings for your data. | ||
For geospatial data, you can use the File Data Visualizer or the | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think you need to call out File Data Visualizer or GeoJSon Uploader. How about just changing to "See <<import-geospatial-data, Import geospatial data>> for uploading files with geospatial data." There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ++ |
||
GeoJSon Uploader, depending on the format of your data. See | ||
<<import-geospatial-data, Import geospatial data>> for details. | ||
|
||
NOTE: This feature is not intended for use as part of a | ||
repeated production process, but rather for the initial exploration of your data. | ||
|
||
[float] | ||
[[upload-geoipdata-kibana]] | ||
=== Upload geospatial data | ||
|
||
To visualize geospatial data in a point or shape file, you can upload it using the <<import-geospatial-data, GeoJSON Upload>> | ||
feature in Maps, and then use that data as a layer in a map. | ||
The data is also available for use in the broader Kibana ecosystem, for example, | ||
in visualizations and Canvas workpads. | ||
With GeoJSON Upload, you can upload a file up to 50 MB. | ||
|
||
[float] | ||
[[add-data-tutorial-kibana]] | ||
=== Index metrics, log, security, and application data | ||
|
||
The built-in data tutorials can help you quickly get up and running with | ||
metrics data, log analytics, security events, and application data. | ||
These tutorials walk you through installing and configuring a | ||
Beats data shipper to periodically collect and send data to {es}. | ||
You can then use the pre-built dashboards to explore and analyze the data. | ||
[discrete] | ||
=== Custom options for loading your data | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thoughts on using "Additional options for loading your data"? or "Options for loading your custom data" |
||
|
||
You access the tutorials from the home page. | ||
If a tutorial doesn’t exist for your data, go to the {beats-ref}/beats-reference.html[Beats overview] | ||
to learn about other data shippers in the Beats family. | ||
If the built-in options don't work for you, you can index your | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Similar to my comment above, thoughts on replacing "If the built-in options don't work for you" with "If one of our multiple ingest options don't work for you"? |
||
data into Elasticsearch with {ref}/getting-started-index.html[REST APIs] | ||
or https://www.elastic.co/guide/en/elasticsearch/client/index.html[client libraries]. | ||
After you add your data, you're required to create an <<index-patterns,index pattern>> to tell | ||
{kib} where to find the data. | ||
|
||
[role="screenshot"] | ||
image::images/add-data-tutorials.png[Add Data tutorials] | ||
|
||
|
||
[float] | ||
[[connect-to-es]] | ||
=== Connect with {es} indices | ||
|
||
To visualize data in existing {es} indices, you must | ||
create an index pattern that matches the names of the indices that you want to explore. | ||
When you add data with the File Data Visualizer, GeoJSON Upload feature, | ||
or built-in tutorial, an index pattern is created for you. | ||
|
||
. Open the main menu, then click *Stack Management > Index Patterns*. | ||
|
||
. Click *Create index pattern*. | ||
|
||
. Specify an index pattern that matches the name of one or more of your Elasticsearch indices. | ||
+ | ||
For example, an index pattern can point to your Apache data from yesterday, | ||
`filebeat-apache-4-3-2022`, or any index that matches the pattern, `filebeat-*`. | ||
Using a wildcard is the more popular approach. | ||
|
||
|
||
. Click *Next Step*, and then select the index field that contains the timestamp you want to use to perform time-based | ||
comparisons. | ||
+ | ||
Kibana reads the index mapping and lists all fields that contain a timestamp. If your | ||
index doesn't have time-based data, choose *I don't want to use the time filter*. | ||
+ | ||
You must select a time field to use global time filters on your dashboards. | ||
|
||
. Click *Create index pattern*. | ||
+ | ||
{kib} is now configured to access your {es} indices. | ||
You’ll see a list of fields configured for the matching index. | ||
You can designate your index pattern as the default by clicking the star icon on this page. | ||
+ | ||
When searching in *Discover* and creating visualizations, you choose a pattern | ||
from the index pattern menu to specify the {es} indices that contain the data you want to explore. | ||
* To add data for Elastic Observability, see {observability-guide}/add-observability-data.html[Send data to Elasticsearch]. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Probably another discussion, but a common naming convention and framework for ingestion "How to's" might make it easier to consume for our end users.
gchaps marked this conversation as resolved.
Show resolved
Hide resolved
|
||
* To add data for Elastic Security, see https://www.elastic.co/guide/en/security/current/ingest-data.html[Ingest data to Elastic Security]. | ||
gchaps marked this conversation as resolved.
Show resolved
Hide resolved
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Similar to the comment from Nathan, should we simply have "Add data" here to avoid confusion?