Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

introduce per document streamprocessing #2

Draft
wants to merge 306 commits into
base: feature/synthtrace-stream-processing-improvements-workers
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
306 commits
Select commit Hold shift + click to select a range
4835a5d
[Security Solution] remove unused BrowserField references (#130206)
Apr 14, 2022
1f05cd6
[Discover] Discover Classic View Filter In/Out placement when `trunca…
kertal Apr 14, 2022
9e900a1
[Ingest Pipelines] add tests for grok processor (#130123)
alisonelizabeth Apr 14, 2022
0ed2f5b
[Cases] Use the bulk create attachment endpoint on the UI (#130124)
cnasikas Apr 14, 2022
f5bb693
[Cases] Limit the number of alerts that can be attached to a case (#1…
cnasikas Apr 14, 2022
f9dd129
skip failing test suite (#130190)
kibanamachine Apr 14, 2022
708d6d0
Further improvements to synthtrace DSL and reintroducing workers to t…
Mpdreamz Apr 14, 2022
20a031b
stagin
Mpdreamz Mar 8, 2022
cf028cd
staging
Mpdreamz Mar 9, 2022
2507c72
staging
Mpdreamz Mar 29, 2022
d4b03fb
stage work on service latency aggregator
Mpdreamz Apr 13, 2022
26294cd
move service latency to metric_aggregate_double
Mpdreamz Apr 13, 2022
9cca330
default_metric is required on metric_aggregate_double
Mpdreamz Apr 13, 2022
d5d798f
[Lens] Improve Toolbar Responsiveness (#130175)
MichaelMarcialis Apr 14, 2022
0a800c4
[data views] Type cleanup, monitoring uses data views service instead…
mattkime Apr 14, 2022
bde1ba5
fix paths in sample code in docs (#130215)
mattkime Apr 14, 2022
9c360f5
[data view mgmt] fix flaky scripted fields test (#126992)
mattkime Apr 14, 2022
fd4bea9
[Security Solution][Endpoint] Add UT tests for fleet UI extensions co…
paul-tavares Apr 14, 2022
3c3e775
remove conflict markers in performance_playwright.sh
Apr 14, 2022
07341a5
[ML] Resizable/Collapsible Top Influencers section (#130018)
darnautov Apr 14, 2022
6adc94b
bump `tsd` to `0.20.0` (#130237)
pgayvallet Apr 14, 2022
1dce1d8
[CI] Balance/split CI groups and remove docker-specific group (#130181)
brianseeders Apr 14, 2022
6f6a332
Fix roles for discover ccs test (#130204)
liza-mae Apr 14, 2022
9c8440a
Update host filter list (#130095)
pjhampton Apr 14, 2022
4f6239d
Exclude Malwarescore + Ransomware EP alerts from DRule telemetry. (#1…
pjhampton Apr 14, 2022
4aedd6c
fix paths in sample code in docs (#130278)
mattkime Apr 14, 2022
fae0afb
[Fleet] Small clean up on add agent instructions (#130193)
jen-huang Apr 14, 2022
19ed205
Make add agent step statuses more consistent (#130198)
jen-huang Apr 14, 2022
3210f11
skip failing performance suite (#130287)
Apr 14, 2022
69847eb
Rename trigger actions to state (#130067)
JiaweiWu Apr 14, 2022
b95c4f8
skip failing test suite (#129219)
kibanamachine Apr 14, 2022
8ef5691
[ML] Fixes alignment of Anomaly Explorer swim lane annotations label …
peteharverson Apr 14, 2022
baad4d6
[Reporting/Docs] User guidance to enable reporting on Cloud (memory c…
jloleysens Apr 14, 2022
0cae95c
Decrypt monitor before trigger (#130252)
shahzad31 Apr 14, 2022
52b2ad2
[artifacts] Extract default i18n messages (#128947)
jbudz Apr 14, 2022
d513ffc
[CI] Resolve ES Snapshot manifest url at the beginning of the build (…
brianseeders Apr 14, 2022
0300e02
[package testing] Show kibana logs on exit (#128619)
jbudz Apr 14, 2022
f0553d3
[APM] Fix broken e2e tests (#129475)
gbamparop Apr 14, 2022
72c40f2
[Lens] reduce sync bundle size and update package size limit (#130156)
drewdaemon Apr 14, 2022
8619986
[Security Solution] Adds rule preview error message for users without…
dplumlee Apr 14, 2022
5eeeb95
[DOCS] Add final case APIs (#130161)
lcawl Apr 14, 2022
a871a29
[Workplace Search] Add new Custom Connectors (Network Drive, Outlook,…
Apr 14, 2022
f08ca7d
[bazel] Skip installing to NPM global if Bazel exists (#128638)
Apr 14, 2022
ef7f28e
[kbn-pm] Link to documentation for Vault setup (#130341)
Apr 14, 2022
874bfb3
broke apart files (#130345)
FrankHassanabad Apr 14, 2022
0b2da10
chore(NA): downgrades rules nodejs into v4.7.0 (#130347)
mistic Apr 15, 2022
3627b86
[Screenshotting] Documentation (#129794)
dokmic Apr 15, 2022
b4bd924
[Fleet] Improve Fleet preconfiguration tests (#130300)
nchaulet Apr 15, 2022
c266af6
fix(NA): use correct rule on yarn_install force at @kbn/pm (#130375)
mistic Apr 15, 2022
92db72b
[Fleet] Add missing step showing K8 instructions for managed enrollme…
criamico Apr 15, 2022
e61167d
[Security Solution][Admin][Policy Details] Policy details tabs mainta…
parkiino Apr 15, 2022
a72d157
[ML] Fixes cell popover in data frame analytics results grid (#130293)
peteharverson Apr 15, 2022
145d827
[Security Solution] Fix broken host link in Cases (#128917)
kevinlog Apr 15, 2022
42dcc03
[build] Use new artifacts API to download {metric,file}beat dependenc…
jbudz Apr 15, 2022
54dffea
[bazel/packages] find typescript projects in new bazel package dirs (…
Apr 15, 2022
425c46a
[generate/package] support specifying full location of generated pack…
Apr 15, 2022
fc7d385
skip flaky suite (#129754)
Apr 15, 2022
cf84437
[DOCS] Updates Discover docs to show Document Explorer (#129549)
gchaps Apr 15, 2022
620b759
[artifacts] Skip publish if untracked branch (#130337)
jbudz Apr 15, 2022
b01c4bc
[Detection Rules] Add 8.2 rules (#130338)
terrancedejesus Apr 15, 2022
27ed2f4
[jest/config-check] treat all package dirs as roots (#130403)
Apr 15, 2022
a77f340
[Security Solution] Updated security_solution plugin code owners for …
YulNaumenko Apr 15, 2022
3ebe072
Fix nav-kibana-dev.docnav.json (#130392)
gtback Apr 15, 2022
5a694fa
Deletes section on editing saved objects from the UI (#130408)
TinaHeiligers Apr 15, 2022
a54d0d6
Update event rendered view action column width to account for session…
kqualters-elastic Apr 16, 2022
e47bf4b
[Shared UX] Adopt multi-package strategy - ExitFullScreenButton (#130…
clintandrewhall Apr 16, 2022
bd8171c
[plugins] use module ids to import across plugins
Apr 3, 2022
3730dd0
fix all violations
Apr 16, 2022
ac5aca4
[Maps] Use content-encoding header from ES for mvt response (#130417)
nickpeihl Apr 16, 2022
1bfeab7
[Cloud Posture] add findings distribution bar (#129639)
orouz Apr 17, 2022
c1d6690
[Cloud Posture] refactoring cspm plugin apis to internal (#130434)
kfirpeled Apr 17, 2022
48ca660
chore(NA): use new and more performant BuildBuddy servers (#130350)
mistic Apr 18, 2022
1119b59
[Screenshotting] Move the capture settings from reporting plugin (#13…
tsullivan Apr 18, 2022
3803209
Fix failing test: Jest Tests.src/plugins/charts/public/services/activ…
alexwizp Apr 18, 2022
968dffa
[User Experience] fix javascript errors href (#130280)
dominiqueclarke Apr 18, 2022
123b231
Update dependency color to ^4.2.3 (main) (#130079)
alexwizp Apr 18, 2022
64befe3
[Uptime] Decrypt Synthetics Monitors on sync and ensure API key is sp…
dominiqueclarke Apr 18, 2022
509f1da
Revert "Fix failing test: Jest Tests.src/plugins/charts/public/servic…
alexwizp Apr 18, 2022
d55c7b3
Add transforms creation to the CSP plugin initialization (#129905)
eyalkraft Apr 18, 2022
884b477
fix multi index pattern load bug (#130428)
flash1293 Apr 18, 2022
5ac1548
Apply doc changes (#130316)
mikecote Apr 18, 2022
c64ebc2
Add require_alias event log when indexing documents (#130332)
mikecote Apr 18, 2022
4cccb69
Unskip managed configuration integration test (#130331)
mikecote Apr 18, 2022
7cf6bf6
Use consistent examples in Task Manager documentation (#130317)
mikecote Apr 18, 2022
a14bcaa
[Cloud Posture] use index with keyword mapping (#130456)
orouz Apr 18, 2022
f962ff2
Rename the term "execution" in config to "run" (#130172)
ersin-erdal Apr 18, 2022
eaf319c
[Workplace Search] Fix broken feedback link (#130475)
byronhulcher Apr 18, 2022
0fdccf7
Update object types for SharePoint Online external connector (#130478)
byronhulcher Apr 18, 2022
34dfeeb
[Security Solution][Endpoint] Fix test stability and un-skip flaky te…
paul-tavares Apr 18, 2022
20f05e2
Add eslint rule to support breaking up packages (#130483)
Apr 18, 2022
21f8999
Add deprecated telemetry (#130458)
Apr 18, 2022
188c420
Fix wrong config in comments (#130378)
Apr 18, 2022
08d61f3
[8.2.1][Security Solution][Session view] fix full screen session view…
Apr 18, 2022
a256227
[kbn/pm] use stable module ids in dist (#130497)
Apr 18, 2022
fb87699
[functional-tests] TimePicker optimizations (#130200)
brianseeders Apr 18, 2022
8364fc7
[CI] Use spot instances for default cigroups in PR CI (#130476)
brianseeders Apr 18, 2022
6a4eb48
[CI] Split alerting-api-integration tests into separate cigroups (#13…
brianseeders Apr 18, 2022
fa89c45
[Maps] fix vector tile load errors not displayed in legend (#130395)
nreese Apr 18, 2022
2d18850
[Visualize] Adds a deprecation warning to the pie app (#130447)
stratoula Apr 19, 2022
f2af25d
[Step 3] Cleanup charts plugin (#130132)
alexwizp Apr 19, 2022
34d5ef8
Add link to troubleshooting guide in confirm data copy (#130420)
jen-huang Apr 19, 2022
74fdae1
Add links to metricbeat module docs (#130519)
matschaffer Apr 19, 2022
a843db3
[Uptime] Update monitor saved object mappings (#130433)
shahzad31 Apr 19, 2022
e8f59a5
[Osquery] Update eslint config (#129637)
patrykkopycinski Apr 19, 2022
7d20ffb
Remove “New!” badge from Curations UI (#130499)
scottybollinger Apr 19, 2022
ef56370
[APM] Use doc_count instead of hits to get service map throughput (#1…
MiriamAparicio Apr 19, 2022
b62c815
[Infra UI] Correctly interpret the `resetting` and `reverting` job st…
weltenwort Apr 19, 2022
813193d
no longer use an enum for home's FeatureCatalogueCategory (#129599)
pgayvallet Apr 19, 2022
720eb23
[Cases] Add delete action to user comments in Cases (#130254)
academo Apr 19, 2022
ccdec85
[KibanaReact] Deprecate KibanaPageTemplate (#130007)
Apr 19, 2022
ab589f1
[Unified search] Allows comma delimiter on the filters multiple selec…
stratoula Apr 19, 2022
44c0cda
add wildcard to CSP transform source index (#130543)
eyalkraft Apr 19, 2022
a961a96
[Cases] Multiple alerts user action on the UI (#130265)
cnasikas Apr 19, 2022
d38333d
Add max configuration for minimumScheduleInterval and number of actio…
mikecote Apr 19, 2022
627c594
Rename "scheduled actions" to "generated actions" (#130315)
mikecote Apr 19, 2022
00effe7
[Stack Monitoring] use CCS constant in place of '*' string (#130466)
neptunian Apr 19, 2022
186f169
Update dependency @elastic/charts to v46 (main) (#130264)
renovate[bot] Apr 19, 2022
3d11fac
[Security Solution] Getting started - fix images in dark mode (#130071)
stephmilovic Apr 19, 2022
810e73c
[Security Solution] Sourcerer docs (#129670)
stephmilovic Apr 19, 2022
a89df70
[Visualize] Adds a deprecation warning to the timelion app (#130525)
stratoula Apr 19, 2022
36be216
Fix the flaky test case (#129237)
ersin-erdal Apr 19, 2022
00a73aa
Fix for the failing health_route workload calculation. (#130136)
ersin-erdal Apr 19, 2022
db7237f
Add `KibanaThemeProvider` to `GlobalSearchBar` and `Newsfeed` plugins…
thompsongl Apr 19, 2022
c8ef572
[Visualize] Adds a functional test for partial rows (#130528)
stratoula Apr 19, 2022
38ba64c
Don't reset retryCount if a left value is not retryable_es_client_err…
rudolf Apr 19, 2022
89ac623
[ML] Increase trial max ram for ml nodes (#130224)
sabarasaba Apr 19, 2022
2507afc
[APM] Callout flashes on service inventory page (#130493)
cauemarcondes Apr 19, 2022
66d04fc
[Synthtrace] Renaming run files (#130505)
cauemarcondes Apr 19, 2022
269ed6b
[Uptime] Synthetics tests - close Monitor Management tour (#130490)
dominiqueclarke Apr 19, 2022
cc34125
[security solution][Endpoint] Refactor host isolation exceptions list…
paul-tavares Apr 19, 2022
3b4806d
break `custom_integrations` -> `presentation_utils` dependency (#129908)
Dosant Apr 19, 2022
f2a1f1b
[ML] Trained model saved objects api tests (#130115)
jgowdyelastic Apr 19, 2022
f54b1bd
Unskip flaky task management test and remove async from unecessary fu…
mikecote Apr 19, 2022
8291010
[Trigger Actions Ui] Add alerts table registry initial type for colum…
academo Apr 19, 2022
c96f2a6
[Lens] accuracy mode for Top Values (#129220)
drewdaemon Apr 19, 2022
1d1903e
[Fleet] Do not compare package version in preconfiguration tests (#13…
nchaulet Apr 19, 2022
0045559
use elastic-charts axis calculation (#130429)
flash1293 Apr 19, 2022
250e3f9
[Uptime] Unskip flaky tests (#130059)
dominiqueclarke Apr 19, 2022
153cd55
[bootstrap] normalize paths written to base tsconfig file (#130569)
Apr 19, 2022
37414f2
[Visualize] Unksip test (#130529)
flash1293 Apr 19, 2022
87fd4a4
[Fleet] Update the link to logstash ssl documentation (#130510)
nchaulet Apr 19, 2022
06740ff
[Enterprise Search] Add documentation link for BYOEI (#130494)
scottybollinger Apr 19, 2022
2a78f35
break out parts of @kbn/dev-utils (#130509)
Apr 19, 2022
30db753
[8.2.1] [Session view] Add loading state for session view when fetchi…
Apr 19, 2022
b7947b7
Fix failing test: Jest Tests.src/plugins/charts/public/services/activ…
alexwizp Apr 19, 2022
d8b5f37
fix missing testing dep for windows dev (#130579)
Apr 19, 2022
2569ca6
[Docs] Update rules list screenshots in docs (#130487)
Zacqary Apr 19, 2022
26b8d28
Update dependency @elastic/charts to v46.0.1 (main) (#130571)
renovate[bot] Apr 19, 2022
6730484
[Security Solution][Codeowners] - Update threat intel ownership to pl…
yctercero Apr 19, 2022
b352e4a
[Session view] Beta Badge (#130512)
mitodrummer Apr 19, 2022
e3e4d11
Fixes embeddable factory default telemetry fn (#130522)
cqliu1 Apr 19, 2022
2a7b219
[cft] Bump Kibana memory to 2GB (#130600)
jbudz Apr 19, 2022
1264fc9
[jest/configs] fix checks for nested packages (#130583)
Apr 19, 2022
0541d2c
[CI] Move apps functional tests from ciGroup1 to ciGroup28 (#130595)
brianseeders Apr 19, 2022
5872b5e
[CI] Split dashboard feature controls tests into two cigroups (#130416)
brianseeders Apr 19, 2022
cde4288
Change updateObjectsSpaces API to prevent multiple objects w/ same or…
jportner Apr 19, 2022
834b178
uptime - add monitor management docs link (#130558)
dominiqueclarke Apr 19, 2022
2ec9f22
Fix overlooked term "execution" in kibana-docker allowed list (#130605)
ersin-erdal Apr 19, 2022
c0ab947
[RAM] Make RuleStatusDropdown shareable (#130205)
Zacqary Apr 19, 2022
64d7bcc
[Dashboard] [Telemetry] Report panels in dashboards by type (#130166)
Heenawter Apr 19, 2022
320df4b
[Uptime] update monitor management tour content (#130601)
dominiqueclarke Apr 19, 2022
a9a6df3
[CI] Move detection engine create exceptions tests to ciGroup14 (#130…
brianseeders Apr 19, 2022
9430e41
fix(NA): removes carriage return character when reading bazel tools c…
mistic Apr 19, 2022
9ed9c02
[kbn/pm] reduce responsibilities (#130592)
Apr 20, 2022
18aa549
[Uptime] monitor management - adjust sort field map for inline monito…
dominiqueclarke Apr 20, 2022
b658a4a
fixing codeowners for cloud-security-posture team after group renamin…
kfirpeled Apr 20, 2022
36a4c99
[Form lib] Add "updateFieldValues" to the form hook API (#130544)
sebelga Apr 20, 2022
cc6057f
update copy per feedback from kellyemurphy (#130587)
jloleysens Apr 20, 2022
29f8d77
[Cloud Posture] Use latest findings and PIT in dashboard (#130540)
ari-aviran Apr 20, 2022
0355c2e
[Discover] Improve document explorer callout (#129972)
dimaanj Apr 20, 2022
a8b0149
[Security Solution][Exceptions] - Fix modal closure on addition of fi…
yctercero Apr 20, 2022
285e336
[Cloud Posture] Create concrete csp rules post create package (#130120)
CohenIdo Apr 20, 2022
9d6aa99
Unit Tests for new o11y rules page (#128576)
mgiota Apr 20, 2022
45851e1
[Security Solution] backfill tests for hiding artifact cards (#130515)
joeypoon Apr 20, 2022
fbc8a71
[EBT] Split `@elastic/analytics` package (#130574)
afharo Apr 20, 2022
5e83dad
[i18n] Integrate 8.2.0 Translations (#130668)
Bamieh Apr 20, 2022
88825ef
[Exploratory view] Increase chart height and reduce paddings (#130645)
shahzad31 Apr 20, 2022
26f042f
fix(NA): update codeowners for kibana-cloud-security-posture team on …
mistic Apr 20, 2022
0646a97
[generate/package] fix rootDir for generated jest.config.js (#130634)
Apr 20, 2022
b3538f8
Bump hapi packages (#130672)
jportner Apr 20, 2022
1dcdc7b
[DOCS] Adds advanced settings for data views (#130294)
KOTungseth Apr 20, 2022
5f003a4
[artifacts] Publish kibana cloud docker context (#130415)
jbudz Apr 20, 2022
e220260
skip suite failing es promotion (#130692)
Apr 20, 2022
3648f74
[EBT] Rename `esOrgId` to `cloudId` (#130663)
afharo Apr 20, 2022
830d231
[es-snapshots] Fix promotion not triggering due to missing dependency…
brianseeders Apr 20, 2022
473781a
add kibana.json json-schema for VSCode users (#130625)
Apr 20, 2022
8c991cf
[Stack Monitoring] Convert APM routes to TypeScript (#130097)
Kerry350 Apr 20, 2022
7df6a0d
[Cloud Posture] add findings group by resource table (#130455)
orouz Apr 20, 2022
37bfb50
[generate/package] run ESLint and auto-fix generated files (#130676)
Apr 20, 2022
146a3e9
[Synthetics] Skip unecessary keys (#130667)
shahzad31 Apr 20, 2022
88c8ed9
[Maps] support term joins for Elasticsearch document source with vect…
nreese Apr 20, 2022
0196a8f
[Session View] Update alert icon color to danger (#130612)
Apr 20, 2022
4b6c19a
skip flaky suite (#130694)
Apr 20, 2022
0d233e9
[ML] Allow to edit job selection on data frame analytics results and …
alvarezmelissa87 Apr 20, 2022
8be1744
[Uptime][Monitor Management] Report all service locations for Run Onc…
awahab07 Apr 20, 2022
a4a0822
[Synthetics] Rename uptime plugin to synthetics (#130037)
shahzad31 Apr 20, 2022
decb95b
[CI] Move some test suites out of ciGroup4 (#130708)
brianseeders Apr 20, 2022
d0643e0
[CI] Move most tests from oss ciGroup2 to ciGroup11 (#130712)
brianseeders Apr 20, 2022
5beae64
[CI] Move most tests from oss ciGroup6 to ciGroup5 (#130709)
brianseeders Apr 20, 2022
0082e0c
[artifact testing] Bump VM boot timeout (#130735)
jbudz Apr 20, 2022
3b37b27
[Controls] Collect Telemetry (#130498)
ThomThomson Apr 20, 2022
1303044
[RAM] Add docs for Snooze UX (#130297)
Zacqary Apr 20, 2022
bb9fc4b
[Infra Monitoring]: add rac to feature privileges for logs and metric…
mgiota Apr 21, 2022
6e8d198
[Dev Tools] Fix cat APIs returning as escaped string (#130638)
mibragimov Apr 21, 2022
7af6915
[APM] Progressive fetching (experimental) (#127598)
dgieselaar Apr 21, 2022
6447923
[Console] Fix condition auto-completion for templates (#126881)
mibragimov Apr 21, 2022
58acda6
unblocks main (#130771)
MadameSheema Apr 21, 2022
b5322c7
[ILM] Remove `axios` dependency in tests (#129009)
sabarasaba Apr 21, 2022
275ce8f
[Maps] update MapLibre to 2.1.latest (#127501)
nreese Apr 21, 2022
5517598
remove autorefresh button from o11y rules page (#130526)
mgiota Apr 21, 2022
a3fd86c
[Workplace Search] New Network Drive, Outlook, Teams, and Zoom Server…
byronhulcher Apr 21, 2022
1a1adb9
consume shareable status dropdown from triggers_actions_ui plugin (#1…
mgiota Apr 21, 2022
502a00b
[Console] Add Kibana APIs Support (#128562)
mibragimov Apr 21, 2022
fb33187
Adds documentation and improves migrations failing on timeouts while …
TinaHeiligers Apr 21, 2022
dccca6b
[Fleet] Remove legacy component templates on package install (#130758)
hop-dev Apr 21, 2022
f41dc1f
Revert "[Console] Add Kibana APIs Support (#128562)"
jbudz Apr 21, 2022
4e9a1d5
Revert "consume shareable status dropdown from triggers_actions_ui pl…
jbudz Apr 21, 2022
0923ff6
[DOCS] Fix section titles in Cases APIs (#130737)
lcawl Apr 21, 2022
8609411
[DOCS] Add trained models to ML sync API (#130626)
lcawl Apr 21, 2022
9ce1546
Attempt fixing flaky recovery context test by adding missing promises…
mikecote Apr 21, 2022
43f0312
[Lens] Fixes flakiness on the visualization switcher (#130783)
stratoula Apr 21, 2022
cb48e11
[Enterprise Search] Rewire overview designs. (#130066)
efegurkan Apr 21, 2022
ef3bd04
[Security Solution][Endpoint] Port trusted apps to `ArtifactlsListPag…
ashokaditya Apr 21, 2022
ee1e2bb
[Fleet] Fix aligment logstash copy API key tooltip (#130815)
nchaulet Apr 21, 2022
995e63a
[Security Solution] [Investigations] [Tech Debt] removes deepEqual ch…
andrew-goldstein Apr 21, 2022
90c3202
update code onwer (#130818)
shahzad31 Apr 21, 2022
4d78a77
Update typescript (main) (#130764)
renovate[bot] Apr 21, 2022
5a86421
[Controls] Clear range/time slider selections when field changes (#12…
Heenawter Apr 21, 2022
e3d221e
[CI] Bump kibana-buildkite-library for getPrChanges fix (#130824)
brianseeders Apr 21, 2022
5fe9437
[ML] Fixing management app docs links (#130776)
jgowdyelastic Apr 21, 2022
47918ae
[Console] Add Kibana APIs Support (#130816)
alisonelizabeth Apr 21, 2022
fa73237
[jest] try running unit tests in parallel (#130823)
Apr 21, 2022
7c01257
[Unified Search] Move autocomplete logic to unified search plugin (#1…
nlatipov Apr 22, 2022
706d926
[test-failed]: Chrome UI Functional Tests1.test/functional/apps/disco…
DianaDerevyankina Apr 22, 2022
a02c00b
Change ContextContainer to lazily initialize providers (#129896)
pgayvallet Apr 22, 2022
0073a06
Soc dashboard - UI (#128087)
angorayc Apr 22, 2022
a927a95
bump rum agent version (#130765)
Apr 22, 2022
fb6f64f
Fix Tests.test/functional/apps/visualize/_tsvb_time_series.ts (#130599)
alexwizp Apr 24, 2022
780333b
[Cloud Posture] Create data yaml based on rules instances (#130671)
CohenIdo Apr 25, 2022
63e95ab
[Fleet] Fallback to bundled packages on GET package route (#129999)
joshdover Apr 25, 2022
b61ef24
fix jest test search_bar (#130876)
nlatipov Apr 25, 2022
154208b
Merge branch 'main' into feature/synthtrace-timeseries
Mpdreamz Apr 25, 2022
b589464
post merge updates
Mpdreamz Apr 25, 2022
0147c68
Pick fields from ApmFields
Mpdreamz May 3, 2022
2f0b604
include service.environment and transaction.type as dimensions
Mpdreamz May 3, 2022
f4d8b17
rename service.latency to transaction.duration.aggregate
Mpdreamz May 3, 2022
6e3f5cd
removed unnecessary intermediate method createFieldsFromState() in fa…
Mpdreamz May 3, 2022
55a52f1
ensure we flush previously held range if current event exceeds max wi…
Mpdreamz May 3, 2022
480bbe4
move the processor.name to 'metric' for now
Mpdreamz May 3, 2022
9fb7905
clean aggregator stream with wildcard for namespace
Mpdreamz May 3, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
stage work on service latency aggregator
  • Loading branch information
Mpdreamz committed Apr 14, 2022
commit d4b03fbb8e3bfbfd628a351fcd1b73e74b57b3c9
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/

import { random } from 'lodash';
import { Client } from '@elastic/elasticsearch';
import { ApmFields, Observer } from '../apm_fields';
import { Fields } from '../../entity';
import { StreamAggregator } from '../../stream_aggregator';

interface LatencyState {
count: number;
mean: number;
timestamp: number;
}

export type ServiceFields = Fields &
Partial<{
'timestamp.us'?: number;
'ecs.version': string;
'metricset.name': string;
observer: Observer;
'processor.event': string;
'processor.name': string;
'service.name': string;
'service.version': string;
'service.environment': string;
'service.latency_mean': number;
'service.latency_count': number;
}>;

export class ServiceLatencyAggregator implements StreamAggregator<ApmFields> {
public readonly name;

constructor() {
this.name = 'service-latency';
}

getDataStreamName(): string {
return 'metrics-apm.service';
}

getMappings(): Record<string, any> {
return {
properties: {
'@timestamp': {
type: 'date',
format: 'date_optional_time||epoch_millis',
},
message: {
type: 'wildcard',
},
service: {
type: 'object',
properties: {
name: {
type: 'keyword',
time_series_dimension: true,
},
latency_mean: {
type: 'double',
time_series_metric: 'gauge',
},
latency_count: {
type: 'long',
time_series_metric: 'gauge',
},
},
},
},
};
}

getDimensions(): string[] {
return ['service.name'];
}

getWriteTarget(document: Record<string, any>): string | null {
if (!document.processor?.event) {
throw Error("'processor.event' is not set on document, can not determine target index");
}
const eventType = document.processor.event;
if (eventType === 'service') return 'metrics-apm.service-default';
return null;
}

private state: Record<string, LatencyState> = {};

private processedComponent: number = 0;

process(event: ApmFields): Fields[] | null {
if (event['processor.event'] !== 'transaction') return null;
if (!event['@timestamp']) return null;

const service = event['service.name']!;
if (!this.state[service]) {
this.state[service] = {
count: 0,
mean: 0,
timestamp: event['@timestamp'],
};
}
const duration = Number(event['transaction.duration.us']);
const count = ++this.state[service].count;
const differential = (duration - this.state[service].mean) / count;
this.state[service].mean = this.state[service].mean + differential;

if (Object.keys(this.state).length === 1000) {
return this.createFieldsFromState();
}

const diff = Math.abs(event['@timestamp'] - this.state[service].timestamp);
if (diff >= 1000 * 60) {
const fields = this.createServiceFields(service);
delete this.state[service];
return [fields];
}
return null;
}

flush(): Fields[] {
return this.createFieldsFromState();
}

private createFieldsFromState(): ServiceFields[] {
const fields = Object.keys(this.state).map((service) => this.createServiceFields(service));
this.state = {};
return fields;
}

private createServiceFields(service: string): ServiceFields {
this.processedComponent = ++this.processedComponent % 1000;
const component = Date.now() % 100;
return {
'@timestamp':
this.state[service].timestamp + random(0, 100) + component + this.processedComponent,
'metricset.name': 'service',
'processor.event': 'service',
'service.name': service,
'service.latency_mean': this.state[service].mean,
'service.latency_count': this.state[service].count,
};
}

async bootstrapElasticsearch(esClient: Client): Promise<void> {}
}
Original file line number Diff line number Diff line change
@@ -58,7 +58,7 @@ export class ApmSynthtraceEsClient {
return info.version.number;
}

async clean() {
async clean(dataStreams?: string[]) {
return this.getWriteTargets().then(async (writeTargets) => {
const indices = Object.values(writeTargets);
this.logger.info(`Attempting to clean: ${indices}`);
@@ -69,7 +69,7 @@ export class ApmSynthtraceEsClient {
logger: this.logger,
});
}
for (const name of indices) {
for (const name of indices.concat(dataStreams ?? [])) {
const dataStream = await this.client.indices.getDataStream({ name }, { ignore: [404] });
if (dataStream.data_streams && dataStream.data_streams.length > 0) {
this.logger.debug(`Deleting datastream: ${name}`);
@@ -165,7 +165,7 @@ export class ApmSynthtraceEsClient {
await this.logger.perf('enumerate_scenario', async () => {
// @ts-ignore
// We just want to enumerate
for await (item of sp.streamToDocumentAsync(sp.toDocument, dataStream)) {
for await (item of sp.streamToDocumentAsync((e) => sp.toDocument(e), dataStream)) {
if (yielded === 0) {
options.itemStartStopCallback?.apply(this, [item, false]);
yielded++;
@@ -185,7 +185,7 @@ export class ApmSynthtraceEsClient {
flushBytes: 500000,
// TODO https://github.com/elastic/elasticsearch-js/issues/1610
// having to map here is awkward, it'd be better to map just before serialization.
datasource: sp.streamToDocumentAsync(sp.toDocument, dataStream),
datasource: sp.streamToDocumentAsync((e) => sp.toDocument(e), dataStream),
onDrop: (doc) => {
this.logger.info(JSON.stringify(doc, null, 2));
},
@@ -229,6 +229,7 @@ export class ApmSynthtraceEsClient {
description: `Mappings for ${datastreamName}-*`,
},
});
this.logger.info(`Created mapping component template for ${datastreamName}-*`);

await this.client.cluster.putComponentTemplate({
name: `${datastreamName}-settings`,
@@ -245,6 +246,7 @@ export class ApmSynthtraceEsClient {
description: `Settings for ${datastreamName}-*`,
},
});
this.logger.info(`Created settings component template for ${datastreamName}-*`);

await this.client.indices.putIndexTemplate({
name: `${datastreamName}-index_template`,
@@ -253,5 +255,8 @@ export class ApmSynthtraceEsClient {
composed_of: [`${datastreamName}-mappings`, `${datastreamName}-settings`],
priority: 500,
});
this.logger.info(`Created index template for ${datastreamName}-*`);

await this.client.indices.createDataStream({ name: datastreamName + '-default' });
}
}
112 changes: 0 additions & 112 deletions packages/elastic-apm-synthtrace/src/lib/stream_aggregator.ts
Original file line number Diff line number Diff line change
@@ -8,24 +8,6 @@

import { Client } from '@elastic/elasticsearch';
import { ApmFields, Fields } from '..';
import { Observer } from './apm/apm_fields';
interface LatencyState {
count: number;
mean: number;
}

export type ServiceFields = Fields &
Partial<{
'timestamp.us'?: number;
'ecs.version': string;
'metricset.name': string;
observer: Observer;
'processor.event': string;
'processor.name': string;
'service.name': string;
'service.version': string;
'service.environment': string;
}>;

export interface StreamAggregator<TFields extends Fields = ApmFields> {
name: string;
@@ -45,97 +27,3 @@ export interface StreamAggregator<TFields extends Fields = ApmFields> {
getMappings(): Record<string, any>;
}

export class ServiceLatencyGenerator implements StreamAggregator<ApmFields> {
public readonly name;

constructor() {
this.name = 'service-latency';
}

getDataStreamName(): string {
return 'metrics-apm.service';
}
getMappings(): Record<string, any> {
return {
properties: {
'@timestamp': {
type: 'date',
format: 'date_optional_time||epoch_millis',
},
message: {
type: 'wildcard',
},
service: {
type: 'object',
properties: {
name: {
type: 'keyword',
time_series_dimension: true,
},
},
},
},
};
}

getDimensions(): string[] {
return ['service.name'];
}

getWriteTarget(document: Record<string, any>): string | null {
if (!document.processor?.event) {
throw Error("'processor.event' is not set on document, can not determine target index");
}
const eventType = document.processor.event;
if (eventType === 'service') return 'metrics-apm.service-default';
return null;
}

private state: Record<string, LatencyState> = {};
private timestamp: number | null = null;

process(event: ApmFields): Fields[] | null {
if (event['processor.event'] !== 'transaction') return null;
if (!event['@timestamp']) return null;
if (!this.timestamp) this.timestamp = event['@timestamp'];

const service = event['service.name']!;
if (!this.state[service]) {
this.state[service] = { count: 0, mean: 0 };
}
const duration = Number(event['transaction.duration.us']);
const count = ++this.state[service].count;
const differential = (duration - this.state[service].mean) / count;
this.state[service].mean = this.state[service].mean + differential;
if (Object.keys(this.state).length === 1000) {
return this.createFieldsFromState();
}
const diff = Math.abs(event['@timestamp'] - this.timestamp);
if (diff >= 1000 * 60) {
return this.createFieldsFromState();
}
return null;
}

flush(): Fields[] {
return this.createFieldsFromState();
}

private createFieldsFromState(): ServiceFields[] {
const fields = Object.keys(this.state).map((service) => {
return {
'@timestamp': this.timestamp!,
'metricset.name': 'service',
'processor.event': 'service',
'service.name': service,
'service.latency_mean': this.state[service].mean,
'service.latency_count': this.state[service].count,
};
});
this.state = {};
this.timestamp = null;
return fields;
}

async bootstrapElasticsearch(esClient: Client): Promise<void> {}
}
4 changes: 0 additions & 4 deletions packages/elastic-apm-synthtrace/src/lib/stream_processor.ts
Original file line number Diff line number Diff line change
@@ -144,10 +144,6 @@ export class StreamProcessor<TFields extends Fields = ApmFields> {
}
}

public aggregatorBootstrap(): Array<(esClient: Client) => Promise<void>> {
return this.streamAggregators.map((a) => a.bootstrapElasticsearch);
}

private calculateFlushAfter(eventDate: number | null, order: 'asc' | 'desc') {
if (order === 'desc') {
return moment(eventDate).subtract(this.intervalAmount, this.intervalUnit).valueOf();
8 changes: 7 additions & 1 deletion packages/elastic-apm-synthtrace/src/scripts/run.ts
Original file line number Diff line number Diff line change
@@ -14,6 +14,8 @@ import { startLiveDataUpload } from './utils/start_live_data_upload';
import { parseRunCliFlags } from './utils/parse_run_cli_flags';
import { getCommonServices } from './utils/get_common_services';
import { ApmSynthtraceKibanaClient } from '../lib/apm/client/apm_synthtrace_kibana_client';
import { StreamAggregator } from '../lib/stream_aggregator';
import {ServiceLatencyAggregator} from "../lib/apm/aggregators/service_latency_aggregator";

function options(y: Argv) {
return y
@@ -186,8 +188,9 @@ yargs(process.argv.slice(2))
await apmEsClient.updateComponentTemplates(runOptions.numShards);
}

const aggregators: StreamAggregator[] = [new ServiceLatencyAggregator()];
if (argv.clean) {
await apmEsClient.clean();
await apmEsClient.clean(aggregators.map((a) => a.getDataStreamName() + '-default'));
}
if (runOptions.gcpRepository) {
await apmEsClient.registerGcpRepository(runOptions.gcpRepository);
@@ -205,6 +208,8 @@ yargs(process.argv.slice(2))
)}`
);

for (const aggregator of aggregators) await apmEsClient.createDataStream(aggregator);

if (runOptions.maxDocs !== 0)
await startHistoricalDataUpload(apmEsClient, logger, runOptions, from, to, version);

@@ -213,4 +218,5 @@ yargs(process.argv.slice(2))
}
}
)
.showHelpOnFail(false)
.parse();
Loading