Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Search on large number of traces can overload Elasticserch #960

Open
pavolloffay opened this issue Jul 26, 2018 · 3 comments
Open

Search on large number of traces can overload Elasticserch #960

pavolloffay opened this issue Jul 26, 2018 · 3 comments

Comments

@pavolloffay
Copy link
Member

pavolloffay commented Jul 26, 2018

Requirement - what kind of business use case are you trying to solve?

Search a large number of traces via /api/traces. For example limit set to 300k can overload elasticsearch. Resulting in running GC in a loop

[2018-07-26T17:09:36,078][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][315] overhead, spent [4s] collecting in the last [4.2s]
[2018-07-26T17:09:39,889][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][316] overhead, spent [3.5s] collecting in the last [3.8s]
[2018-07-26T17:09:43,519][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][317] overhead, spent [3.4s] collecting in the last [3.6s]
[2018-07-26T17:09:46,900][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][318] overhead, spent [3.1s] collecting in the last [2.9s]
[2018-07-26T17:09:50,112][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][319] overhead, spent [2.9s] collecting in the last [3.6s]
[2018-07-26T17:09:53,126][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][320] overhead, spent [2.5s] collecting in the last [2.7s]
[2018-07-26T17:09:56,015][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][321] overhead, spent [3s] collecting in the last [3.1s]
[2018-07-26T17:09:58,613][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][322] overhead, spent [2.4s] collecting in the last [2.5s]
[2018-07-26T17:10:01,885][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][323] overhead, spent [3.1s] collecting in the last [3.2s]
[2018-07-26T17:10:04,490][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][324] overhead, spent [2.4s] collecting in the last [2.6s]
[2018-07-26T17:10:07,791][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][325] overhead, spent [3.1s] collecting in the last [3.2s]
[2018-07-26T17:10:10,437][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][326] overhead, spent [2.5s] collecting in the last [2.6s]
[2018-07-26T17:10:14,049][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][327] overhead, spent [3.5s] collecting in the last [3.6s]
[2018-07-26T17:10:16,558][WARN ][o.e.m.j.JvmGcMonitorService] [EZpOoVR] [gc][328] overhead, spent [2.3s] collecting in the last [2.5s]

Steps to reproduce:
docker run -it --rm "ES_JAVA_OPTS=-Xms=2g -Xmx2g" -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "xpack.security.enabled=false" --name=elasticsearch docker.elastic.co/elasticsearch/elasticsearch:5.6.10
SPAN_STORAGE_TYPE=elasticsearch go run -tags ui cmd/standalone/main.go --collector.queue-size=300000
QUERY_FROM=jaeger-query ITERATIONS=3000 THREAD_COUNT=100 mvn clean package exec:java from https://github.com/pavolloffay/jaeger-perf-tests

Proposal - what do you suggest to solve the problem or improve the existing situation?

Maybe use scroll API https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html #696 (comment) . Or define max limit and provide pagination. UI already defines hard limit to Related issue jaegertracing/jaeger-ui#73

@NoobIsBack
Copy link

Could u pls add your query, this would provide proper insights into the problem and resolution

@pavolloffay
Copy link
Member Author

You can query jaeger traces API with a large limit parameter.

@arruda
Copy link

arruda commented Jul 5, 2021

One of the things that I thought about that could help (since there is no pagination on the ui) is to use multiple requests with a very high limit, but using a very small window of time for the search( using lookback=custom and setting the start/end parameters). Not sure if this indeed would cover the case you are talking about, and you would also need to check for duplicate traces between the different requests results, but I believe it might be workaround on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants