Docs - hints for handling errors and identifying queries and responses #1049

gingerwizard · 2020-08-20T11:30:39Z

Some doc hints for dealing with Rally errors and extracting requests and responses.

Closes #791

dliappis

Thanks for adding this!

I left a few comments regarding Sphinx/rst style.

Main think, use make docs and then preview the generated page under _docs/build/html/recipes.html.

dliappis · 2020-08-21T09:16:59Z

docs/recipes.rst

+
+This query requires the field ``source.geo.location`` to be mapped as a ``geo_point`` type. If incorrectly mapped, Elasticsearch will respond with an error. 
+
+Rally will not exit on errors (unless fatal e.g. [http://man7.org/linux/man-pages/man2/connect.2.html](ECONNREFUSED)) by default, instead reporting errors in the summary report via the [Error Rate](https://esrally.readthedocs.io/en/stable/summary_report.html?highlight=on-error#error-rate) statistic. This can potentially leading to misleading results. This behavior is by design and consistent with other load testing tools such as JMeter i.e. In most cases it is desirable that a large long running benchmark should not fail because of a single error response. 


Links in rst should look like:

`ECONNREFUSED <https://man7.org/linux/man-pages/man2/connect.2.html>`_

See https://raw.githubusercontent.com/elastic/rally/master/docs/adding_tracks.rst for examples.

For the internal link to the summary_report.html page, one way to do it is to add a cross-reference link just before the Error rate\n------- in https://raw.githubusercontent.com/elastic/rally/master/docs/summary_report.rst like below: (see also an existing example in install.rst)

$ git diff docs/summary_report.rst diff --git a/docs/summary_report.rst b/docs/summary_report.rst index 157a9e2..58891f6 100644 --- a/docs/summary_report.rst +++ b/docs/summary_report.rst @@ -187,6 +187,8 @@ Rally reports several percentile numbers for each task. Which percentiles are sh * **Definition**: Time period between start of request processing and receiving the complete response. This metric can easily be mixed up with ``latency`` but does not include waiting time. This is what most load testing tools refer to as "latency" (although it is incorrect). * **Corresponding metrics key**: ``service_time`` +.. _summary_report_error_rate: + Error rate ----------

With that done, you can simple reference it using:

instead reporting errors in the summary report via the :ref:`Error Rate <summary_report_error_rate>` statistic.

By the way you can easily preview your docs by using make docs and then preview the locally generated files under docs/_build/html/recipes.html.

dliappis · 2020-08-21T09:29:38Z

docs/recipes.rst

+
+Rally will not exit on errors (unless fatal e.g. [http://man7.org/linux/man-pages/man2/connect.2.html](ECONNREFUSED)) by default, instead reporting errors in the summary report via the [Error Rate](https://esrally.readthedocs.io/en/stable/summary_report.html?highlight=on-error#error-rate) statistic. This can potentially leading to misleading results. This behavior is by design and consistent with other load testing tools such as JMeter i.e. In most cases it is desirable that a large long running benchmark should not fail because of a single error response. 
+
+ This behavior can also be changed, by invoking Rally with the [--on-error](https://esrally.readthedocs.io/en/stable/command_line_reference.html?highlight=on-error#on-error) switch e.g.


See earlier comments on how to build internal cross-references.

Also there is a leading whitespace here, ruining formatting.

dliappis · 2020-08-21T09:33:29Z

docs/recipes.rst

+
+ This behavior can also be changed, by invoking Rally with the [--on-error](https://esrally.readthedocs.io/en/stable/command_line_reference.html?highlight=on-error#on-error) switch e.g.
+
+	esrally --track=geonames --on-error=abort


This renders weirdly (try make docs as mentioned above), what if we just merge it with the previous line like

... switch e.g. ``esrally --track=geonames --on-error=abort``.

dliappis · 2020-08-21T09:33:42Z

docs/recipes.rst

+
+	esrally --track=geonames --on-error=abort
+
+Errors can also be investigated if you have configured a [dedicated Elasticsearch metrics store](https://esrally.readthedocs.io/en/stable/configuration.html#advanced-configuration).


Same comment for cross-reference links.

dliappis · 2020-08-21T09:39:43Z

docs/recipes.rst

+      }
+    }
+
+For this term query to match the field ``http.request.method`` needs to be type `keyword`. Should this field be [dynamically mapped](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html), its default type will be ``text`` causing the value `GET` to be [analyzed](https://www.elastic.co/guide/en/elasticsearch/reference/current/text.html), and indexed as `get`. The above query will in turn return `0` hits. The field should either be correctly mapped or the query modified to match on `http.request.method.keyword`.


See earlier comment (with ECONNREFUSED ) about how to use external links. Should have mentioned earlier that https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html will be useful. For code samples we use:

``code sample``

dliappis · 2020-08-21T09:40:06Z

docs/recipes.rst

+
+Issues such as this can lead to misleading benchmarking results. Prior to running any benchmarks for analysis, we therefore recommended users ascertain whether queries are behaving as intended. Rally provides several tools to assist with this.
+
+Firstly, users can modify the [logging level](https://esrally.readthedocs.io/en/stable/configuration.html?highlight=logging#logging) of Rally to `DEBUG`. Specifically, modify the ``elasticsearch`` logger i.e.::


Similar comment like above for cross-references.

dliappis · 2020-08-21T09:41:59Z

docs/recipes.rst

+	  }
+	}
+
+This will inturn ensure logs include the Elasticsearch query and accompanying response e.g.


s/inturn/in turn ?

Also you need to end the line in :: so that the next lines are properly formatted.

dliappis · 2020-08-21T10:01:56Z

docs/recipes.rst

+
+Users should discard any performance metrics collected from a benchmark with DEBUG logging. This will likely cause a client-side bottleneck so once the correctness of the queries have been established, disable this setting and re-run any benchmarks.
+
+The number of hits from queries can also be investigated if you have configured a [dedicated Elasticsearch metrics store](https://esrally.readthedocs.io/en/stable/configuration.html#advanced-configuration). Specifically, documents within the index pattern ``rally-metrics-*`` contain a ``meta`` field with summary of individual responses e.g.::


See above for cross-references.

gingerwizard · 2020-08-21T12:57:20Z

@dliappis i believe this renders better and fixes the issues you raised. Appreciate the feedback on rst formatting and hints.

dliappis

LGTM thanks for iterating!

Left a few ideas for clarifications and one or two minor grammatical observations.

dliappis · 2020-08-21T13:02:59Z

docs/recipes.rst

+
+Issues such as this can lead to misleading benchmarking results. Prior to running any benchmarks for analysis, we therefore recommended users ascertain whether queries are behaving as intended. Rally provides several tools to assist with this.
+
+Firstly, users can modify the :ref:`logging level <logging>` of Rally to ``DEBUG``. Specifically, modify the ``elasticsearch`` logger i.e.::


Just a thought: Maybe we could also make it a bit clearer that this logger is specific to the Elasticsearch client, e.g. by saying:

... modify the logging level for the ``elasticsearch`` client i.e.::

dliappis · 2020-08-21T13:04:43Z

docs/recipes.rst

+	2019-12-16 14:56:08,389 -not-actor-/PID:9790 elasticsearch DEBUG > {"sort":[{"geonameid":"asc"}],"query":{"match_all":{}}}
+	2019-12-16 14:56:08,389 -not-actor-/PID:9790 elasticsearch DEBUG < {"took":1,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":{"value":1000,"relation":"eq"},"max_score":null,"hits":[{"_index":"geonames","_type":"_doc","_id":"Lb81D28Bu7VEEZ3mXFGw","_score":null,"_source":{"geonameid": 2986043, "name": "Pic de Font Blanca", "asciiname": "Pic de Font Blanca", "alternatenames": "Pic de Font Blanca,Pic du Port", "feature_class": "T", "feature_code": "PK", "country_code": "AD", "admin1_code": "00", "population": 0, "dem": "2860", "timezone": "Europe/Andorra", "location": [1.53335, 42.64991]},"sort":[2986043]},
+
+Users should discard any performance metrics collected from a benchmark with DEBUG logging. This will likely cause a client-side bottleneck so once the correctness of the queries have been established, disable this setting and re-run any benchmarks.


s/DEBUG/``DEBUG``?

also once the correctness of the queries have been established -> once the correctness of the queries has been established?

dliappis · 2020-08-21T13:05:40Z

docs/recipes.rst

+
+Users should discard any performance metrics collected from a benchmark with DEBUG logging. This will likely cause a client-side bottleneck so once the correctness of the queries have been established, disable this setting and re-run any benchmarks.
+
+The number of hits from queries can also be investigated if you have configured a :ref:`dedicated Elasticsearch metrics store <advanced_configuration>`. Specifically, documents within the index pattern ``rally-metrics-*`` contain a ``meta`` field with summary of individual responses e.g.::


with summary of individual responses -> with a summary of individual responses?

dliappis

LGTM

Hints for handling errors and identifying queries and responses

8aa9283

gingerwizard requested a review from dliappis August 20, 2020 11:30

Fix formatting errors

5db75b5

dliappis reviewed Aug 21, 2020

View reviewed changes

gingerwizard added :Docs Changes to the documentation enhancement Improves the status quo labels Aug 21, 2020

gingerwizard self-assigned this Aug 21, 2020

Fix links and formatting

7419fcf

gingerwizard requested a review from dliappis August 21, 2020 12:57

dliappis approved these changes Aug 21, 2020

View reviewed changes

Grammatical fixes

e2ec43d

gingerwizard requested a review from dliappis August 21, 2020 13:17

dliappis approved these changes Aug 21, 2020

View reviewed changes

gingerwizard merged commit 030f71b into elastic:master Aug 21, 2020

dliappis mentioned this pull request Sep 9, 2020

Force Merge Runner Improvements - Polling #1051

Closed

danielmitterdorfer added this to the 2.0.2 milestone Oct 26, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docs - hints for handling errors and identifying queries and responses #1049

Docs - hints for handling errors and identifying queries and responses #1049

gingerwizard commented Aug 20, 2020

dliappis left a comment

dliappis Aug 21, 2020

dliappis Aug 21, 2020

dliappis Aug 21, 2020

dliappis Aug 21, 2020

dliappis Aug 21, 2020

dliappis Aug 21, 2020

dliappis Aug 21, 2020

dliappis Aug 21, 2020

dliappis Aug 21, 2020

dliappis Aug 21, 2020

dliappis Aug 21, 2020

gingerwizard commented Aug 21, 2020

dliappis left a comment

dliappis Aug 21, 2020

dliappis Aug 21, 2020

dliappis Aug 21, 2020

dliappis left a comment


		This query requires the field ``source.geo.location`` to be mapped as a ``geo_point`` type. If incorrectly mapped, Elasticsearch will respond with an error.

		Rally will not exit on errors (unless fatal e.g. [http://man7.org/linux/man-pages/man2/connect.2.html](ECONNREFUSED)) by default, instead reporting errors in the summary report via the [Error Rate](https://esrally.readthedocs.io/en/stable/summary_report.html?highlight=on-error#error-rate) statistic. This can potentially leading to misleading results. This behavior is by design and consistent with other load testing tools such as JMeter i.e. In most cases it is desirable that a large long running benchmark should not fail because of a single error response.


		Rally will not exit on errors (unless fatal e.g. [http://man7.org/linux/man-pages/man2/connect.2.html](ECONNREFUSED)) by default, instead reporting errors in the summary report via the [Error Rate](https://esrally.readthedocs.io/en/stable/summary_report.html?highlight=on-error#error-rate) statistic. This can potentially leading to misleading results. This behavior is by design and consistent with other load testing tools such as JMeter i.e. In most cases it is desirable that a large long running benchmark should not fail because of a single error response.

		This behavior can also be changed, by invoking Rally with the [--on-error](https://esrally.readthedocs.io/en/stable/command_line_reference.html?highlight=on-error#on-error) switch e.g.


		This behavior can also be changed, by invoking Rally with the [--on-error](https://esrally.readthedocs.io/en/stable/command_line_reference.html?highlight=on-error#on-error) switch e.g.

		esrally --track=geonames --on-error=abort


		esrally --track=geonames --on-error=abort

		Errors can also be investigated if you have configured a [dedicated Elasticsearch metrics store](https://esrally.readthedocs.io/en/stable/configuration.html#advanced-configuration).


		Issues such as this can lead to misleading benchmarking results. Prior to running any benchmarks for analysis, we therefore recommended users ascertain whether queries are behaving as intended. Rally provides several tools to assist with this.

		Firstly, users can modify the [logging level](https://esrally.readthedocs.io/en/stable/configuration.html?highlight=logging#logging) of Rally to `DEBUG`. Specifically, modify the ``elasticsearch`` logger i.e.::


		Users should discard any performance metrics collected from a benchmark with DEBUG logging. This will likely cause a client-side bottleneck so once the correctness of the queries have been established, disable this setting and re-run any benchmarks.

		The number of hits from queries can also be investigated if you have configured a [dedicated Elasticsearch metrics store](https://esrally.readthedocs.io/en/stable/configuration.html#advanced-configuration). Specifically, documents within the index pattern ``rally-metrics-*`` contain a ``meta`` field with summary of individual responses e.g.::


		Issues such as this can lead to misleading benchmarking results. Prior to running any benchmarks for analysis, we therefore recommended users ascertain whether queries are behaving as intended. Rally provides several tools to assist with this.

		Firstly, users can modify the :ref:`logging level <logging>` of Rally to ``DEBUG``. Specifically, modify the ``elasticsearch`` logger i.e.::

Docs - hints for handling errors and identifying queries and responses #1049

Docs - hints for handling errors and identifying queries and responses #1049

Conversation

gingerwizard commented Aug 20, 2020

dliappis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gingerwizard commented Aug 21, 2020

dliappis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dliappis left a comment

Choose a reason for hiding this comment