Scripting: Conditionally use java time api in scripting #31441

rjernst · 2018-06-19T18:06:44Z

This commit adds a boolean system property, es.scripting.use_java_time,
which controls the concrete return type used by doc values within
scripts. The return type of accessing doc values for a date field is
changed to Object, essentially duck typing the type to allow
co-existence during the transition from joda time to java time.

This commit adds a boolean system property, `es.scripting.use_java_time`, which controls the concrete return type used by doc values within scripts. The return type of accessing doc values for a date field is changed to Object, essentially duck typing the type to allow co-existence during the transition from joda time to java time.

elasticmachine · 2018-06-19T18:06:46Z

Pinging @elastic/es-core-infra

s1monw · 2018-06-25T08:21:37Z

server/src/main/java/org/elasticsearch/index/fielddata/ScriptDocValues.java

@@ -141,17 +144,27 @@ public int size() {
        }
    }

-    public static final class Dates extends ScriptDocValues<ReadableDateTime> {
-        protected static final DeprecationLogger deprecationLogger = new DeprecationLogger(ESLoggerFactory.getLogger(Dates.class));
+    public static final class Dates extends ScriptDocValues<Object> {


wouldn't is be simpler to have 2 Dates classes and keep the type safety instead? I mean I understand we would duplicate the code I still kinda like that better and we can put the conditional in a single place?

We don't actually have type safety already, because painless exposes raw ScriptDocValues (because it can't know for a given field name which ScriptDocValues specific class is to be used at compile time). I think it would also detach the relevant code for little gain (having to do this selection of which Dates class to use inside field data code where these are constructed).

rjernst · 2018-07-25T18:19:59Z

@spinscale @s1monw After a suggestion from @jasontedor, I have reworked this to emit a deprecation message on each use of joda time within scripts. While this is more verbose, it will more narrowly target those using the dates api, thus eliminating noise for those that don't.

spinscale

left one minor correction and one question regarding a string representation change. Codewise it looks good to me.

Do we need to document this further for users?

spinscale · 2018-07-31T16:14:07Z

server/src/main/java/org/elasticsearch/index/fielddata/ScriptDocValues.java

-                for (int i = 0; i < backup.length; i++) {
-                    dates[i].setMillis(in.nextValue());
+                for (int i = 0; i < count; ++i) {
+                    dates[i] = ZonedDateTime.ofInstant(Instant.ofEpochMilli(in.nextValue()), JAVA_TIME_UTC);


you could just use ZoneOffset.UTC here

Unfortunately not. The variant of ofInstant that takes a ZoneOffset also still requires ZoneId.

Oh a ZoneOffset is a ZoneId. Sorry I missed this before. Yes, I will update.

spinscale · 2018-07-31T16:33:39Z

modules/lang-painless/src/test/resources/rest-api-spec/test/painless/50_script_doc_values.yml

@@ -95,7 +95,7 @@ setup:
                    field:
                        script:
                            source: "doc.date.get(0)"
-    - match: { hits.hits.0.fields.field.0: '2017-01-01T12:11:12.000Z' }
+    - match: { hits.hits.0.fields.field.0: '2017-01-01T12:11:12Z' }


Are we good with this change, as it changes the string representation of dates? I suppose this just calls LocalTime.toString() which only accounts for nanoseconds if they are greater than zero.

Yes that is correct. We can mention it in the 7.0 docs when the bwc behavior is removed. I don't think it has any real impact, since with or without the sub second part, the adheres to ISO 8601.

rjernst

Do we need to document this further for users?

Do you have a suggestion for where to put additional documentation? I already added a note to the painless getting started guide, where we currently mention ReadableDateTime.

This commit adds a boolean system property, `es.scripting.use_java_time`, which controls the concrete return type used by doc values within scripts. The return type of accessing doc values for a date field is changed to Object, essentially duck typing the type to allow co-existence during the transition from joda time to java time.

@throws

* 6.x: Fix scriptdocvalues tests with dates Correct minor typo in explain.asciidoc for HLRC Fix painless whitelist and warnings from backporting #31441 Build: Add elastic maven to repos used by BuildPlugin (#32549) Scripting: Conditionally use java time api in scripting (#31441) [ML] Improve error when no available field exists for rule scope (#32550) [ML] Improve error for functions with limited rule condition support (#32548) [ML] Remove multiple_bucket_spans [ML] Fix thread leak when waiting for job flush (#32196) (#32541) Painless: Clean Up PainlessField (#32525) Add @AwaitsFix for #32554 Remove broken @link in Javadoc Add AwaitsFix to failing test - see #32546 SQL: Added support for string manipulating functions with more than one parameter (#32356) [DOCS] Reloadable Secure Settings (#31713) Fix compilation error introduced by #32339 [Rollup] Remove builders from TermsGroupConfig (#32507) Use hostname instead of IP with SPNEGO test (#32514) Switch x-pack rolling restart to new style Requests (#32339) [DOCS] Small fixes in rule configuration page (#32516) Painless: Clean up PainlessMethod (#32476) SQL: Add test for handling of partial results (#32474) Docs: Add missing migration doc for logging change Build: Remove shadowing from benchmarks (#32475) Docs: Add all JDKs to CONTRIBUTING.md Logging: Make node name consistent in logger (#31588) High-level client: fix clusterAlias parsing in SearchHit (#32465) REST high-level client: parse back _ignored meta field (#32362) backport fix of reduceRandom fix (#32508) Add licensing enforcement for FIPS mode (#32437) INGEST: Clean up Java8 Stream Usage (#32059) (#32485) Improve the error message when an index is incompatible with field aliases. (#32482) Mute testFilterCacheStats Scripting: Fix painless compiler loader to know about context classes (#32385) [ML][DOCS] Fix typo applied_to => applies_to Mute SSLTrustRestrictionsTests on JDK 11 Changed ReindexRequest to use Writeable.Reader (#32401) Increase max chunk size to 256Mb for repo-azure (#32101) Mute KerberosAuthenticationIT fix no=>not typo (#32463) HLRC: Add delete watch action (#32337) Fix calculation of orientation of polygons (#27967) [Kerberos] Add missing javadocs (#32469) Fix missing JavaDoc for @throws in several places in KerberosTicketValidator. Make get all app privs requires "*" permission (#32460) Ensure KeyStoreWrapper decryption exceptions are handled (#32472) update rollover to leverage write-alias semantics (#32216) [Kerberos] Remove Kerberos bootstrap checks (#32451) Switch security to new style Requests (#32290) Switch security spi example to new style Requests (#32341) Painless: Add PainlessConstructor (#32447) Update Fuzzy Query docs to clarify default behavior re max_expansions (#30819) Remove > from Javadoc (fatal with Java 11) Tests: Fix convert error tests to use fixed value (#32415) IndicesClusterStateService should replace an init. replica with an init. primary with the same aId (#32374) auto-interval date histogram - 6.x backport (#32107) [CI] Mute DocumentSubsetReaderTests testSearch [TEST] Mute failing InternalEngineTests#testSeqNoAndCheckpoints TEST: testDocStats should always use forceMerge (#32450) TEST: Avoid deletion in FlushIT AwaitsFix IndexShardTests#testDocStats Painless: Add method type to method. (#32441) Remove reference to non-existent store type (#32418) [TEST] Mute failing FlushIT test Fix ordering of bootstrap checks in docs (#32417) Wrong discovery.type for azure in breaking changes (#32432) Mute ConvertProcessorTests failing tests TESTS: Move netty leak detection to paranoid level (#32354) (#32425) Upgrade to Lucene-7.5.0-snapshot-608f0277b0 (#32390) [Kerberos] Avoid vagrant update on precommit (#32416) TEST: Avoid triggering merges in FlushIT [DOCS] Fixes formatting of scope object in job resource Switch x-pack/plugin to new style Requests (#32327) Release requests in cors handle (#32410) Remove BouncyCastle dependency from runtime (#32402) Copy missing segment attributes in getSegmentInfo (#32396) Rest HL client: Add put license action (#32214) Docs: Correcting a typo in tophits (#32359) Build: Stop double generating buildSrc pom (#32408) Switch x-pack full restart to new style Requests (#32294) Painless: Clean Up PainlessClass Variables (#32380) [ML] Consistent pattern for strict/lenient parser names (#32399) Add Restore Snapshot High Level REST API Update update-settings.asciidoc (#31378) Introduce index store plugins (#32375) Rank-Eval: Reduce scope of an unchecked supression Make sure _forcemerge respects `max_num_segments`. (#32291)

* master: HLRC: Move commercial clients from XPackClient (#32596) Add cluster UUID to Cluster Stats API response (#32206) Security: move User to protocol project (#32367) [TEST] Test for shard failures, add debug to testProfileMatchesRegular Minor fix for javadoc (applicable for java 11). (#32573) Painless: Move Some Lookup Logic to PainlessLookup (#32565) TEST: Avoid merges in testSeqNoAndCheckpoints [Rollup] Remove builders from HistoGroupConfig (#32533) Mutes failing SQL string function tests due to #32589 fixed elements in array of produced terms (#32519) INGEST: Enable default pipelines (#32286) Remove cluster state initial customs (#32501) Mutes LicensingDocumentationIT due to #32580 [ML] Remove multiple_bucket_spans (#32496) [ML] Rename JobProvider to JobResultsProvider (#32551) Correct minor typo in explain.asciidoc for HLRC Build: Add elastic maven to repos used by BuildPlugin (#32549) Clarify the error message when a pipeline agg is used in the 'order' parameter. (#32522) Revert "[test] turn on host io cache for opensuse (#32053)" Enable packaging tests on suse boxes [ML] Improve error when no available field exists for rule scope (#32550) [ML] Improve error for functions with limited rule condition support (#32548) Painless: Clean Up PainlessField (#32525) Add @AwaitsFix for #32554 Remove broken @link in Javadoc Scripting: Conditionally use java time api in scripting (#31441) [ML] Fix thread leak when waiting for job flush (#32196) (#32541) Add AwaitsFix to failing test - see #32546 Core: Minor size reduction for AbstractComponent (#32509) SQL: Added support for string manipulating functions with more than one parameter (#32356) [DOCS] Reloadable Secure Settings (#31713) Watcher: Reenable HttpSecretsIntegrationTests#testWebhookAction test (#32456) [Rollup] Remove builders from TermsGroupConfig (#32507) Use hostname instead of IP with SPNEGO test (#32514) Switch x-pack rolling restart to new style Requests (#32339) NETWORKING: Fix Netty Leaks by upgrading to 4.1.28 (#32511) [DOCS] Small fixes in rule configuration page (#32516) Painless: Clean up PainlessMethod (#32476) Build: Remove shadowing from benchmarks (#32475) Docs: Add all JDKs to CONTRIBUTING.md Add licensing enforcement for FIPS mode (#32437) SQL: Add test for handling of partial results (#32474) Mute testFilterCacheStats [ML][DOCS] Fix typo applied_to => applies_to Scripting: Fix painless compiler loader to know about context classes (#32385)

conet · 2018-11-16T17:01:12Z

I'm not sure that it is related to this but after upgrading to elasticsearch 6.5.0 from 6.4.3 some queries using timestamps in scripted bool queries exhibited a serious performance issue, the old query was something like:

(doc[\"timestamp\"].value.getMillis() + doc[\"duration\"].value * 1000L) < params.endMillis

the same query from a couple of seconds on 6.4.3 jumped to about 80 seconds on 6.5.0, I got lucky and removed the call to getMillis() then I got a deprecation warning pointing me to .toInstant().toEpochMilli(), after changing the query to:

(doc[\"timestamp\"].value.toInstant().toEpochMilli() + doc[\"duration\"].value * 1000L) < params.endMillis

the query time went back to a couple of seconds. I would make sure that whatever is causing this is properly documented in the release notes, I wouldn't want everybody hitting this to start digging after the source of the problem.

rjernst · 2018-11-20T00:33:37Z

@conet Using getMillis() triggers a deprecation warning. How many of these do you see written from a single request? Every call to the method (thus every document the script is run on) will call the log deprecation method, but we attempt to suppress spamming the log by keeping track of the last 128 unique locations that we saw. Since we don't have 128 different script uses in all of Elasticsearch, you should theoretically only see one log message regarding getMillis(), but I suppose there could be something else going on with the multi threaded behavior there.

conet · 2018-11-20T06:32:57Z

@rjernst I just checked the logs and I don't see any deprecation entries for getMillis().

rjernst · 2018-11-20T18:26:51Z

Thanks @conet. Without further information (something reproducible), I'm not sure what else can be done. I did not notice any slowdowns like this when developing the deprecation messages, and if there are no log messages, it would appear something odd happened when we tried to log that indefinitely stalled the log write.

conet · 2018-11-20T19:15:22Z

I don't know what you'd expect to make it reproducible, do you have a publicly available cluster where I can reproduce this? I have two completely unrelated clusters (running 6.5.0) and running a query with:

doc['timestamp'].value.getMillis()
...
#! Deprecation: Use of the joda time method [getMillis()] is deprecated. Use [toInstant().toEpochMilli()] instead.
{
  "took" : 8610,

is 20 times slower than running it with (regardless of the actual data that is stored in the cluster):

doc['timestamp'].value.toInstant().toEpochMilli()
...
{
  "took" : 307,
...

What more proof do you need? Please tell me so I can provide it... all I wanted to do is save some other unlucky guy's time when hitting this, I already solved my problem, I just wanted to make it easier for others.

rjernst · 2018-11-20T20:21:40Z

By reproducible I mean a concrete set of steps that can be run to exhibit the problem. Your posts thus far have been two snippets of code without context. They don't reveal how many documents are in your index, the structure of your index, or most importantly, where in the api you are using this script. When I do a script field with getMillis() locally, I do not see any slowness. Additionally, posting onto the end of a closed PR isn't great for visibility by others. Can you please open a new issue with steps for reproducing?

rjernst · 2018-11-20T20:24:33Z

To further clarify what I mean by steps to reproduce: could you please list a set of curl commands to (1) create an index, (2) index one or more documents and (3) run a query which exhibits the slowness using your script?

conet · 2018-11-20T21:27:26Z

Opened #35754.

rjernst · 2018-11-20T21:31:34Z

Thank you @conet

rjernst added :Core/Infra/Scripting Scripting abstractions, Painless, and Mustache >deprecation v7.0.0 v6.4.0 labels Jun 19, 2018

rjernst added 2 commits June 19, 2018 12:18

remove unused import

f42474c

fix compile

d3d2908

rjernst mentioned this pull request Jun 20, 2018

Add support for getZone on ReadableDateTime objects in Painless #31438

Closed

rjernst added 3 commits June 20, 2018 14:43

Merge branch 'master' into timeapi2

9188b15

change setting name and move check into ScriptModule

1a23214

Merge branch 'master' into timeapi2

8f63f76

s1monw reviewed Jun 25, 2018

View reviewed changes

rjernst added 14 commits June 27, 2018 09:40

Merge branch 'master' into timeapi2

99b288c

fix checkstyle

8ad8573

Merge branch 'master' into timeapi2

931d2b1

better tests

2bd06d9

fix script conversions from date to number

fc019aa

more conversions

0650ac8

Merge branch 'master' into timeapi2

c37f33d

fix test class used

c599e46

fix test

4383c43

Merge branch 'master' into timeapi2

cd8a7a7

fix missing imports

dee8abe

fix test

7a67906

Merge branch 'master' into timeapi2

c4b19cf

rework to emit deprecation on each use of script with dates

f3add18

rjernst added v6.5.0 and removed v6.4.0 labels Jul 25, 2018

fix test

4042747

rjernst mentioned this pull request Jul 31, 2018

Scripted metric aggregations: add deprecation warning and system property to control legacy params #31597

Merged

fix more tests to use normalized zoneid

74c9b89

spinscale reviewed Jul 31, 2018

View reviewed changes

rjernst commented Jul 31, 2018

View reviewed changes

rjernst added 5 commits July 31, 2018 10:08

use ZoneOffset.UTC

96e843f

remove unused import

055d2c0

another unused import...

b28ce90

Merge branch 'master' into timeapi2

efa653b

and one more

63e941b

spinscale approved these changes Aug 1, 2018

View reviewed changes

rjernst merged commit 478f6d6 into elastic:master Aug 1, 2018

rjernst deleted the timeapi2 branch August 1, 2018 15:58

rjernst added a commit that referenced this pull request Aug 1, 2018

Fix painless whitelist and warnings from backporting #31441

b879df5

rjernst mentioned this pull request Sep 7, 2018

Core: Migrating from joda time to java.time #27330

Closed

18 tasks

conet mentioned this pull request Nov 20, 2018

Script queries using joda time getMillis() slower than using java time toInstant().toEpochMilli() #35754

Closed

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scripting: Conditionally use java time api in scripting #31441

Scripting: Conditionally use java time api in scripting #31441

rjernst commented Jun 19, 2018

elasticmachine commented Jun 19, 2018

s1monw Jun 25, 2018

rjernst Jun 25, 2018

rjernst commented Jul 25, 2018

spinscale left a comment

spinscale Jul 31, 2018

rjernst Jul 31, 2018

rjernst Jul 31, 2018

spinscale Jul 31, 2018

rjernst Jul 31, 2018

rjernst left a comment

conet commented Nov 16, 2018

rjernst commented Nov 20, 2018

conet commented Nov 20, 2018

rjernst commented Nov 20, 2018

conet commented Nov 20, 2018

rjernst commented Nov 20, 2018

rjernst commented Nov 20, 2018

conet commented Nov 20, 2018

rjernst commented Nov 20, 2018

Scripting: Conditionally use java time api in scripting #31441

Scripting: Conditionally use java time api in scripting #31441

Conversation

rjernst commented Jun 19, 2018

elasticmachine commented Jun 19, 2018

s1monw Jun 25, 2018

Choose a reason for hiding this comment

rjernst Jun 25, 2018

Choose a reason for hiding this comment

rjernst commented Jul 25, 2018

spinscale left a comment

Choose a reason for hiding this comment

spinscale Jul 31, 2018

Choose a reason for hiding this comment

rjernst Jul 31, 2018

Choose a reason for hiding this comment

rjernst Jul 31, 2018

Choose a reason for hiding this comment

spinscale Jul 31, 2018

Choose a reason for hiding this comment

rjernst Jul 31, 2018

Choose a reason for hiding this comment

rjernst left a comment

Choose a reason for hiding this comment

conet commented Nov 16, 2018

rjernst commented Nov 20, 2018

conet commented Nov 20, 2018

rjernst commented Nov 20, 2018

conet commented Nov 20, 2018

rjernst commented Nov 20, 2018

rjernst commented Nov 20, 2018

conet commented Nov 20, 2018

rjernst commented Nov 20, 2018