Elasticsearch connection problem

[szmizorsz]

Hi,

I wanted to experiment with Splainer with Elasticsearch, but I am struggling to make it connect to ES.

With http://splainer.io/ and running local splainer app I face the same problem:

"Error with your query. Double check that the URL is correct.
Network Error! (host not found)
or CORS needs to be configured for your Elasticsearch
Enable CORS in elasticsearch.yml:
http.cors.allow-origin: "/https?://(.*?.)?(quepid.com|splainer.io)/"http.cors.enabled: true"

My relevant elasticsearch.yml configuration:
http.cors.allow-origin: ""
http.cors.enabled: true
I needed the "" to let kibana to connect.

I also tried to set other CORS configuration, but they did not help:
http.cors.allow-methods: OPTIONS, HEAD, GET, POST, PUT, DELETE
http.cors.allow-headers: "X-Requested-With, Content-Type, Content-Length, X-User"

Elasticsearch version: 2.3.1

Thanks for your help in advance.
Szabolcs

[softwaredoug]

Hey @szmizorsz

There's actually a bug in Elasticsearch 2.3 line that causes Elasticsearch to ignore the CORS settings.

So unfortunately there's not much I can do to get Splainer to support Elasticsearch 2.3. Eventually I'd like to sidestep this CORS business by making Splainer an ES plugin or perhaps a chrome extension. However, that causes Splainer to lose some useful features like sharing http://splainer.io URLs around to colleagues, etc.

[softwaredoug]

I believe this issue is resolved in the recently released Elasticsearch 2.3.2. If you are able to upgrade, please let me know if that does it for you.

[szmizorsz]

Hey,
After the upgrade everything works fine.
Thanks for your fast help.
Regards,
Szabolcs

[softwaredoug]

Thanks! And don't hesitate to reach out for further help

[szmizorsz]

Hey Doug,

I played a bit with Splainer and came up with a question.

In our use cases we have many ES queries where score calculation is based on constant score queries wrapped by a function score query with many function score boosters.
A typical query looks like this:

{
"size" : 47,
"query" : {
"function_score" : {
"query" : {
"constant_score" : {
"filter" : {
"bool" : {
"filter" : [ {
"terms" : {
"Section" : [ "public", "members_only", "private" ]
}
}, {
"term" : {
"IsHD" : "1"
}
}, {
"terms" : {
"Selection" : [ "new" ]
}
}, {
"term" : {
"MainCategory" : "any"
}
}, {
"term" : {
"IsTestAccount" : "0"
}
} ],
"must_not" : [ {
"term" : {
"hidden" : "true"
}
}, {
"regexp" : {
"BannedCountry.raw" : {
"value" : "(.,)?(CUS:00|CUS:CT)(,.)?",
"flags_value" : 65535
}
}
}, {
"terms" : {
"itemId.raw" : [ "ALREADY_RECOMMENDED" ]
}
} ]
}
}
}
},
"functions" : [ {
"filter" : {
"term" : {
"Section" : "public"
}
},
"weight" : 600000.0
}, {
"filter" : {
"term" : {
"Section" : "members_only"
}
},
"weight" : 400000.0
}, {
"filter" : {
"term" : {
"Section" : "private"
}
},
"weight" : 200000.0
}, {
"filter" : {
"term" : {
"HasProfilePicture" : "1"
}
},
"weight" : 1000000.0
}, {
"filter" : {
"term" : {
"Miscellaneous" : "selected"
}
},
"weight" : 100.0
}, {
"field_value_factor" : {
"field" : "PopularityNormalized",
"factor" : 1.0,
"missing" : 0.0
}
}, {
"field_value_factor" : {
"field" : "RatingNormalized",
"factor" : 1.0,
"missing" : 0.0
}
}, {
"field_value_factor" : {
"field" : "QualityCategoryNormalized",
"factor" : 1.0,
"missing" : 0.0
}
} ],
"score_mode" : "sum",
"boost_mode" : "multiply"
}
},
"_source" : {
"includes" : [ "HasProfilePicture", "BannedCountry", "PersonHash", "Section", "IsHD", "IsTestAccount" ],
"excludes" : [ ]
}
}

Our data is coming to Elasticsearch from an RDBMS where tables contains enum-like data and not free text data. In our use cases TF/IDF scoring was a bit misleading that's why we replaced with constant score queries and function score boosters. But we can have many boosters (much more than I provided in the example above), so when we get the final score it would be helpful to clearly identify which booster contributed to the final score without digging into the ES explain.

When I checked it with Splainer I got only this information (e.g.: for one item):
Summarized:
1600127.8 Product of following:
1600127.8 function score, score mode [sum]
1 Constant Scored Query
Hot matches:
1600127.8 Product of following:
1600127.8 function score, score mode [sum]
1 Constant Scored Query
And the full explain from ES.

So the question is do you plan to introduce some features in Splainer that handles the function score queries?

BTW: we also play in some cases with multi-match queries where we rely on TF/IDF score calculation. In those cases Splainer did an awesome job to simply explain the score.

Thanks and Regards,
Szabolcs

[softwaredoug]

that does look like a bug, Szabolcs, I'll create an issue.

Thanks!
-Doug

[softwaredoug]

@szmizorsz can you go into Splainer and get the "full explain" JSON for this query? If you do that I'll go and create an issue.

[szmizorsz]

Sure:

{
"value": 1600127.8,
"description": "function score, product of:",
"details": -[
-{
"value": 1,
"description": "ConstantScore(#ConstantScore(Section:public Section:members_only Section:private) #IsHD:\b\u0000\u0000\u0000\u0001 #ConstantScore(Selection:new) #MainCategory:any #IsTestAccount:\b\u0000\u0000\u0000\u0000 -hidden:true -BannedCountry.raw:/(.,)?(CUS:00|CUS:CT)(,.)?/ -ConstantScore(itemId.raw:ALREADY_RECOMMENDED)), product of:",
"details": -[
-{
"value": 1,
"description": "boost",
"details": [ ]
},
-{
"value": 1,
"description": "queryNorm",
"details": [ ]
}
]
},
-{
"value": 1600127.8,
"description": "min of:",
"details": -[
-{
"value": 1600127.8,
"description": "function score, score mode [sum]",
"details": -[
-{
"value": 600000,
"description": "function score, product of:",
"details": -[
-{
"value": 1,
"description": "match filter: Section:public",
"details": [ ]
},
-{
"value": 600000,
"description": "product of:",
"details": -[
-{
"value": 1,
"description": "constant score 1.0 - no function provided",
"details": [ ]
},
-{
"value": 600000,
"description": "weight",
"details": [ ]
}
]
}
]
},
-{
"value": 1000000,
"description": "function score, product of:",
"details": -[
-{
"value": 1,
"description": "match filter: HasProfilePicture:`\b\u0000\u0000\u0000\u0001",
"details": [ ]
},
-{
"value": 1000000,
"description": "product of:",
"details": -[
-{
"value": 1,
"description": "constant score 1.0 - no function provided",
"details": [ ]
},
-{
"value": 1000000,
"description": "weight",
"details": [ ]
}
]
}
]
},
-{
"value": 100,
"description": "function score, product of:",
"details": -[
-{
"value": 1,
"description": "match filter: Miscellaneous:selected",
"details": [ ]
},
-{
"value": 100,
"description": "product of:",
"details": -[
-{
"value": 1,
"description": "constant score 1.0 - no function provided",
"details": [ ]
},
-{
"value": 100,
"description": "weight",
"details": [ ]
}
]
}
]
},
-{
"value": 8.174926,
"description": "function score, product of:",
"details": -[
-{
"value": 1,
"description": "match filter: :",
"details": [ ]
},
-{
"value": 8.174926,
"description": "field value function: none(doc['PopularityNormalized'].value?:0.0 * factor=1.0)",
"details": [ ]
}
]
},
-{
"value": 10.598113,
"description": "function score, product of:",
"details": -[
-{
"value": 1,
"description": "match filter: :",
"details": [ ]
},
-{
"value": 10.598113,
"description": "field value function: none(doc['RatingNormalized'].value?:0.0 * factor=1.0)",
"details": [ ]
}
]
},
-{
"value": 9,
"description": "function score, product of:",
"details": -[
-{
"value": 1,
"description": "match filter: :",
"details": [ ]
},
-{
"value": 9,
"description": "field value function: none(doc['QualityCategoryNormalized'].value?:0.0 * factor=1.0)",
"details": [ ]
}
]
}
]
},
-{
"value": 3.4028235e+38,
"description": "maxBoost",
"details": [ ]
}
]
}
]
}

Thanks