elastic

A general purpose R interface to Elasticsearch

Elasticsearch DSL

Also check out elasticdsl - an R DSL for Elasticsearch - https://github.com/ropensci/elasticdsl - It's not currently maintained, but if you'd like to contribute le'me know

Elasticsearch info

Compatibility

This client is developed following the latest stable releases, currently v7.2.0. It is generally compatible with older versions of Elasticsearch. Unlike the Python client, we try to keep as much compatibility as possible within a single version of this client, as that's an easier setup in R world.

Security

You're fine running ES locally on your machine, but be careful just throwing up ES on a server with a public IP address - make sure to think about security.

Shield - This is a paid product provided by Elastic - so probably only applicable to enterprise users
DIY security - there are a variety of techniques for securing your Elasticsearch. A number of resources are collected in a blog post - tools include putting your ES behind something like Nginx, putting basic auth on top of it, using https, etc.

Installation

Stable version from CRAN

install.packages("elastic")

Development version from GitHub

install.packages("devtools")
devtools::install_github("ropensci/elastic")

library('elastic')

Install Elasticsearch

Elasticsearch installation help

w/ Docker

Pull the official elasticsearch image

docker pull elasticsearch

Then start up a container

docker run -d -p 9200:9200 elasticsearch

Then elasticsearch should be available on port 9200, try curl localhost:9200 and you should get the familiar message indicating ES is on.

If you're using boot2docker, you'll need to use the IP address in place of localhost. Get it by doing boot2docker ip.

on OSX

Download zip or tar file from Elasticsearch see here for download, e.g., curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.2.0-darwin-x86_64.tar.gz
Extract: tar -zxvf elasticsearch-7.2.0-darwin-x86_64.tar.gz
Move it: sudo mv elasticsearch-7.2.0 /usr/local
Navigate to /usr/local: cd /usr/local
Delete symlinked elasticsearch directory: rm -rf elasticsearch
Add shortcut: sudo ln -s elasticsearch-7.2.0 elasticsearch (replace version with your version)

You can also install via Homebrew: brew install elasticsearch

Note: for the 1.6 and greater upgrades of Elasticsearch, they want you to have java 8 or greater. I downloaded Java 8 from here http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html and it seemed to work great.

Upgrading Elasticsearch

I am not totally clear on best practice here, but from what I understand, when you upgrade to a new version of Elasticsearch, place old elasticsearch/data and elasticsearch/config directories into the new installation (elasticsearch/ dir). The new elasticsearch instance with replaced data and config directories should automatically update data to the new version and start working. Maybe if you use homebrew on a Mac to upgrade it takes care of this for you - not sure.

Obviously, upgrading Elasticsearch while keeping it running is a different thing (some help here from Elastic).

Start Elasticsearch

Navigate to elasticsearch: cd /usr/local/elasticsearch
Start elasticsearch: bin/elasticsearch

I create a little bash shortcut called es that does both of the above commands in one step (cd /usr/local/elasticsearch && bin/elasticsearch).

Initialization

The function connect() is used before doing anything else to set the connection details to your remote or local elasticsearch store. The details created by connect() are written to your options for the current session, and are used by elastic functions.

x <- connect(port = 9200)

If you're following along here with a local instance of Elasticsearch, you'll use x below to do more stuff.

For AWS hosted elasticsearch, make sure to specify path = "" and the correct port - transport schema pair.

connect(host = <aws_es_endpoint>, path = "", port = 80, transport_schema  = "http")
  # or
connect(host = <aws_es_endpoint>, path = "", port = 443, transport_schema  = "https")

If you are using Elastic Cloud or an installation with authentication (X-pack), make sure to specify path = "", user = "", pwd = "" and the correct port - transport schema pair.

connect(host = <ec_endpoint>, path = "", user="test", pwd = "1234", port = 9243, transport_schema  = "https")

Get some data

Elasticsearch has a bulk load API to load data in fast. The format is pretty weird though. It's sort of JSON, but would pass no JSON linter. I include a few data sets in elastic so it's easy to get up and running, and so when you run examples in this package they'll actually run the same way (hopefully).

I have prepare a non-exported function useful for preparing the weird format that Elasticsearch wants for bulk data loads, that is somewhat specific to PLOS data (See below), but you could modify for your purposes. See make_bulk_plos() and make_bulk_gbif() here.

Shakespeare data

Elasticsearch provides some data on Shakespeare plays. I've provided a subset of this data in this package. Get the path for the file specific to your machine:

shakespeare <- system.file("examples", "shakespeare_data.json", package = "elastic")
# If you're on Elastic v6 or greater, use this one with 1 type instead of 3:
shakespeare <- system.file("examples", "shakespeare_data_.json", package = "elastic")

Then load the data into Elasticsearch:

make sure to create your connection object with connect()

# x <- connect()  # do this now if you didn't do this above
invisible(docs_bulk(x, shakespeare))

If you need some big data to play with, the shakespeare dataset is a good one to start with. You can get the whole thing and pop it into Elasticsearch (beware, may take up to 10 minutes or so.):

curl -XGET https://download.elastic.co/demos/kibana/gettingstarted/shakespeare_6.0.json > shakespeare.json
curl -XPUT localhost:9200/_bulk --data-binary @shakespeare.json

Public Library of Science (PLOS) data

A dataset inluded in the elastic package is metadata for PLOS scholarly articles. Get the file path, then load:

if (index_exists(x, "plos")) index_delete(x, "plos")
plosdat <- system.file("examples", "plos_data.json", package = "elastic")
invisible(docs_bulk(x, plosdat))

Global Biodiversity Information Facility (GBIF) data

A dataset inluded in the elastic package is data for GBIF species occurrence records. Get the file path, then load:

if (index_exists(x, "gbif")) index_delete(x, "gbif")
gbifdat <- system.file("examples", "gbif_data.json", package = "elastic")
invisible(docs_bulk(x, gbifdat))

GBIF geo data with a coordinates element to allow geo_shape queries

if (index_exists(x, "gbifgeo")) index_delete(x, "gbifgeo")
gbifgeo <- system.file("examples", "gbif_geo.json", package = "elastic")
invisible(docs_bulk(x, gbifgeo))

More data sets

There are more datasets formatted for bulk loading in the ropensci/elastic_data GitHub repository. Find it at https://github.com/ropensci/elastic_data

Search

Search the plos index and only return 1 result

Search(x, index = "plos", size = 1)$hits$hits
#> [[1]]
#> [[1]]$`_index`
#> [1] "plos"
#> 
#> [[1]]$`_type`
#> [1] "article"
#> 
#> [[1]]$`_id`
#> [1] "0"
#> 
#> [[1]]$`_score`
#> [1] 1
#> 
#> [[1]]$`_source`
#> [[1]]$`_source`$id
#> [1] "10.1371/journal.pone.0007737"
#> 
#> [[1]]$`_source`$title
#> [1] "Phospholipase C-β4 Is Essential for the Progression of the Normal Sleep Sequence and Ultradian Body Temperature Rhythms in Mice"

Search the plos index, and the article document type, and query for antibody, limit to 1 result

Search(x, index = "plos", type = "article", q = "antibody", size = 1)$hits$hits
#> [[1]]
#> [[1]]$`_index`
#> [1] "plos"
#> 
#> [[1]]$`_type`
#> [1] "article"
#> 
#> [[1]]$`_id`
#> [1] "813"
#> 
#> [[1]]$`_score`
#> [1] 5.18676
#> 
#> [[1]]$`_source`
#> [[1]]$`_source`$id
#> [1] "10.1371/journal.pone.0107638"
#> 
#> [[1]]$`_source`$title
#> [1] "Sortase A Induces Th17-Mediated and Antibody-Independent Immunity to Heterologous Serotypes of Group A Streptococci"

Get documents

Get document with id=4

docs_get(x, index = 'plos', type = 'article', id = 4)
#> $`_index`
#> [1] "plos"
#> 
#> $`_type`
#> [1] "article"
#> 
#> $`_id`
#> [1] "4"
#> 
#> $`_version`
#> [1] 1
#> 
#> $`_seq_no`
#> [1] 4
#> 
#> $`_primary_term`
#> [1] 1
#> 
#> $found
#> [1] TRUE
#> 
#> $`_source`
#> $`_source`$id
#> [1] "10.1371/journal.pone.0107758"
#> 
#> $`_source`$title
#> [1] "Lactobacilli Inactivate Chlamydia trachomatis through Lactic Acid but Not H2O2"

Get certain fields

docs_get(x, index = 'plos', type = 'article', id = 4, fields = 'id')
#> $`_index`
#> [1] "plos"
#> 
#> $`_type`
#> [1] "article"
#> 
#> $`_id`
#> [1] "4"
#> 
#> $`_version`
#> [1] 1
#> 
#> $`_seq_no`
#> [1] 4
#> 
#> $`_primary_term`
#> [1] 1
#> 
#> $found
#> [1] TRUE

Get multiple documents via the multiget API

Same index and type, different document ids

docs_mget(x, index = "plos", type = "article", id = 1:2)
#> $docs
#> $docs[[1]]
#> $docs[[1]]$`_index`
#> [1] "plos"
#> 
#> $docs[[1]]$`_type`
#> [1] "article"
#> 
#> $docs[[1]]$`_id`
#> [1] "1"
#> 
#> $docs[[1]]$`_version`
#> [1] 1
#> 
#> $docs[[1]]$`_seq_no`
#> [1] 1
#> 
#> $docs[[1]]$`_primary_term`
#> [1] 1
#> 
#> $docs[[1]]$found
#> [1] TRUE
#> 
#> $docs[[1]]$`_source`
#> $docs[[1]]$`_source`$id
#> [1] "10.1371/journal.pone.0098602"
#> 
#> $docs[[1]]$`_source`$title
#> [1] "Population Genetic Structure of a Sandstone Specialist and a Generalist Heath Species at Two Levels of Sandstone Patchiness across the Strait of Gibraltar"
#> 
#> 
#> 
#> $docs[[2]]
#> $docs[[2]]$`_index`
#> [1] "plos"
#> 
#> $docs[[2]]$`_type`
#> [1] "article"
#> 
#> $docs[[2]]$`_id`
#> [1] "2"
#> 
#> $docs[[2]]$`_version`
#> [1] 1
#> 
#> $docs[[2]]$`_seq_no`
#> [1] 2
#> 
#> $docs[[2]]$`_primary_term`
#> [1] 1
#> 
#> $docs[[2]]$found
#> [1] TRUE
#> 
#> $docs[[2]]$`_source`
#> $docs[[2]]$`_source`$id
#> [1] "10.1371/journal.pone.0107757"
#> 
#> $docs[[2]]$`_source`$title
#> [1] "Cigarette Smoke Extract Induces a Phenotypic Shift in Epithelial Cells; Involvement of HIF1α in Mesenchymal Transition"

Different indeces, types, and ids

docs_mget(x, index_type_id = list(c("plos", "article", 1), c("gbif", "record", 1)))$docs[[1]]
#> $`_index`
#> [1] "plos"
#> 
#> $`_type`
#> [1] "article"
#> 
#> $`_id`
#> [1] "1"
#> 
#> $`_version`
#> [1] 1
#> 
#> $`_seq_no`
#> [1] 1
#> 
#> $`_primary_term`
#> [1] 1
#> 
#> $found
#> [1] TRUE
#> 
#> $`_source`
#> $`_source`$id
#> [1] "10.1371/journal.pone.0098602"
#> 
#> $`_source`$title
#> [1] "Population Genetic Structure of a Sandstone Specialist and a Generalist Heath Species at Two Levels of Sandstone Patchiness across the Strait of Gibraltar"

Parsing

You can optionally get back raw json from Search(), docs_get(), and docs_mget() setting parameter raw=TRUE.

For example:

(out <- docs_mget(x, index = "plos", type = "article", id = 1:2, raw = TRUE))
#> [1] "{\"docs\":[{\"_index\":\"plos\",\"_type\":\"article\",\"_id\":\"1\",\"_version\":1,\"_seq_no\":1,\"_primary_term\":1,\"found\":true,\"_source\":{\"id\":\"10.1371/journal.pone.0098602\",\"title\":\"Population Genetic Structure of a Sandstone Specialist and a Generalist Heath Species at Two Levels of Sandstone Patchiness across the Strait of Gibraltar\"}},{\"_index\":\"plos\",\"_type\":\"article\",\"_id\":\"2\",\"_version\":1,\"_seq_no\":2,\"_primary_term\":1,\"found\":true,\"_source\":{\"id\":\"10.1371/journal.pone.0107757\",\"title\":\"Cigarette Smoke Extract Induces a Phenotypic Shift in Epithelial Cells; Involvement of HIF1α in Mesenchymal Transition\"}}]}"
#> attr(,"class")
#> [1] "elastic_mget"

Then parse

jsonlite::fromJSON(out)
#> $docs
#>   _index   _type _id _version _seq_no _primary_term found
#> 1   plos article   1        1       1             1  TRUE
#> 2   plos article   2        1       2             1  TRUE
#>                     _source.id
#> 1 10.1371/journal.pone.0098602
#> 2 10.1371/journal.pone.0107757
#>                                                                                                                                                _source.title
#> 1 Population Genetic Structure of a Sandstone Specialist and a Generalist Heath Species at Two Levels of Sandstone Patchiness across the Strait of Gibraltar
#> 2                                     Cigarette Smoke Extract Induces a Phenotypic Shift in Epithelial Cells; Involvement of HIF1α in Mesenchymal Transition

Known pain points

On secure Elasticsearch servers:
- HEAD requests don't seem to work, not sure why
- If you allow only GET requests, a number of functions that require POST requests obviously then won't work. A big one is Search(), but you can use Search_uri() to get around this, which uses GET instead of POST, but you can't pass a more complicated query via the body

Screencast

A screencast introducing the package: vimeo.com/124659179

Name		Name	Last commit message	Last commit date
Latest commit History 779 Commits
.github		.github
R		R
inst		inst
man-roxygen		man-roxygen
man		man
revdep		revdep
tests		tests
tools		tools
vignettes		vignettes
.Rbuildignore		.Rbuildignore
.gitignore		.gitignore
.travis.yml		.travis.yml
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
DESCRIPTION		DESCRIPTION
LICENSE		LICENSE
LICENSE.md		LICENSE.md
Makefile		Makefile
NAMESPACE		NAMESPACE
NEWS.md		NEWS.md
README.Rmd		README.Rmd
README.md		README.md
_pkgdown.yml		_pkgdown.yml
codemeta.json		codemeta.json
cran-comments.md		cran-comments.md
es_matrix_travis.yml		es_matrix_travis.yml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Licenses found

Repository files navigation

elastic

Elasticsearch DSL

Elasticsearch info

Compatibility

Security

Installation

Install Elasticsearch

Upgrading Elasticsearch

Start Elasticsearch

Initialization

Get some data

Shakespeare data

Public Library of Science (PLOS) data

Global Biodiversity Information Facility (GBIF) data

More data sets

Search

Get documents

Get multiple documents via the multiget API

Parsing

Known pain points

Screencast

Meta

About

Licenses found

Releases

Packages

Languages

License

Licenses found

EmilBode/elastic

Folders and files

Latest commit

History

Repository files navigation

elastic

Elasticsearch DSL

Elasticsearch info

Compatibility

Security

Installation

Install Elasticsearch

Upgrading Elasticsearch

Start Elasticsearch

Initialization

Get some data

Shakespeare data

Public Library of Science (PLOS) data

Global Biodiversity Information Facility (GBIF) data

More data sets

Search

Get documents

Get multiple documents via the multiget API

Parsing

Known pain points

Screencast

Meta

About

Resources

License

Licenses found

Code of conduct

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages