es-google-drive

This is a test of G Drive connector for indexing. It is built from previous repo by lbroudoux.

Google Drive river for Elasticsearch

This river plugin helps to index documents from a Google Drive account.

Next steps:

1. update tika version
2. update google api
3. update dependencies
4. convert to plugin for latest ES version 5.x

Versions

Google Drive River Plugin	ElasticSearch	Attachment Plugin	Tika	Updates
master (1.3.2-SNAPSHOT)	1.3.9	n/a	1.4	added description field
master (1.3.1-SNAPSHOT)	1.3.x	n/a	1.4
1.3.0	1.3.x	n/a	1.4

Build Status

Travis CI

Getting Started

Installation

Just install a regular Elasticsearch plugin by typing :

$ bin/plugin --install com.github.chadwiki.elasticsearch/google-drive/1.3.1

This will do the job...

-> Installing com.github.chadwiki.elasticsearch/google-drive/1.3.1...
Trying http://download.elasticsearch.org/com.github.chadwiki.elasticsearch/google-drive/google-drive-1.3.0.zip...
Trying http://search.maven.org/remotecontent?filepath=com/github/chadwiki/elasticsearch/google-drive/1.3.0/google-drive-1.3.0.zip...
Downloading ......DONE
Installed google-drive

Get Google Drive credentials (clientId, clientSecret and refreshToken)

First, you need to login to Google account owning the drive to scan and then create your own application and declare it as using the Drive API into the Google APIs Console.

In order to do that, you can follow the 1st step of the Drive developers quickstart here : (https://developers.google.com/drive/quickstart-java#step_1_enable_the_drive_api).

Once done, you should note your clientId and clientSecret codes.

You need then to get an Authorization Code from the user owning the Google account (you !) for this new application.

Just open the _drive REST Endpoint with your clientId and clientSecret parameters: http://localhost:9200/_river/oauth/clientId/clientSecret

$ curl http://localhost:9200/_drive/oauth/{clientId}/{clientSecret}

You will get back a URL:

{
  "url" : "http://...."
}

Open the URL in your browser. You will be asked by Google to allow your application to access your Drive in offline mode.

Once you get back the success reply from Google, you can get the future river refreshToken by pasting gathered Authorization Code and calling

$ curl http://localhost:9200/_drive/oauth/{clientId}/{clientSecret}/{authorizationCode}

You will get back a JSON document like the following :

{
  "accessToken" : "yourAccessToken",
  "refreshToken" : "yourRefreshToken"
}

accessToken is just a one time token ; it is not used later but allow you to ensure access to the target Drive is enable. Note the given refreshToken for later creation of the river into Elasticsearch.

Creating a Google Drive river

We create first an index to store our documents (optional):

$ curl -XPUT 'http://localhost:9200/mydocs/' -d '{}'

We create the river with the following properties :

• clientId : AAAAAAAAAAAAAAAA
• clientSecret: BBBBBBBBBBBBBBBB
• refreshToken : XXXXXXXXXXXXXXXX
• Google Drive folder to index : Work (This is optional. If specified, it should be a root folder)
• Update Rate : every 15 minutes (15 * 60 * 1000 = 900000 ms)
• Get only docs like *.doc and *.pdf
• Don't index *.zip and *.gz

$ curl -XPUT 'http://localhost:9200/_river/mydocs/_meta' -d '{
  "type": "google-drive",
  "google-drive": {
    "clientId": "AAAAAAAAAAAAAAAA",
    "clientSecret": "BBBBBBBBBBBBBBBB",
    "refreshToken": "XXXXXXXXXXXXXXXX",
    "name": "My GDrive feed",
    "folder": "Work",
    "update_rate": 900000,
    "includes": "*.doc,*.pdf",
    "excludes": "*.zip,*.gz"
  }
}'

By default, river is using an index that have the same name (mydocs in the above example).

Specifying index options

Index options can be specified when creating a google-drive-river. The properties are the following :

• Index name : "drivedocs"
• Type of documents : "doc"
• Size of an indexation bulk : 50 (default is 100)

You'll have to use them as follow when creating a river :

$ curl -XPUT 'http://localhost:9200/_river/mydocs/_meta' -d '{
  "type": "google-drive",
  "google-drive": {
    "clientId": "AAAAAAAAAAAAAAAA",
    "clientSecret": "BBBBBBBBBBBBBBBB",
    "refreshToken": "XXXXXXXXXXXXXXXX",
    "name": "My GDrive feed",
    "folder": "Work",
    "update_rate": 900000,
    "includes": "*.doc,*.pdf",
    "excludes": "*.zip,*.gz"
  },
  "index": {
    "index": "drivedocs",
    "type": "doc",
    "bulk_size": 50
  }
}'

Advanced

Management actions

If you need to stop a river, you can call the _drive endpoint with your river name followed by the _stop command like this :

GET _drive/mydocs/_stop

To restart the river from the previous point, just call the corresponding _start endpoint :

GET _drive/mydocs/_start

Autogenerated mapping

When the river detect a new type, it creates automatically a mapping for this type.

{
  "doc" : {
    "properties" : {
      "title" : {
        "type" : "string",
        "analyzer" : "keyword"
      },
      "createdDate" : {
        "type" : "date",
        "format" : "dateOptionalTime"
      },
      "modifiedDate" : {
        "type" : "date",
        "format" : "dateOptionalTime"
      },
      "file" : {
        "type" : "attachment",
        "fields" : {
          "file" : {
            "type" : "string",
            "store" : "yes",
            "term_vector" : "with_positions_offsets"
          },
          "title" : {
            "type" : "string",
            "store" : "yes"
          }
        }
      }
    }
  }
}

License

This software is licensed under the Apache 2 license, quoted below.

~~Copyright 2013 Laurent Broudoux~~
Copyright 2017 Chad Pryor

Licensed under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.