2.1.2-GA

Added two new configuration options to address problems with the plugin trying to add checkpoint metadata documents to indexes that are not related to replication from Couchbase, such as .watcher and .kibana.

• couchbase.ignoreDotIndexes - Enabled by default (true). Causes the plugin to completely ignore indexes/aliases whose name starts with ".", such as ".kibana", ".marvel", etc.
• couchbase.includeIndexes - Specifying one or more index/alias names (as a comma or semicolon delimited string) here will cause the plugin to ignore the existence of all other indexes and aliases. For example, if you have only a few indexes replicated from Couchbase, there's no reason to store checkpoint metadata in all other indexes.

Note that the includeIndexes setting takes precedence over the ignoreDotIndexes setting. If you have a ".index" white-listed, it will be used even if ignoreDotIndexes is enabled.

Also, bumped the supported Couchbase versions to 2.5.x - 4.x.

2.1.1

This is the 2.1.1 GA release version of the ElasticSearch transport-couchbase plugin. It includes compatibility with ES up to v1.7.1, multiple new features and further bug fixes, including several issues found in 2.1.0. In particular, this release fixes a long-standing bug with an incorrect concurrent bulk request counter, which could eventually cause the plugin to stop accepting requests from Couchbase Server altogether.

Note that some of the configuration option names have changed, please consult the readme below.

Installation

To install the RC, run the following command from your ElasticSearch installation folder:

bin/plugin -install transport-couchbase -url https://github.com/couchbaselabs/elasticsearch-transport-couchbase/releases/download/2.1.1/elasticsearch-transport-couchbase-2.1.1.zip

Configuration

Configuration for the plugin is specified as part of the ElasticSearch config file (usually elasticsearch.yml) and is currently only read when ElasticSearch starts. Dynamic configuration support is planned for the future.

Basic Settings

• couchbase.port - The port the plugin will listen on, default 9091
• couchbase.username - The username for HTTP basic auth, default Administrator
• couchbase.password - The password for HTTP basic auth, no default
• couchbase.num_vbuckets - The number of vbuckets that ElasticSearch should pretend to have (default on Mac is 64, 1024 on all other platforms) This value MUST match the number of vbuckets on the source Couchbase cluster.
• couchbase.maxConcurrentRequests - The number of concurrent requests that the plugin will allow, default 1024 (lower this if the load on the machine gets too high)

Advanced Settings

• couchbase.ignoreFailures - Enabling this flag will cause the plugin to return a success status to Couchbase even if it cannot index some of the documents. This will prevent the XDCR replication from being stalled due to indexing errors in ElasticSearch, for example when a schema change breaks some of the ES type mappings. Default is false.
• couchbase.ignoreDeletes - Specifying one or more index names here will cause the plugin to ignore document deletion and expiration for those indexes. This can be used to turn ElasticSearch into a sort of searchable archive for a Couchbase bucket. Note that this also means that the index will continue to grow indefinitely.
• couchbase.wrapCounters - Enabling this flag will cause the plugin to wrap integer values from Couchbase, which are not valid JSON documents, in a simple document before indexing them in ElasticSearch. The resulting document is in the format { "value" : <value> } and is stored under the ID of the original value from Couchbase.

Mapping Couchbase documents to ElasticSearch types

• couchbase.typeSelector - The type selector class to use for mapping documents to types.
  • org.elasticsearch.transport.couchbase.capi.DefaultTypeSelector - Maps all documents to the specified type. As the name implies, this is the default type selector and can be omitted from the configuration file.
    • couchbase.typeSelector.defaultDocumentType - The document type to which the DefaultTypeSelector will map all documents. Defaults to "couchbaseDocument".
    • couchbase.typeSelector.checkpointDocumentType - The document type to which replication checkpoint documents will be mapped. Defaults to "couchbaseCheckpoint".
  • org.elasticsearch.transport.couchbase.capi.DelimiterTypeSelector - If the document ID is of the format <type><delimiter><*>, this type selector will map these documents to the type <type>, otherwise it will use the DefaultTypeSelector for the type mapping. The default delimiter is :, so for example a document with the ID user:123 will be indexed under the type user.
    • couchbase.typeSelector.documentTypeDelimiter - Optional. The delimiter to use for the DelimiterTypeSelector. Default is :.
  • org.elasticsearch.transport.couchbase.capi.GroupRegexTypeSelector - Maps documents that match the specified regular expression with a capture group named type. If the document doesn't match the regular expression, or the regular expression doesn't define a capture group named type, the DefaultTypeSelector is used instead.
    • couchbase.typeSelector.documentTypesRegex - Specified the regular expression for mapping Couchbase document IDs to ElasticSearch types. Example: ^(?<type>\w+)::.+$ will map document IDs of the format <type>::<stuff> to the type <type>, so the ID user::123 will be indexed under the type user.
  • org.elasticsearch.transport.couchbase.capi.RegexTypeSelector - Maps document IDs that match the specified regular expressions to the named types. If the ID doesn't match any of the specified expressions, DefaultTypeSeletor is used to select the type.
    • couchbase.typeSelector.documentTypesRegex.* - Specifies a regular expression with a named type. For example, couchbase.typeSelector.documentTypesRegex.users: ^user-.+$ will map all document IDs that start with the string user- to the type users.

Mapping parent-child relationships

• couchbase.parentSelector - The parent selector class to use for mapping child documents to parents. Note that because of the nature of XDCR, it's possible that the child document will be replicated before the parent, leading to unpredictable behaviour on the ElasticSearch side.
  • org.elasticsearch.transport.couchbase.capi.DefaultParentSelector - Maps documents to parents according to a predefined map of types to field names.
    • couchbase.parentSelector.documentTypeParentFields.* - Specifies which document field contains the ID of the parent document for that particular type. For example, couchbase.parentSelector.documentTypeParentFields.order: doc.user_id will set the parent ID of all documents in the type order to the value of the user_id field.
  • org.elasticsearch.transport.couchbase.capi.RegexParentSelector - Maps documents to parents according to a specified regular expression with the capture group parent. Optionally lets you specify the format for the parent document ID.
    • couchbase.parentSelector.documentTypesParentRegex.* - A named regular expression for matching the parent document ID. For example, couchbase.documentTypesParentRegex.typeA: ^typeA::(?<parent>.+) with the document ID typeA::123 will use 123 as the parent document ID.
    • couchbase.parentSelector.documentTypesParentFormat.* - Specifies an optional format for the parent document ID matched by the regular expression above. Uses <parent> as the placeholder for the matched ID. For example, couchbase.documentTypesParentFormat.typeA: parentType::<parent> with the previous example will produce the parent document ID parentType::123.

Specifying custom document routing

• couchbase.documentTypeRoutingFields.* - A mapping of types to custom document routing paths. For example, specifying couchbase.documentTypeRoutingFields.users: user_id will use the field user_id as the custom routing path for type users.

Filtering documents on the ElasticSearch side

• couchbase.keyFilter - The document filter class to use for filtering documents on the plugin side. Note that Couchbase sends all documents through XDCR no matter what, the document filter simply chooses whether to index or ignore certain documents according to their ID.
  • org.elasticsearch.transport.couchbase.capi.DefaultKeyFilter - The default filter, which lets all documents through. Can be omitted from the configuration file.
  • org.elasticsearch.transport.couchbase.capi.RegexKeyFilter - The
    • couchbase.keyFilter.type - [include|exclude] Specifies whether the filter will include or exclude the documents based on the matched regular expression. If include, then only documents with IDs that match one of the regular expressions will be indexed. If exclude, then only documents that do not match any of the regular expressions will be indexed.
    • couchbase.keyFilter.keyFiltersRegex.* - Specifies one or more regular expressions to match against the document ID before indexing them in ElasticSearch. For example, couchbase.keyFilter.type: exclude + couchbase.keyFilter.keyFiltersRegex.temp: ^temp.*$ will cause the plugin to ignore any documents whose IDs start with temp.

v2.1.1-RC

This is a release candidate of the 2.1.1 version of the ElasticSearch transport-couchbase plugin. It includes compatibility with ES up to v1.5.2, multiple new features and further bug fixes, including several issues found in 2.1.0. In particular, this release fixes a long-standing bug with an incorrect concurrent bulk request counter, which could eventually cause the plugin to stop accepting requests from Couchbase Server altogether.

Note that some of the configuration option names have changed, please consult the readme below.

Installation

To install the RC, run the following command from your ElasticSearch installation folder:

bin/plugin -install transport-couchbase -url https://github.com/couchbaselabs/elasticsearch-transport-couchbase/releases/download/2.1.1-RC/elasticsearch-transport-couchbase-2.1.1-SNAPSHOT.zip

Configuration

Configuration for the plugin is specified as part of the ElasticSearch config file (usually elasticsearch.yml) and is currently only read when ElasticSearch starts. Dynamic configuration support is planned for the future.

Basic Settings

• couchbase.port - The port the plugin will listen on, default 9091
• couchbase.username - The username for HTTP basic auth, default Administrator
• couchbase.password - The password for HTTP basic auth, no default
• couchbase.num_vbuckets - The number of vbuckets that ElasticSearch should pretend to have (default on Mac is 64, 1024 on all other platforms) This value MUST match the number of vbuckets on the source Couchbase cluster.
• couchbase.maxConcurrentRequests - The number of concurrent requests that the plugin will allow, default 1024 (lower this if the load on the machine gets too high)

Advanced Settings

• couchbase.ignoreFailures - Enabling this flag will cause the plugin to return a success status to Couchbase even if it cannot index some of the documents. This will prevent the XDCR replication from being stalled due to indexing errors in ElasticSearch, for example when a schema change breaks some of the ES type mappings. Default is false.
• couchbase.ignoreDeletes - Specifying one or more index names here will cause the plugin to ignore document deletion and expiration for those indexes. This can be used to turn ElasticSearch into a sort of searchable archive for a Couchbase bucket. Note that this also means that the index will continue to grow indefinitely.
• couchbase.wrapCounters - Enabling this flag will cause the plugin to wrap integer values from Couchbase, which are not valid JSON documents, in a simple document before indexing them in ElasticSearch. The resulting document is in the format { "value" : <value> } and is stored under the ID of the original value from Couchbase.

Mapping Couchbase documents to ElasticSearch types

• couchbase.typeSelector - The type selector class to use for mapping documents to types.
  • org.elasticsearch.transport.couchbase.capi.DefaultTypeSelector - Maps all documents to the specified type. As the name implies, this is the default type selector and can be omitted from the configuration file.
    • couchbase.typeSelector.defaultDocumentType - The document type to which the DefaultTypeSelector will map all documents. Defaults to "couchbaseDocument".
    • couchbase.typeSelector.checkpointDocumentType - The document type to which replication checkpoint documents will be mapped. Defaults to "couchbaseCheckpoint".
  • org.elasticsearch.transport.couchbase.capi.DelimiterTypeSelector - If the document ID is of the format <type><delimiter><*>, this type selector will map these documents to the type <type>, otherwise it will use the DefaultTypeSelector for the type mapping. The default delimiter is :, so for example a document with the ID user:123 will be indexed under the type user.
    • couchbase.typeSelector.documentTypeDelimiter - Optional. The delimiter to use for the DelimiterTypeSelector. Default is :.
  • org.elasticsearch.transport.couchbase.capi.GroupRegexTypeSelector - Maps documents that match the specified regular expression with a capture group named type. If the document doesn't match the regular expression, or the regular expression doesn't define a capture group named type, the DefaultTypeSelector is used instead.
    • couchbase.typeSelector.documentTypesRegex - Specified the regular expression for mapping Couchbase document IDs to ElasticSearch types. Example: ^(?<type>\w+)::.+$ will map document IDs of the format <type>::<stuff> to the type <type>, so the ID user::123 will be indexed under the type user.
  • org.elasticsearch.transport.couchbase.capi.RegexTypeSelector - Maps document IDs that match the specified regular expressions to the named types. If the ID doesn't match any of the specified expressions, DefaultTypeSeletor is used to select the type.
    • couchbase.typeSelector.documentTypesRegex.* - Specifies a regular expression with a named type. For example, couchbase.typeSelector.documentTypesRegex.users: ^user-.+$ will map all document IDs that start with the string user- to the type users.

Mapping parent-child relationships

• couchbase.parentSelector - The parent selector class to use for mapping child documents to parents. Note that because of the nature of XDCR, it's possible that the child document will be replicated before the parent, leading to unpredictable behaviour on the ElasticSearch side.
  • org.elasticsearch.transport.couchbase.capi.DefaultParentSelector - Maps documents to parents according to a predefined map of types to field names.
    • couchbase.parentSelector.documentTypeParentFields.* - Specifies which document field contains the ID of the parent document for that particular type. For example, couchbase.parentSelector.documentTypeParentFields.order: doc.user_id will set the parent ID of all documents in the type order to the value of the user_id field.
  • org.elasticsearch.transport.couchbase.capi.RegexParentSelector - Maps documents to parents according to a specified regular expression with the capture group parent. Optionally lets you specify the format for the parent document ID.
    • couchbase.parentSelector.documentTypesParentRegex.* - A named regular expression for matching the parent document ID. For example, couchbase.documentTypesParentRegex.typeA: ^typeA::(?<parent>.+) with the document ID typeA::123 will use 123 as the parent document ID.
    • couchbase.parentSelector.documentTypesParentFormat.* - Specifies an optional format for the parent document ID matched by the regular expression above. Uses <parent> as the placeholder for the matched ID. For example, couchbase.documentTypesParentFormat.typeA: parentType::<parent> with the previous example will produce the parent document ID parentType::123.

Specifying custom document routing

• couchbase.documentTypeRoutingFields.* - A mapping of types to custom document routing paths. For example, specifying couchbase.documentTypeRoutingFields.users: user_id will use the field user_id as the custom routing path for type users.

Filtering documents on the ElasticSearch side

• couchbase.keyFilter - The document filter class to use for filtering documents on the plugin side. Note that Couchbase sends all documents through XDCR no matter what, the document filter simply chooses whether to index or ignore certain documents according to their ID.
  • org.elasticsearch.transport.couchbase.capi.DefaultKeyFilter - The default filter, which lets all documents through. Can be omitted from the configuration file.
  • org.elasticsearch.transport.couchbase.capi.RegexKeyFilter - The
    • couchbase.keyFilter.type - [include|exclude] Specifies whether the filter will include or exclude the documents based on the matched regular expression. If include, then only documents with IDs that match one of the regular expressions will be indexed. If exclude, then only documents that do not match any of the regular expressions will be indexed.
    • couchbase.keyFilter.keyFiltersRegex.* - Specifies one or more regular expressions to match against the document ID before indexing them in ElasticSearch. For example, couchbase.keyFilter.type: exclude + couchbase.keyFilter.keyFiltersRegex.temp: ^temp.*$ will cause the plugin to ignore any documents whose IDs start with temp.