After update of a document values are missing in FULLTEXT index

[EricSchreiner]

OrientDB Version: 3.0.8, 3.0.9

Java Version: <1.8>

OS: <Windows 10>

Expected behavior

After changing a value in a document it seems that values disappear from the Index.
I've attached a unittest to reproduce. This seems to be new. Before orientdb 3.x
we did not have that problem.

Actual behavior

Search for "mytesttag" in Index
select from index:idxkeywords where key = ? (?=mytesttag)
Number of documents found in index: 1

Now Update Document
Found indexDocument: {key:mytesttag,rid:#25:44} Docid=44
Found document: test#25:44{name:Test44,keywords:keywordone44tag tagone44tag tagtwo44tag mytesttag} v1
Updated document: test#25:44{name:Test44,keywords:keywordone44 tagone44 tagtwo44 mytesttag onemorekeyword} v2

Now search again
select from index:idxkeywords where key = ? (?=mytesttag)
Number of documents found in index: 0 THIS SHOULD BE ONE AND NOT ZERO

Steps to reproduce

`
package de.contecon.picapport.db;

import java.io.File;
import java.util.List;

import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;

import com.orientechnologies.orient.core.db.ODatabasePool;
import com.orientechnologies.orient.core.db.ODatabaseSession;
import com.orientechnologies.orient.core.db.ODatabaseType;
import com.orientechnologies.orient.core.db.OrientDB;
import com.orientechnologies.orient.core.db.OrientDBConfig;
import com.orientechnologies.orient.core.id.ORecordId;
import com.orientechnologies.orient.core.metadata.schema.OClass;
import com.orientechnologies.orient.core.metadata.schema.OClass.INDEX_TYPE;
import com.orientechnologies.orient.core.metadata.schema.OType;
import com.orientechnologies.orient.core.record.ORecord;
import com.orientechnologies.orient.core.record.impl.ODocument;
import com.orientechnologies.orient.core.sql.executor.OResultSet;
import com.orientechnologies.orient.core.sql.query.OSQLSynchQuery;

public class TestOrientDbIndexUpdate {
private int insertcount;
private String databaseName= this.getClass().getSimpleName();
private OrientDB orientDB;
private ODatabasePool dbPool;

@before
public void setUp() throws Exception {
String databaseName= this.getClass().getSimpleName();
orientDB = new OrientDB("embedded:./playground/target/databases/", OrientDBConfig.defaultConfig());
orientDB.create(databaseName, ODatabaseType.PLOCAL);
ODatabaseSession dbs = orientDB.open(databaseName,"admin","admin");
insertcount = 100;
OClass test = dbs.getMetadata().getSchema().createClass("test");
dbs.command("ALTER CLASS test CLUSTERSELECTION default"); // This is very imortant for us
test.createProperty("name", OType.STRING);
test.createProperty("keywords", OType.STRING).setMandatory(false);
test.createIndex("idxkeywords", INDEX_TYPE.FULLTEXT, "keywords");
dbs.close();
dbPool = new ODatabasePool(orientDB, databaseName,"admin","admin");
}

@after
public void tearDown() throws Exception {
//drop
dbPool.close();
orientDB.drop(databaseName);
File dbFolder = new File("./playground/target/databases/"+ databaseName);
Assert.assertEquals(false, dbFolder.exists());
}

@test
public void testOdbIdexUpdate1() {
ODatabaseSession dbs = dbPool.acquire();
fillDb(dbs, insertcount);
dbs.close();
// Search Document
testExistenceOfTag("mytesttag");
updateKeyword("mytesttag", "keywordone44 tagone44 tagtwo44 mytesttag onemorekeyword");
testExistenceOfTag("mytesttag"); // <---- Test fails here !!!!!!
}

private void testExistenceOfTag(String tag) {
ODatabaseSession dbs = dbPool.acquire();
System.out.println("select from index:idxkeywords where key = ? (?="+tag+")");
List result =dbs.command(new OSQLSynchQuery("select from index:idxkeywords where key = ?")).execute(tag);
System.out.println("Number of documents found in index: "+result.size());
Assert.assertEquals(1, result.size());
dbs.close();
}

private void updateKeyword(String tag, String newKeyword) {
ODatabaseSession dbs = dbPool.acquire();
List result =dbs.command(new OSQLSynchQuery("select from index:idxkeywords where key = ?")).execute(tag);
Assert.assertEquals(1, result.size());
// Search document in index
ODocument oDocumentIndex = result.get(0);
int docId = getDocId(oDocumentIndex);
System.out.println("Found indexDocument: "+oDocumentIndex.getRecord().toString()+ " Docid="+docId);
// load original Document aud update
int clusterId = dbs.getClusterIdByName("test");
ORecordId oRecordId = new ORecordId(clusterId, docId);
ODocument oDocumentTest = dbs.load(oRecordId);
System.out.println("Found document: "+oDocumentTest);
oDocumentTest.field("keywords", newKeyword);
ORecord saved = dbs.save(oDocumentTest);
System.out.println("Updated document: "+oDocumentTest);
dbs.close();
}

private void fillDb(ODatabaseSession dbs, int count) {
for (int i = 0; i < count; i++) {
String keywords="keywordone"+i +"tag tagone"+i+ "tag tagtwo"+i+"tag";
if(i==44) {
keywords+=" mytesttag";
}
ODocument doc = new ODocument("test");
doc.field("name", "Test" + i);
doc.field("keywords", keywords);
ORecord saved = dbs.save(doc);
System.out.println("Insert document: "+saved);
}
OResultSet result = dbs.command("select * from test");
Assert.assertEquals(count, result.elementStream().count());
result.close();
}

private int getDocId(ODocument iDoc) {
String sid = iDoc.getRecord().toString();
int id=parsePid(sid, 1);
return id;
}

private final int parsePid(String recId, int rt) {
return string2int(recId.substring(recId.lastIndexOf(':')+1, recId.length()-rt));
}

private final int string2int(String in) {
int i=0;
try {
i=Integer.parseInt(in.trim());
}
catch(Exception e) {
i=0;
}

return i;
}

}

`

[EricSchreiner]

Hi @luigidellaquila,
for me it seems that @tglman is currently not active. We use orientdb since version1 in our free photoserver picapport(www.picapport.de) . The described problem above causes us a lot of trouble at the moment. It would be great if someone can have a look at it. I think this is really a serious bug.

[EricSchreiner]

[EricSchreiner]

Hi @luigidellaquila, hi @tglman,
problem still exists wit 3.0.9

[wolf4ood]

Hi @EricSchreiner

i've reproduce this issue and trying to fix this bug

Thanks

[EricSchreiner]

Hi @wolf4ood
thank you. Let me know if you need help or more information.

[EricSchreiner]

Hi @wolf4ood, hi @luigidellaquila , hi @tglman
any news on this. The issue has not even been tagged as bug or solved ???????

[lvca]

@EricSchreiner, have you tried using LUCENE index for full-text instead of the internal one? It's way more stable and fast than the (old) internal one.

[EricSchreiner]

Has this issue been fixed? If yes in which release?

[andrii0lomakin]

Hi @EricSchreiner we do not support Full-Text index anymore for a long time because of many issues with it. Please use the Lucene index for such cases.