DELETE /conversation/{conversationId} does not delete the conversation

[ThomasRoehl]

Expected Behaviour

I use the API to delete an existing conversation from Smarti.
The conversation still exists in the database and the GET /legacy/rocket.chat still returns the deleted conversation with a 200 Code.
On the other hand GET /conversation/{conversationId} can't find the conversation anymore and returns a 400 Code.

Actual Behaviour

DELETE /conversation/{conversationId} should delete the conversation from the database

Steps to Reproduce

• POST /conversation
• GET /legacy/rocket.chat returns the conversation for the specified channelId
• GET /conversation/{conversationId} returns the conversation for the specified conversationId
• DELETE /conversation/{conversationId} returns success for deletion 204
• GET /conversation/{conversationId} returns 404 conversation not found
• GET /legacy/rocket.chat still returns the conversation for the specified channelId
• DELETE /conversation/{conversationId} returns 404 conversation not found

Environment

Smarti version 0.7.4-SNAPSHOT
CommitId: d4db0ea

[ruKurz]

See also: #95

[westei]

The DELETE operation does mark the conversation as deleted AND deletes all messages. The conversation and its metadata are preserved as those information are required for cloud-sync of the embedded Solr Servers managed by the different Smarti instances.

For clean up one could run a Mongo task that physically deletes all conversations that where marked as deleted for longer as a given time period (e.g. 1 day).

NOTES:

• Even when using an external Solr those Information are critical to sync the Index after a down period (as one needs to know what conversations need to be deleted from the index).
• When a clean up process is in place Solr Indexes that are out of sync for longer as the configured time period of the clean up process need to be deleted and fully rebuild.

[westei]

#259 (#260) aim to fix that GET /legacy/rocket.chat returns conversations that are marked as deleted.

[ruKurz]

Hi @westei,

thanks for investigating on this issue and fixing the PR.

I like the idea using a configureable time period to run a clean up process for obsolete conversations.

I noticed that deleting conversations (and analysis) on MongoDB results in Solr responses that have a numfound e.g. 26, but the docs in the response is less or 0. So it would be nice to trigger the Solr synchronization manually (e.g. via API).

IMOH we should create a separate issue about "Smarti housekeeping".

[westei]

noticed that deleting conversations (and analysis) on MongoDB results in Solr responses that have a numfound e.g. 26, but the docs in the response is less or 0. So it would be nice to trigger the Solr synchronization manually (e.g. via API).

This could be the case If the conversations in the Response are actually loaded from the MongoDB (based on the ID in the SolrIndex) and NOT reconstructured from the data in the SolrIndex. I will check if this is the case, because it could also explain why sometimes the numFound does not correspond with the number of actual results

IMOH we should create a separate issue about "Smarti housekeeping".

This is part of #222

[westei]

noticed that deleting conversations (and analysis) on MongoDB results in Solr responses that have a numfound e.g. 26, but the docs in the response is less or 0. So it would be nice to trigger the Solr synchronization manually (e.g. via API).

This could be the case If the conversations in the Response are actually loaded from the MongoDB (based on the ID in the SolrIndex) and NOT reconstructured from the data in the SolrIndex. I will check if this is the case, because it could also explain why sometimes the numFound does not correspond with the number of actual results

The ConversationSearchService loads Conversations and Messages from the MongoDB. In case a Conversation present in the SolrIndex is missing in the MongoDB the following Log message is written on IINFO level

ConversationIndex out of Sync with ConversationRepository: Conversation[id:{id}] is present in the index but not in the Repository!

Solr Results for those conversations are removed. So the number of Results will therefore differ from the numResults reported by Solr

[westei]

We can configure Mongo to delete conversations that are maked as deleted automatically after a predefined period see https://docs.mongodb.com/manual/core/index-ttl/

[westei]

will move this to v0.9.0 as Cloudsync might get away with the Solr Update and if this is the case this will be resolved as we could delete conversations instead of marking them as deleted