Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Message /search API : Support CJK full text search #630

Open
GrassBlock1 opened this issue Nov 29, 2024 · 8 comments
Open

Message /search API : Support CJK full text search #630

GrassBlock1 opened this issue Nov 29, 2024 · 8 comments
Labels
enhancement New feature or request no-priority

Comments

@GrassBlock1
Copy link

GrassBlock1 commented Nov 29, 2024

Clients like cinny implements message (room events) search via the /_matrix/client/v3/search api, and it works out of the box for English and other language using the space to divide the words. But when message was written in Chinese (Korean and Japanese), conduwuit only returns the message that starts with the word.

For example, when searching for "可能", it only returns message like this:
image

and messages like "还有可能是定期检修" can't be found by this method.

I also tried to look up the conduwuit documentation but seems that it's not mentioned.

I would like to know if there are solutions for this kind of messages as I use Chinese as my main language and it will truly make experience better for users who use them too.

@Kimiblock
Copy link
Contributor

Use clients that implement searching themselves, like nheko. AFAIK Synapse also has this problem.

@GrassBlock1
Copy link
Author

Use clients that implement searching themselves, like nheko. AFAIK Synapse also has this problem.

Oh that's a way.
Since I have account using nheko and it doesn't support multiple accounts, (fluffychat also supports this but it was laggy on my pc) I had to use cinny.

keep open in hope there will be better support for that.

@Kimiblock
Copy link
Contributor

nheko --help | grep profile

@GrassBlock1
Copy link
Author

GrassBlock1 commented Nov 29, 2024

nheko --help | grep profile

Oh it works well for me, thanks!

@girlbossceo girlbossceo added enhancement New feature or request no-priority labels Nov 29, 2024
@girlbossceo
Copy link
Owner

The text search implementation we inherited is pretty bad, so it's likely that a whole new system would need to be developed. Synapse has the same problems and has a poor text search implementation too.

@GrassBlock1
Copy link
Author

The text search implementation we inherited is pretty bad, so it's likely that a whole new system would need to be developed. Synapse has the same problems and has a poor text search implementation too.

I see.
Is there any plans on it?

@girlbossceo
Copy link
Owner

Not in the near future, which is why I marked it as no priority, especially with the various other higher priority stuff we have going on.

It's a very huge undertaking of developing an in-house full text search implementation, and arguably maybe it shouldn't even be our problem and let something like ElasticSearch or whatever deal with it.

@GrassBlock1
Copy link
Author

Not in the near future, which is why I marked it as no priority, especially with the various other higher priority stuff we have going on.

OK, thanks for your consideration.

It's a very huge undertaking of developing an in-house full text search implementation, and arguably maybe it shouldn't even be our problem and let something like ElasticSearch or whatever deal with it.

Yes, I think it's a way to let external search engine to manage these stuff since I have got Meilisearch working and it works well for CJK search.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request no-priority
Projects
None yet
Development

No branches or pull requests

3 participants