Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pika 4.0 使用 codis-fe 迁移 slot 时偶尔有报错 #2870

Closed
zenxme opened this issue Aug 16, 2024 · 7 comments
Closed

Pika 4.0 使用 codis-fe 迁移 slot 时偶尔有报错 #2870

zenxme opened this issue Aug 16, 2024 · 7 comments
Labels
☢️ Bug Something isn't working

Comments

@zenxme
Copy link

zenxme commented Aug 16, 2024

Is this a regression?

Yes

Description

一边写入数据, 一边迁移, 就会有概率出现这个问题

codis-fe 报错截图:
image

客户端收到的错误:
redis.exceptions.ResponseError: handle request, bad slotsmgrt-exec-wrapper resp: type(array[0]) = <int>, len(array[0].value) = 2

pika-server 迁移源节点日志:
pika_migrate_thread.cc:662] PikaMigrateThread::ReqMigrateOne Slot : 886 is not the migrating slot:-1

pika-server 目标节点日志:

W20240816 10:16:59.247891   349 pika_migrate_thread.cc:662] PikaMigrateThread::ReqMigrateOne Slot : 886 is not the migrating slot:25
W20240816 10:16:59.247941   341 pika_slot_command.cc:1250] Slot batch migrate keys error
I20240816 10:16:59.440676   419 pika_migrate_thread.cc:943] PikaMigrateThread::ThreadMain send_num:367 response_num:367
I20240816 10:16:59.440834   419 pika_migrate_thread.cc:967] PikaMigrateThread::ThreadMain slot_size:367 moved_num:367
I20240816 10:16:59.440999   419 pika_migrate_thread.cc:755] PikaMigrateThread::DestroyThread
I20240816 10:16:59.441212   420 pika_migrate_thread.cc:519] PikaParseSendThread::ThreadMain :140736425092800 exit !!!

Please provide a link to a minimal reproduction of the bug

No response

Screenshots or videos

No response

Please provide the version you discovered this bug in (check about page for version information)

No response

Anything else?

No response

@zenxme zenxme added the ☢️ Bug Something isn't working label Aug 16, 2024
@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


Title: Pika 4.0 occasionally reports errors when using codis-fe to migrate slots.

@chejinge
Copy link
Collaborator

chejinge commented Sep 6, 2024

迁移数据的时候是会对slot进行加锁的,加锁的时候这个slot读写是阻塞的,所以读写请求都进不去

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


When migrating data, the slot will be locked. When locked, reading and writing in this slot are blocked, so read and write requests cannot enter.

@zenxme
Copy link
Author

zenxme commented Sep 6, 2024

迁移数据的时候是会对slot进行加锁的,加锁的时候这个slot读写是阻塞的,所以读写请求都进不去

如果要加锁的话怎么实现在不影响在线业务的情况下,动态迁移slot ? codis 是可以的,pika 不支持吗?

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


When migrating data, the slot will be locked. When locked, the slot is blocked for reading and writing, so read and write requests cannot enter.

If locking is required, how can the slot be dynamically migrated without affecting the online business? Codis is ok, but pika doesn’t support it?

@chejinge
Copy link
Collaborator

迁移数据的时候是会对slot进行加锁的,加锁的时候这个slot读写是阻塞的,所以读写请求都进不去

如果要加锁的话怎么实现在不影响在线业务的情况下,动态迁移slot ? codis 是可以的,pika 不支持吗?

尽可能的把数据打散,把slot个数多一些 会在迁移的过程中对于用户的影响尽可能的少

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


When migrating data, the slot will be locked. When locked, the slot is blocked for reading and writing, so read and write requests cannot enter.

If locking is required, how can the slot be dynamically migrated without affecting the online business? Codis is ok, but pika doesn’t support it?

Scattering the data as much as possible and increasing the number of slots will have as little impact on users as possible during the migration process.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
☢️ Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants