Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Query failed with error failed to query: segment lacks[segment=xxx]: channel not available after upgrade from v2.2.14 to 2.3-20231226-6eeb4b7f #29523

Closed
1 task done
zhuwenxing opened this issue Dec 27, 2023 · 5 comments
Closed
1 task done

[Bug]: Query failed with error failed to query: segment lacks[segment=xxx]: channel not available after upgrade from v2.2.14 to 2.3-20231226-6eeb4b7f #29523

zhuwenxing opened this issue Dec 27, 2023 · 5 comments
Assignees
@zhuwenxing @weiliu1031
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.3.4

Comments

@zhuwenxing
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:v2.2.14 --> 2.3-20231226-6eeb4b7f
- Deployment mode(standalone or cluster):both
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior


[2023-12-27T03:25:35.798Z] [2023-12-27 03:24:14 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)

[2023-12-27T03:25:35.798Z] [2023-12-27 03:24:18 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2023-12-27T03:25:35.798Z] [2023-12-27 03:24:18 - DEBUG - ci_test]: (api_request)  : [Collection.search] args: [[[0.12656170373220194, 0.1303871444704426, 0.129108794269997, 0.11839435940154026, 0.09930617262445839, 0.09770409513730348, 0.10455359077904242, 0.02051758339735825, 0.1301441327010182, 0.14150575286566736, 0.11910017877586448, 0.008290466696869411, 0.07199358749987018, 0.09782508606531198, 0.0449......, kwargs: {} (api_request.py:62)

[2023-12-27T03:25:35.798Z] [2023-12-27 03:24:18 - DEBUG - ci_test]: (api_response) : ["['id: 5985400146088504466, distance: 22.133277893066406, entity: {}']"]  (api_request.py:37)

[2023-12-27T03:25:35.798Z] [2023-12-27 03:24:18 - INFO - ci_test]: assert search: 0.44865870475769043 (test_all_collections_after_chaos.py:100)

[2023-12-27T03:25:35.798Z] [2023-12-27 03:24:18 - DEBUG - ci_test]: (api_request)  : [Collection.query] args: ['int64 in [-3000, -2999, -2998, -2997, -2996, -2995, -2994, -2993, -2992, -2991, -2990, -2989, -2988, -2987, -2986, -2985, -2984, -2983, -2982, -2981, -2980, -2979, -2978, -2977, -2976, -2975, -2974, -2973, -2972, -2971, -2970, -2969, -2968, -2967, -2966, -2965, -2964, -2963, -2962, -2961, -2960, -......, kwargs: {} (api_request.py:62)

[2023-12-27T03:25:35.798Z] [2023-12-27 03:24:18 - ERROR - pymilvus.decorators]: RPC error: [query], <MilvusException: (code=503, message=failed to query: segment lacks[segment=446600831088804466]: channel not available[channel=by-dev-rootcoord-dml_0_446600439437411774v1])>, <Time:{'RPC start': '2023-12-27 03:24:18.907731', 'RPC error': '2023-12-27 03:24:18.970291'}> (decorators.py:128)

[2023-12-27T03:25:35.798Z] [2023-12-27 03:24:18 - ERROR - ci_test]: Traceback (most recent call last):

[2023-12-27T03:25:35.798Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper

[2023-12-27T03:25:35.798Z]     res = func(*args, **_kwargs)

[2023-12-27T03:25:35.798Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request

[2023-12-27T03:25:35.798Z]     return func(*arg, **kwargs)

[2023-12-27T03:25:35.798Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 915, in query

[2023-12-27T03:25:35.798Z]     return conn.query(

[2023-12-27T03:25:35.798Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 129, in handler

[2023-12-27T03:25:35.799Z]     raise e from e

[2023-12-27T03:25:35.799Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 125, in handler

[2023-12-27T03:25:35.799Z]     return func(*args, **kwargs)

[2023-12-27T03:25:35.799Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 164, in handler

[2023-12-27T03:25:35.799Z]     return func(self, *args, **kwargs)

[2023-12-27T03:25:35.799Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 104, in handler

[2023-12-27T03:25:35.799Z]     raise e from e

[2023-12-27T03:25:35.799Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 68, in handler

[2023-12-27T03:25:35.799Z]     return func(*args, **kwargs)

[2023-12-27T03:25:35.799Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 1314, in query

[2023-12-27T03:25:35.799Z]     check_status(response.status)

[2023-12-27T03:25:35.799Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/utils.py", line 54, in check_status

[2023-12-27T03:25:35.799Z]     raise MilvusException(status.code, status.reason, status.error_code)

[2023-12-27T03:25:35.799Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=503, message=failed to query: segment lacks[segment=446600831088804466]: channel not available[channel=by-dev-rootcoord-dml_0_446600439437411774v1])>

[2023-12-27T03:25:35.799Z]  (api_request.py:45)

[2023-12-27T03:25:35.799Z] [2023-12-27 03:24:18 - ERROR - ci_test]: (api_response) : <MilvusException: (code=503, message=failed to query: segment lacks[segment=446600831088804466]: channel not available[channel=by-dev-rootcoord-dml_0_446600439437411774v1])> (api_request.py:46)

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/deploy_test_kafka_for_release_cron/detail/deploy_test_kafka_for_release_cron/1756/pipeline

log:

artifacts-kafka-cluster-reinstall-1756-server-second-deployment-logs.tar.gz

artifacts-kafka-cluster-reinstall-1756-server-first-deployment-logs.tar.gz

artifacts-kafka-cluster-reinstall-1756-pytest-logs.tar.gz

Anything else?

The search works well, but the query failed.

cluster: 4am
namespace: chaos-testing
release name: kafka-cluster-reinstall-1756
@zhuwenxing zhuwenxing added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 27, 2023
@zhuwenxing zhuwenxing added this to the 2.3.4 milestone Dec 27, 2023
@zhuwenxing zhuwenxing added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. labels Dec 27, 2023
@zhuwenxing
Copy link
Contributor Author

cluster mode also failed

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/deploy_test_kafka_for_release_cron/detail/deploy_test_kafka_for_release_cron/1758/pipeline
log:

artifacts-kafka-cluster-upgrade-1758-server-second-deployment-logs.tar.gz
artifacts-kafka-cluster-upgrade-1758-server-first-deployment-logs.tar.gz
artifacts-kafka-cluster-upgrade-1758-pytest-logs.tar.gz

@yanliang567
Copy link
Contributor

/assign @weiliu1031
/unassign

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 27, 2023
This was referenced Dec 27, 2023
Merged
Closed
Merged
sre-ci-robot pushed a commit that referenced this issue Dec 28, 2023
@weiliu1031
fix: Choose wrong shard leader during balance channel(#29525) (#29532)
07ef52e 
issue: #29523
pr: #29525

readable shard leader should still be the old one during channel
balance, if the new shard leader is not ready.
This PR fixed that query coord choose wrong shard leader during balance
channel

Signed-off-by: Wei Liu <[email protected]>
sre-ci-robot pushed a commit that referenced this issue Dec 28, 2023
@weiliu1031
fix: Choose wrong shard leader during balance channel (#29529)
5474bce 
issue: #29523

readable shard leader should still be the old one during channel
balance, if the new shard leader is not ready.
This PR fixed that query coord choose wrong shard leader during balance
channel

Signed-off-by: Wei Liu <[email protected]>
@weiliu1031
Copy link
Contributor

please verify this with latest image

@weiliu1031
Copy link
Contributor

/assign @zhuwenxing

@zhuwenxing
Copy link
Contributor Author

zhuwenxing commented Dec 29, 2023 •
edited
Loading

Not reproduced in recent run.
Remove the critical label, but keep watching

@zhuwenxing zhuwenxing removed priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. labels Dec 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zhuwenxing zhuwenxing

@weiliu1031 weiliu1031

Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.3.4
Development

No branches or pull requests
3 participants
@zhuwenxing @yanliang567 @weiliu1031