Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: hybrid search failed on full text index with error code=2200, message=incomplete query result #38528

Open
1 task done
Squire-tomsk opened this issue Dec 17, 2024 · 11 comments
Open
1 task done

[Bug]: hybrid search failed on full text index with error code=2200, message=incomplete query result #38528

Squire-tomsk opened this issue Dec 17, 2024 · 11 comments
Assignees
@congqixia
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.3

Comments

@Squire-tomsk
Copy link

Squire-tomsk commented Dec 17, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: 2.5.0-beta
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 2.5.0
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

When I perform hybrid search call:

sparse_req = AnnSearchRequest(
    data=[keywords_separate_with_whitespace],
    anns_field="sparse_embedding",
    param={"metric_type": "BM25", "drop_ratio_search": 0.2},
    limit=10
)

dense_req = AnnSearchRequest(
    data=[dense_emb_1024_dim],
    anns_field="embedding",
    param={"metric_type": "COSINE", "params": {}},
    limit=10
)

ranker = RRFRanker(60)

collection.hybrid_search(
    [dense_req, sparse_req],
    rerank=ranker,
    limit=10,
    output_fields=["text"],
)

I see

RPC error: [hybrid_search], <MilvusException: (code=2200, message=incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result)>, <Time:{'RPC start': '2024-12-17 09:00:43.574024', 'RPC error': '2024-12-17 09:01:05.340891'}>

Explicit search by dense and sparse embeddings separatly works correct on same collection.

Steps To Reproduce

1. I have used default Milvus standalone 2.5.0-beta docker.
2. Collection schema


schema = MilvusClient.create_schema(auto_id=False, enable_dynamic_field=True)
schema.add_field(
    field_name="id",
    datatype=DataType.VARCHAR,
    max_length=65535,
    is_primary=True,
)
schema.add_field(
    field_name="embedding",
    datatype=DataType.FLOAT_VECTOR,
    dim=1024,
)
schema.add_field(
    field_name="text",
    datatype=DataType.VARCHAR, 
    max_length=65_535, 
    enable_analyzer=False
)
schema.add_field(
    field_name="keywords",
    datatype=DataType.VARCHAR, 
    max_length=65_535, 
    enable_analyzer=True
)
schema.add_field(
    field_name="sparse_embedding",
    datatype=DataType.SPARSE_FLOAT_VECTOR,
)
bm25_function = Function(
    name="text_bm25_emb",
    input_field_names=["keywords"],
    output_field_names=["sparse_embedding"],
    function_type=FunctionType.BM25,
)
schema.add_function(bm25_function)
milvusclient.create_collection(
    collection_name=collection_name, schema=schema
)
  1. Indexes
collection = Collection(collection_name, using=milvusclient._using)

sparse_index = {"index_type": "AUTOINDEX", "metric_type": "BM25"}
collection.create_index("sparse_embedding", sparse_index)

dense_index = {"params": {}, "metric_type": "COSINE", "index_type": "FLAT"}
collection.create_index("embedding", dense_index)
  1. Insert few thousands records and delete few hundreds.
  2. Bug occurs. The bug does not occur immediately on a small amount of data, but if it occurs, it is reproduced stably.


### Milvus Log

2024-12-17 12:20:19 [2024/12/17 09:20:19.977 +00:00] [INFO] [datacoord/services.go:872] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=fa34366b1ee90bcb3a6309d95265e658] [collectionID=454651567748415768] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_454651567748415768v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=10] ["# of indexed segments"=0] ["# of l0 segments"=0]
2024-12-17 12:20:21 [2024/12/17 09:20:21.389 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=454651567748415768] [replicaID=454651567907799041] [nodes="[\"{NodeID: 2, AssignedScore: 26150.000000, CurrentScore: 28765.000000, Priority: 2615}\"]"]
2024-12-17 12:20:22 [2024/12/17 09:20:22.022 +00:00] [INFO] [proxy/impl.go:6169] ["connect received"] [traceID=6a83fc36b562fd77d6315fa7055af419] [sdk_type=Python] [sdk_version=2.5.0] [local_time="2024-12-17 09:20:22.020270"] [user=] [host=docker-desktop] [db=default]
2024-12-17 12:20:22 [2024/12/17 09:20:22.022 +00:00] [INFO] [rootcoord/root_coord.go:1007] ["received request to list databases"] [traceID=6a83fc36b562fd77d6315fa7055af419] [msgID=0]
2024-12-17 12:20:22 [2024/12/17 09:20:22.022 +00:00] [INFO] [rootcoord/task.go:176] [NewLockerKeyChain] [lockerKeys=1]
2024-12-17 12:20:22 [2024/12/17 09:20:22.022 +00:00] [INFO] [rootcoord/root_coord.go:1033] ["done to list databases"] [traceID=6a83fc36b562fd77d6315fa7055af419] [msgID=0] ["num of databases"=1]
2024-12-17 12:20:22 [2024/12/17 09:20:22.023 +00:00] [INFO] [connection/manager.go:106] ["client register"] [traceID=6a83fc36b562fd77d6315fa7055af419] [sdk_type=Python] [sdk_version=2.5.0] [local_time="2024-12-17 09:20:22.020270"] [user=] [host=docker-desktop] [identifier=454669689687638018] [last_active_time=2024/12/17 09:20:22.023 +00:00]
2024-12-17 12:20:22 [2024/12/17 09:20:22.026 +00:00] [INFO] [rootcoord/task.go:176] [NewLockerKeyChain] [lockerKeys=2]
2024-12-17 12:20:22 [2024/12/17 09:20:22.029 +00:00] [INFO] [rootcoord/task.go:176] [NewLockerKeyChain] [lockerKeys=3]
2024-12-17 12:20:22 [2024/12/17 09:20:22.031 +00:00] [INFO] [rootcoord/task.go:176] [NewLockerKeyChain] [lockerKeys=3]
2024-12-17 12:20:24 [2024/12/17 09:20:24.393 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=454651567748415768] [replicaID=454651567907799041] [nodes="[\"{NodeID: 2, AssignedScore: 26150.000000, CurrentScore: 28765.000000, Priority: 2615}\"]"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.123 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.123 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.123 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.123 +00:00] [WARN] [retry/retry.go:130] ["retry func failed"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [retried=0] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.378 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.378 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.378 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.845 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.845 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.846 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:26 [2024/12/17 09:20:26.702 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:26 [2024/12/17 09:20:26.702 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:26 [2024/12/17 09:20:26.702 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:27 [2024/12/17 09:20:27.388 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=454651567748415768] [replicaID=454651567907799041] [nodes="[\"{NodeID: 2, AssignedScore: 26150.000000, CurrentScore: 28765.000000, Priority: 2615}\"]"]
2024-12-17 12:20:28 [2024/12/17 09:20:28.359 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:28 [2024/12/17 09:20:28.359 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:28 [2024/12/17 09:20:28.359 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:28 [2024/12/17 09:20:28.359 +00:00] [WARN] [retry/retry.go:130] ["retry func failed"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [retried=4] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"]
2024-12-17 12:20:29 [2024/12/17 09:20:29.975 +00:00] [INFO] [rootcoord/task.go:176] [NewLockerKeyChain] [lockerKeys=3]
2024-12-17 12:20:29 [2024/12/17 09:20:29.977 +00:00] [INFO] [querynodev2/services.go:1295] ["sync action"] [traceID=ea79fce7f6e42eab14e68dfac07ca640] [collectionID=454651567748415768] [channel=by-dev-rootcoord-dml_0_454651567748415768v0] [currentNodeID=2] [Action=UpdateVersion] [TargetVersion=1734427219977736171]
2024-12-17 12:20:29 [2024/12/17 09:20:29.977 +00:00] [INFO] [delegator/distribution.go:305] ["Update readable segment version"] [oldVersion=1734427209977139042] [newVersion=1734427219977736171] [growingSegmentNum=0] [sealedSegmentNum=1]
2024-12-17 12:20:29 [2024/12/17 09:20:29.978 +00:00] [INFO] [observers/target_observer.go:531] ["observer trigger update current target"] [collectionID=454651567748415768]
2024-12-17 12:20:29 [2024/12/17 09:20:29.978 +00:00] [INFO] [datacoord/services.go:857] ["get recovery info request received"] [traceID=a5d9baf714036d0dfdcceb4bf1e9a1cc] [collectionID=454651567748415768] [partitionIDs="[]"]
2024-12-17 12:20:29 [2024/12/17 09:20:29.978 +00:00] [INFO] [datacoord/handler.go:244] [GetQueryVChanPositions] [collectionID=454651567748415768] [channel=by-dev-rootcoord-dml_0_454651567748415768v0] [numOfSegments=11] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"="{\"454651567748415769\":0}"]
2024-12-17 12:20:29 [2024/12/17 09:20:29.978 +00:00] [INFO] [datacoord/handler.go:348] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_454651567748415768v0] [posTs=454669679176187907] [posTime=2024/12/17 09:19:41.916 +00:00]
2024-12-17 12:20:29 [2024/12/17 09:20:29.978 +00:00] [INFO] [datacoord/services.go:872] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=a5d9baf714036d0dfdcceb4bf1e9a1cc] [collectionID=454651567748415768] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_454651567748415768v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=10] ["# of indexed segments"=0] ["# of l0 segments"=0]
2024-12-17 12:20:30 [2024/12/17 09:20:30.388 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=454651567748415768] [replicaID=454651567907799041] [nodes="[\"{NodeID: 2, AssignedScore: 26150.000000, CurrentScore: 28765.000000, Priority: 2615}\"]"]
2024-12-17 12:20:31 [2024/12/17 09:20:31.414 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:31 [2024/12/17 09:20:31.414 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:31 [2024/12/17 09:20:31.414 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:33 [2024/12/17 09:20:33.386 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=454651567748415768] [replicaID=454651567907799041] [nodes="[\"{NodeID: 2, AssignedScore: 26150.000000, CurrentScore: 28765.000000, Priority: 2615}\"]"]
2024-12-17 12:20:34 [2024/12/17 09:20:34.477 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:34 [2024/12/17 09:20:34.477 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:34 [2024/12/17 09:20:34.477 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:36 [2024/12/17 09:20:36.388 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=454651567748415768] [replicaID=454651567907799041] [nodes="[\"{NodeID: 2, AssignedScore: 26150.000000, CurrentScore: 28765.000000, Priority: 2615}\"]"]
2024-12-17 12:20:37 [2024/12/17 09:20:37.552 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:37 [2024/12/17 09:20:37.552 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:37 [2024/12/17 09:20:37.552 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:39 [2024/12/17 09:20:39.387 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=454651567748415768] [replicaID=454651567907799041] [nodes="[\"{NodeID: 2, AssignedScore: 26150.000000, CurrentScore: 28765.000000, Priority: 2615}\"]"]
2024-12-17 12:20:39 [2024/12/17 09:20:39.976 +00:00] [INFO] [rootcoord/task.go:176] [NewLockerKeyChain] [lockerKeys=3]
2024-12-17 12:20:39 [2024/12/17 09:20:39.977 +00:00] [INFO] [querynodev2/services.go:1295] ["sync action"] [traceID=b5f8a653173085e961b873d451801ace] [collectionID=454651567748415768] [channel=by-dev-rootcoord-dml_0_454651567748415768v0] [currentNodeID=2] [Action=UpdateVersion] [TargetVersion=1734427229979150968]
2024-12-17 12:20:39 [2024/12/17 09:20:39.977 +00:00] [INFO] [delegator/distribution.go:305] ["Update readable segment version"] [oldVersion=1734427219977736171] [newVersion=1734427229979150968] [growingSegmentNum=0] [sealedSegmentNum=1]
2024-12-17 12:20:39 [2024/12/17 09:20:39.977 +00:00] [INFO] [observers/target_observer.go:531] ["observer trigger update current target"] [collectionID=454651567748415768]
2024-12-17 12:20:39 [2024/12/17 09:20:39.977 +00:00] [INFO] [datacoord/services.go:857] ["get recovery info request received"] [traceID=f33b577d1a63ec1a98ab8f3bd3c008ae] [collectionID=454651567748415768] [partitionIDs="[]"]
2024-12-17 12:20:39 [2024/12/17 09:20:39.977 +00:00] [INFO] [datacoord/handler.go:244] [GetQueryVChanPositions] [collectionID=454651567748415768] [channel=by-dev-rootcoord-dml_0_454651567748415768v0] [numOfSegments=11] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"="{\"454651567748415769\":0}"]
2024-12-17 12:20:39 [2024/12/17 09:20:39.977 +00:00] [INFO] [datacoord/handler.go:348] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_454651567748415768v0] [posTs=454669679176187907] [posTime=2024/12/17 09:19:41.916 +00:00]
2024-12-17 12:20:39 [2024/12/17 09:20:39.977 +00:00] [INFO] [datacoord/services.go:872] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=f33b577d1a63ec1a98ab8f3bd3c008ae] [collectionID=454651567748415768] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_454651567748415768v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=10] ["# of indexed segments"=0] ["# of l0 segments"=0]
2024-12-17 12:20:40 [2024/12/17 09:20:40.604 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:40 [2024/12/17 09:20:40.604 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:40 [2024/12/17 09:20:40.605 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:40 [2024/12/17 09:20:40.605 +00:00] [WARN] [retry/retry.go:130] ["retry func failed"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [retried=8] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"]
2024-12-17 12:20:42 [2024/12/17 09:20:42.387 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=454651567748415768] [replicaID=454651567907799041] [nodes="[\"{NodeID: 2, AssignedScore: 26150.000000, CurrentScore: 28765.000000, Priority: 2615}\"]"]
2024-12-17 12:20:43 [2024/12/17 09:20:43.676 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:43 [2024/12/17 09:20:43.677 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:43 [2024/12/17 09:20:43.677 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567750106802]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567750123608]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567750157216]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567749026040]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567750106800]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567750140412]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567748921126]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567750089996]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567750088021]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567750106804]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567750455507]
@Squire-tomsk Squire-tomsk added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 17, 2024
@chenyiminch
Copy link

The same problem. The bug does not occur on a small amount of data (e.g. thousand samples in database). But when the number of samples in the database is more than 100 thousand, the bug occurs.

@yanliang567
Copy link
Contributor

/assign @congqixia
/unassign

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 24, 2024
@yanliang567 yanliang567 added this to the 2.5.1 milestone Dec 24, 2024
@DotaArtist
Copy link

The same problem.

@yanliang567 yanliang567 modified the milestones: 2.5.1, 2.5.2 Dec 30, 2024
@ChiNoel-osu
Copy link

Can confirm it fails on large amount of data.

@yanliang567 yanliang567 modified the milestones: 2.5.2, 2.5.3 Jan 6, 2025
@so-hx
Copy link

so-hx commented Jan 6, 2025

Confirming this issue. Has anyone identified what version of Milvus introduced this issue?

@yanliang567
Copy link
Contributor

@congqixia is working on the fix.
BTW, if any one can reproduce this issue without any upsert or delete requested, please upload the milvus logs for investigation.

@r-4bb1t
Copy link

r-4bb1t commented Jan 7, 2025 •
edited
Loading 

milvus-standalone-test  | 2025-01-07T05:01:37.988443148Z [2025/01/07 05:01:37.988 +00:00] [INFO] [observers/target_observer.go:521] ["observer trigger update current target"] [collectionID=455141007491597098]
milvus-standalone-test  | 2025-01-07T05:01:37.988957234Z [2025/01/07 05:01:37.988 +00:00] [INFO] [datacoord/services.go:858] ["get recovery info request received"] [traceID=a44befe11447250fd93ae044bc4f32a6] [collectionID=455141007491597098] [partitionIDs="[]"]
milvus-standalone-test  | 2025-01-07T05:01:37.989030985Z [2025/01/07 05:01:37.988 +00:00] [INFO] [datacoord/handler.go:204] [GetQueryVChanPositions] [collectionID=455141007491597098] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] [numOfSegments=2] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
milvus-standalone-test  | 2025-01-07T05:01:37.989059728Z [2025/01/07 05:01:37.988 +00:00] [INFO] [datacoord/handler.go:436] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] [posTs=455141238399303681] [posTime=2025/01/07 05:00:37.595 +00:00]
milvus-standalone-test  | 2025-01-07T05:01:37.989080888Z [2025/01/07 05:01:37.989 +00:00] [INFO] [datacoord/services.go:873] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=a44befe11447250fd93ae044bc4f32a6] [collectionID=455141007491597098] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=1] ["# of indexed segments"=0] ["# of l0 segments"=0]
milvus-standalone-test  | 2025-01-07T05:01:39.996815178Z [2025/01/07 05:01:39.996 +00:00] [INFO] [balance/score_based_balancer.go:582] ["node channel workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 10.000000, CurrentScore: 10.000000, Priority: 0}\"]"]
milvus-standalone-test  | 2025-01-07T05:01:39.996866332Z [2025/01/07 05:01:39.996 +00:00] [INFO] [balance/score_based_balancer.go:514] ["node segment workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 4604.000000, CurrentScore: 5064.400000, Priority: 460}\"]"]
milvus-standalone-test  | 2025-01-07T05:01:42.997266634Z [2025/01/07 05:01:42.996 +00:00] [INFO] [balance/score_based_balancer.go:582] ["node channel workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 10.000000, CurrentScore: 10.000000, Priority: 0}\"]"]
milvus-standalone-test  | 2025-01-07T05:01:42.997312775Z [2025/01/07 05:01:42.997 +00:00] [INFO] [balance/score_based_balancer.go:514] ["node segment workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 4604.000000, CurrentScore: 5064.400000, Priority: 460}\"]"]
milvus-standalone-test  | 2025-01-07T05:01:45.996884215Z [2025/01/07 05:01:45.996 +00:00] [INFO] [balance/score_based_balancer.go:582] ["node channel workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 10.000000, CurrentScore: 10.000000, Priority: 0}\"]"]
milvus-standalone-test  | 2025-01-07T05:01:45.996934333Z [2025/01/07 05:01:45.996 +00:00] [INFO] [balance/score_based_balancer.go:514] ["node segment workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 4604.000000, CurrentScore: 5064.400000, Priority: 460}\"]"]
milvus-standalone-test  | 2025-01-07T05:01:47.982633514Z [2025/01/07 05:01:47.982 +00:00] [INFO] [datacoord/meta.go:1778] ["UpdateChannelCheckpoint done"] [traceID=55a34b784f4741510494f5884e7af07a] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] [ts=455141254180372482] [time=2025/01/07 05:01:37.795 +00:00]
milvus-standalone-test  | 2025-01-07T05:01:47.988729756Z [2025/01/07 05:01:47.988 +00:00] [INFO] [querynodev2/services.go:1295] ["sync action"] [traceID=dedc06d9baa72cb4d68ca43d06dd6e11] [collectionID=455141007491597098] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] [currentNodeID=1] [Action=UpdateVersion] [TargetVersion=1736226097989258591] [partitions="[455141007491597099]"]
milvus-standalone-test  | 2025-01-07T05:01:47.988763324Z [2025/01/07 05:01:47.988 +00:00] [INFO] [delegator/distribution.go:313] ["Update readable segment version"] [partitions="[455141007491597099]"] [oldVersion=1736226087989393804] [newVersion=1736226097989258591] [growingSegmentNum=0] [sealedSegmentNum=1]
milvus-standalone-test  | 2025-01-07T05:01:47.988935058Z [2025/01/07 05:01:47.988 +00:00] [INFO] [observers/target_observer.go:521] ["observer trigger update current target"] [collectionID=455141007491597098]
milvus-standalone-test  | 2025-01-07T05:01:47.989383049Z [2025/01/07 05:01:47.989 +00:00] [INFO] [datacoord/services.go:858] ["get recovery info request received"] [traceID=aa785da0939a6668b8f981c6a8eb5a13] [collectionID=455141007491597098] [partitionIDs="[]"]
milvus-standalone-test  | 2025-01-07T05:01:47.989418379Z [2025/01/07 05:01:47.989 +00:00] [INFO] [datacoord/handler.go:204] [GetQueryVChanPositions] [collectionID=455141007491597098] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] [numOfSegments=2] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
milvus-standalone-test  | 2025-01-07T05:01:47.989450049Z [2025/01/07 05:01:47.989 +00:00] [INFO] [datacoord/handler.go:436] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] [posTs=455141254180372482] [posTime=2025/01/07 05:01:37.795 +00:00]
milvus-standalone-test  | 2025-01-07T05:01:47.989460000Z [2025/01/07 05:01:47.989 +00:00] [INFO] [datacoord/services.go:873] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=aa785da0939a6668b8f981c6a8eb5a13] [collectionID=455141007491597098] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=1] ["# of indexed segments"=0] ["# of l0 segments"=0]
milvus-standalone-test  | 2025-01-07T05:01:48.397746707Z [2025/01/07 05:01:48.397 +00:00] [INFO] [proxy/impl.go:6272] ["connect received"] [traceID=05213682050527361da699affcf9d144] [sdk_type=Python] [sdk_version=2.5.3] [local_time="2025-01-07 05:01:48.396477"] [user=] [host=6685ed025bfc] [db=default]
milvus-standalone-test  | 2025-01-07T05:01:48.398336415Z [2025/01/07 05:01:48.398 +00:00] [INFO] [rootcoord/root_coord.go:977] ["received request to list databases"] [traceID=05213682050527361da699affcf9d144] [msgID=0]
milvus-standalone-test  | 2025-01-07T05:01:48.398367286Z [2025/01/07 05:01:48.398 +00:00] [INFO] [rootcoord/root_coord.go:1003] ["done to list databases"] [traceID=05213682050527361da699affcf9d144] [msgID=0] ["num of databases"=1]
milvus-standalone-test  | 2025-01-07T05:01:48.399477384Z [2025/01/07 05:01:48.399 +00:00] [INFO] [connection/manager.go:106] ["client register"] [traceID=05213682050527361da699affcf9d144] [sdk_type=Python] [sdk_version=2.5.3] [local_time="2025-01-07 05:01:48.396477"] [user=] [host=6685ed025bfc] [identifier=455141256959098883] [last_active_time=2025/01/07 05:01:48.399 +00:00]
milvus-standalone-test  | 2025-01-07T05:01:48.492245665Z [2025/01/07 05:01:48.491 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:48.492323666Z [2025/01/07 05:01:48.492 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:48.492633144Z [2025/01/07 05:01:48.492 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[*]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:48.492715439Z [2025/01/07 05:01:48.492 +00:00] [WARN] [retry/retry.go:130] ["retry func failed"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [retried=0] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"]
milvus-standalone-test  | 2025-01-07T05:01:48.704637927Z [2025/01/07 05:01:48.704 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:48.704697187Z [2025/01/07 05:01:48.704 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:48.704929071Z [2025/01/07 05:01:48.704 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[embedding,$meta,id,content]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:48.996934611Z [2025/01/07 05:01:48.996 +00:00] [INFO] [balance/score_based_balancer.go:582] ["node channel workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 10.000000, CurrentScore: 10.000000, Priority: 0}\"]"]
milvus-standalone-test  | 2025-01-07T05:01:48.996977149Z [2025/01/07 05:01:48.996 +00:00] [INFO] [balance/score_based_balancer.go:514] ["node segment workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 4604.000000, CurrentScore: 5064.400000, Priority: 460}\"]"]
milvus-standalone-test  | 2025-01-07T05:01:49.115916866Z [2025/01/07 05:01:49.115 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:49.116043391Z [2025/01/07 05:01:49.115 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:49.116279428Z [2025/01/07 05:01:49.115 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[content,embedding,$meta,id]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:49.926950279Z [2025/01/07 05:01:49.926 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:49.927075532Z [2025/01/07 05:01:49.926 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:49.927312678Z [2025/01/07 05:01:49.927 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[content,embedding,$meta,id]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:51.537067391Z [2025/01/07 05:01:51.536 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:51.537143073Z [2025/01/07 05:01:51.536 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:51.537277752Z [2025/01/07 05:01:51.536 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[content,embedding,$meta,id]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:51.537340840Z [2025/01/07 05:01:51.537 +00:00] [WARN] [retry/retry.go:130] ["retry func failed"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [retried=4] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"]
milvus-standalone-test  | 2025-01-07T05:01:51.997256570Z [2025/01/07 05:01:51.996 +00:00] [INFO] [balance/score_based_balancer.go:582] ["node channel workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 10.000000, CurrentScore: 10.000000, Priority: 0}\"]"]
milvus-standalone-test  | 2025-01-07T05:01:51.997297060Z [2025/01/07 05:01:51.997 +00:00] [INFO] [balance/score_based_balancer.go:514] ["node segment workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 4604.000000, CurrentScore: 5064.400000, Priority: 460}\"]"]
milvus-standalone-test  | 2025-01-07T05:01:54.547492706Z [2025/01/07 05:01:54.547 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:54.547603357Z [2025/01/07 05:01:54.547 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:54.547825559Z [2025/01/07 05:01:54.547 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[content,embedding,$meta,id]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:54.996882550Z [2025/01/07 05:01:54.996 +00:00] [INFO] [balance/score_based_balancer.go:582] ["node channel workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 10.000000, CurrentScore: 10.000000, Priority: 0}\"]"]
milvus-standalone-test  | 2025-01-07T05:01:54.996925399Z [2025/01/07 05:01:54.996 +00:00] [INFO] [balance/score_based_balancer.go:514] ["node segment workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 4604.000000, CurrentScore: 5064.400000, Priority: 460}\"]"]
milvus-standalone-test  | 2025-01-07T05:01:56.155621850Z [2025/01/07 05:01:56.155 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=265]
milvus-standalone-test  | 2025-01-07T05:01:57.437034175Z [2025/01/07 05:01:57.436 +00:00] [INFO] [datacoord/import_checker.go:137] ["import task stats"] [type=PreImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
milvus-standalone-test  | 2025-01-07T05:01:57.437091838Z [2025/01/07 05:01:57.436 +00:00] [INFO] [datacoord/import_checker.go:137] ["import task stats"] [type=ImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
milvus-standalone-test  | 2025-01-07T05:01:57.557730385Z [2025/01/07 05:01:57.557 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:57.557820435Z [2025/01/07 05:01:57.557 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:57.558041102Z [2025/01/07 05:01:57.557 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[content,embedding,$meta,id]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:01:57.988149118Z [2025/01/07 05:01:57.987 +00:00] [INFO] [querynodev2/services.go:1295] ["sync action"] [traceID=c8dbccd6656f6279f83d60e5143e0a8e] [collectionID=455141007491597098] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] [currentNodeID=1] [Action=UpdateVersion] [TargetVersion=1736226107989623489] [partitions="[455141007491597099]"]
milvus-standalone-test  | 2025-01-07T05:01:57.988185945Z [2025/01/07 05:01:57.988 +00:00] [INFO] [delegator/distribution.go:313] ["Update readable segment version"] [partitions="[455141007491597099]"] [oldVersion=1736226097989258591] [newVersion=1736226107989623489] [growingSegmentNum=0] [sealedSegmentNum=1]
milvus-standalone-test  | 2025-01-07T05:01:57.988387501Z [2025/01/07 05:01:57.988 +00:00] [INFO] [observers/target_observer.go:521] ["observer trigger update current target"] [collectionID=455141007491597098]
milvus-standalone-test  | 2025-01-07T05:01:57.988910967Z [2025/01/07 05:01:57.988 +00:00] [INFO] [datacoord/services.go:858] ["get recovery info request received"] [traceID=fa4b93f8128e09a0669b0338ac8fbb91] [collectionID=455141007491597098] [partitionIDs="[]"]
milvus-standalone-test  | 2025-01-07T05:01:57.988993066Z [2025/01/07 05:01:57.988 +00:00] [INFO] [datacoord/handler.go:204] [GetQueryVChanPositions] [collectionID=455141007491597098] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] [numOfSegments=2] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
milvus-standalone-test  | 2025-01-07T05:01:57.989025789Z [2025/01/07 05:01:57.988 +00:00] [INFO] [datacoord/handler.go:436] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] [posTs=455141254180372482] [posTime=2025/01/07 05:01:37.795 +00:00]
milvus-standalone-test  | 2025-01-07T05:01:57.989041114Z [2025/01/07 05:01:57.988 +00:00] [INFO] [datacoord/services.go:873] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=fa4b93f8128e09a0669b0338ac8fbb91] [collectionID=455141007491597098] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=1] ["# of indexed segments"=0] ["# of l0 segments"=0]
milvus-standalone-test  | 2025-01-07T05:01:57.996991349Z [2025/01/07 05:01:57.996 +00:00] [INFO] [balance/score_based_balancer.go:582] ["node channel workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 10.000000, CurrentScore: 10.000000, Priority: 0}\"]"]
milvus-standalone-test  | 2025-01-07T05:01:57.997021757Z [2025/01/07 05:01:57.996 +00:00] [INFO] [balance/score_based_balancer.go:514] ["node segment workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 4604.000000, CurrentScore: 5064.400000, Priority: 460}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:00.567697064Z [2025/01/07 05:02:00.567 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:00.567801461Z [2025/01/07 05:02:00.567 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:00.568013341Z [2025/01/07 05:02:00.567 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[id,content,embedding,$meta]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:00.996998978Z [2025/01/07 05:02:00.996 +00:00] [INFO] [balance/score_based_balancer.go:582] ["node channel workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 10.000000, CurrentScore: 10.000000, Priority: 0}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:00.997038540Z [2025/01/07 05:02:00.996 +00:00] [INFO] [balance/score_based_balancer.go:514] ["node segment workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 4604.000000, CurrentScore: 5064.400000, Priority: 460}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:03.577823818Z [2025/01/07 05:02:03.577 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:03.577886557Z [2025/01/07 05:02:03.577 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:03.578075685Z [2025/01/07 05:02:03.577 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[content,embedding,$meta,id]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:03.578127916Z [2025/01/07 05:02:03.578 +00:00] [WARN] [retry/retry.go:130] ["retry func failed"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [retried=8] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"]
milvus-standalone-test  | 2025-01-07T05:02:03.997316165Z [2025/01/07 05:02:03.997 +00:00] [INFO] [balance/score_based_balancer.go:582] ["node channel workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 10.000000, CurrentScore: 10.000000, Priority: 0}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:03.997358448Z [2025/01/07 05:02:03.997 +00:00] [INFO] [balance/score_based_balancer.go:514] ["node segment workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 4604.000000, CurrentScore: 5064.400000, Priority: 460}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:06.586871389Z [2025/01/07 05:02:06.586 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:06.586926084Z [2025/01/07 05:02:06.586 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:06.587015531Z [2025/01/07 05:02:06.586 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=8eedc64622edb7e3ba8af7b6c0b84f44] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[$meta,id,content,embedding]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:06.996966955Z [2025/01/07 05:02:06.996 +00:00] [INFO] [balance/score_based_balancer.go:582] ["node channel workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 10.000000, CurrentScore: 10.000000, Priority: 0}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:06.997063972Z [2025/01/07 05:02:06.996 +00:00] [INFO] [balance/score_based_balancer.go:514] ["node segment workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 4604.000000, CurrentScore: 5064.400000, Priority: 460}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:07.988565714Z [2025/01/07 05:02:07.988 +00:00] [INFO] [querynodev2/services.go:1295] ["sync action"] [traceID=7367bc9ba79f44b89307742b96ca7272] [collectionID=455141007491597098] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] [currentNodeID=1] [Action=UpdateVersion] [TargetVersion=1736226117989281548] [partitions="[455141007491597099]"]
milvus-standalone-test  | 2025-01-07T05:02:07.988621781Z [2025/01/07 05:02:07.988 +00:00] [INFO] [delegator/distribution.go:313] ["Update readable segment version"] [partitions="[455141007491597099]"] [oldVersion=1736226107989623489] [newVersion=1736226117989281548] [growingSegmentNum=0] [sealedSegmentNum=1]
milvus-standalone-test  | 2025-01-07T05:02:07.988846765Z [2025/01/07 05:02:07.988 +00:00] [INFO] [observers/target_observer.go:521] ["observer trigger update current target"] [collectionID=455141007491597098]
milvus-standalone-test  | 2025-01-07T05:02:07.989402187Z [2025/01/07 05:02:07.989 +00:00] [INFO] [datacoord/services.go:858] ["get recovery info request received"] [traceID=b2122451b988d4f6dcbad7674bd36199] [collectionID=455141007491597098] [partitionIDs="[]"]
milvus-standalone-test  | 2025-01-07T05:02:07.989471548Z [2025/01/07 05:02:07.989 +00:00] [INFO] [datacoord/handler.go:204] [GetQueryVChanPositions] [collectionID=455141007491597098] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] [numOfSegments=2] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
milvus-standalone-test  | 2025-01-07T05:02:07.989489358Z [2025/01/07 05:02:07.989 +00:00] [INFO] [datacoord/handler.go:436] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] [posTs=455141254180372482] [posTime=2025/01/07 05:01:37.795 +00:00]
milvus-standalone-test  | 2025-01-07T05:02:07.989520716Z [2025/01/07 05:02:07.989 +00:00] [INFO] [datacoord/services.go:873] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=b2122451b988d4f6dcbad7674bd36199] [collectionID=455141007491597098] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=1] ["# of indexed segments"=0] ["# of l0 segments"=0]
milvus-standalone-test  | 2025-01-07T05:02:09.679571276Z [2025/01/07 05:02:09.679 +00:00] [INFO] [proxy/impl.go:6272] ["connect received"] [traceID=f607f9f9e5c4c521d03ca3580889f7ff] [sdk_type=Python] [sdk_version=2.5.3] [local_time="2025-01-07 05:02:09.678667"] [user=] [host=6685ed025bfc] [db=default]
milvus-standalone-test  | 2025-01-07T05:02:09.680257529Z [2025/01/07 05:02:09.680 +00:00] [INFO] [rootcoord/root_coord.go:977] ["received request to list databases"] [traceID=f607f9f9e5c4c521d03ca3580889f7ff] [msgID=0]
milvus-standalone-test  | 2025-01-07T05:02:09.680288891Z [2025/01/07 05:02:09.680 +00:00] [INFO] [rootcoord/root_coord.go:1003] ["done to list databases"] [traceID=f607f9f9e5c4c521d03ca3580889f7ff] [msgID=0] ["num of databases"=1]
milvus-standalone-test  | 2025-01-07T05:02:09.681250983Z [2025/01/07 05:02:09.681 +00:00] [INFO] [connection/manager.go:106] ["client register"] [traceID=f607f9f9e5c4c521d03ca3580889f7ff] [sdk_type=Python] [sdk_version=2.5.3] [local_time="2025-01-07 05:02:09.678667"] [user=] [host=6685ed025bfc] [identifier=455141262529921026] [last_active_time=2025/01/07 05:02:09.681 +00:00]
milvus-standalone-test  | 2025-01-07T05:02:09.744617397Z [2025/01/07 05:02:09.744 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:09.744707604Z [2025/01/07 05:02:09.744 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=786fbb322b7aeeacaa913a28109c5f42] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:09.744900005Z [2025/01/07 05:02:09.744 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[*]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:09.744978492Z [2025/01/07 05:02:09.744 +00:00] [WARN] [retry/retry.go:130] ["retry func failed"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [retried=0] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"]
milvus-standalone-test  | 2025-01-07T05:02:09.954184410Z [2025/01/07 05:02:09.953 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:09.954237743Z [2025/01/07 05:02:09.953 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=786fbb322b7aeeacaa913a28109c5f42] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:09.954518476Z [2025/01/07 05:02:09.954 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[id,content,embedding,$meta]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:09.996991208Z [2025/01/07 05:02:09.996 +00:00] [INFO] [balance/score_based_balancer.go:582] ["node channel workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 10.000000, CurrentScore: 10.000000, Priority: 0}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:09.997027824Z [2025/01/07 05:02:09.996 +00:00] [INFO] [balance/score_based_balancer.go:514] ["node segment workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 4604.000000, CurrentScore: 5064.400000, Priority: 460}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:10.363592669Z [2025/01/07 05:02:10.363 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:10.363650936Z [2025/01/07 05:02:10.363 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=786fbb322b7aeeacaa913a28109c5f42] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:10.363888530Z [2025/01/07 05:02:10.363 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[$meta,id,content,embedding]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:11.171490746Z [2025/01/07 05:02:11.171 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:11.171566207Z [2025/01/07 05:02:11.171 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=786fbb322b7aeeacaa913a28109c5f42] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:11.171682000Z [2025/01/07 05:02:11.171 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[$meta,id,content,embedding]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:12.781547435Z [2025/01/07 05:02:12.781 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:12.781607047Z [2025/01/07 05:02:12.781 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=786fbb322b7aeeacaa913a28109c5f42] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:12.781810853Z [2025/01/07 05:02:12.781 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[id,content,embedding,$meta]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:12.781866865Z [2025/01/07 05:02:12.781 +00:00] [WARN] [retry/retry.go:130] ["retry func failed"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [retried=4] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"]
milvus-standalone-test  | 2025-01-07T05:02:12.997139269Z [2025/01/07 05:02:12.996 +00:00] [INFO] [balance/score_based_balancer.go:582] ["node channel workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 10.000000, CurrentScore: 10.000000, Priority: 0}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:12.997225932Z [2025/01/07 05:02:12.997 +00:00] [INFO] [balance/score_based_balancer.go:514] ["node segment workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 4604.000000, CurrentScore: 5064.400000, Priority: 460}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:15.791563931Z [2025/01/07 05:02:15.791 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:15.791710872Z [2025/01/07 05:02:15.791 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=786fbb322b7aeeacaa913a28109c5f42] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:15.791952917Z [2025/01/07 05:02:15.791 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[embedding,$meta,id,content]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:15.996490069Z [2025/01/07 05:02:15.996 +00:00] [INFO] [balance/score_based_balancer.go:582] ["node channel workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 10.000000, CurrentScore: 10.000000, Priority: 0}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:15.996529981Z [2025/01/07 05:02:15.996 +00:00] [INFO] [balance/score_based_balancer.go:514] ["node segment workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 4604.000000, CurrentScore: 5064.400000, Priority: 460}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:17.988828662Z [2025/01/07 05:02:17.988 +00:00] [INFO] [querynodev2/services.go:1295] ["sync action"] [traceID=ed7a8ada95e1b4dfe6b780933d90c0cf] [collectionID=455141007491597098] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] [currentNodeID=1] [Action=UpdateVersion] [TargetVersion=1736226127989720003] [partitions="[455141007491597099]"]
milvus-standalone-test  | 2025-01-07T05:02:17.988875093Z [2025/01/07 05:02:17.988 +00:00] [INFO] [delegator/distribution.go:313] ["Update readable segment version"] [partitions="[455141007491597099]"] [oldVersion=1736226117989281548] [newVersion=1736226127989720003] [growingSegmentNum=0] [sealedSegmentNum=1]
milvus-standalone-test  | 2025-01-07T05:02:17.989028283Z [2025/01/07 05:02:17.988 +00:00] [INFO] [observers/target_observer.go:521] ["observer trigger update current target"] [collectionID=455141007491597098]
milvus-standalone-test  | 2025-01-07T05:02:17.989486538Z [2025/01/07 05:02:17.989 +00:00] [INFO] [datacoord/services.go:858] ["get recovery info request received"] [traceID=1607939f581f0884d5c9f320eeac3011] [collectionID=455141007491597098] [partitionIDs="[]"]
milvus-standalone-test  | 2025-01-07T05:02:17.989562489Z [2025/01/07 05:02:17.989 +00:00] [INFO] [datacoord/handler.go:204] [GetQueryVChanPositions] [collectionID=455141007491597098] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] [numOfSegments=2] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
milvus-standalone-test  | 2025-01-07T05:02:17.989578049Z [2025/01/07 05:02:17.989 +00:00] [INFO] [datacoord/handler.go:436] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] [posTs=455141254180372482] [posTime=2025/01/07 05:01:37.795 +00:00]
milvus-standalone-test  | 2025-01-07T05:02:17.989617563Z [2025/01/07 05:02:17.989 +00:00] [INFO] [datacoord/services.go:873] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=1607939f581f0884d5c9f320eeac3011] [collectionID=455141007491597098] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_455141007491597098v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=1] ["# of indexed segments"=0] ["# of l0 segments"=0]
milvus-standalone-test  | 2025-01-07T05:02:18.801424611Z [2025/01/07 05:02:18.800 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:18.801538847Z [2025/01/07 05:02:18.801 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=786fbb322b7aeeacaa913a28109c5f42] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:18.801707542Z [2025/01/07 05:02:18.801 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[content,embedding,$meta,id]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:18.996767090Z [2025/01/07 05:02:18.996 +00:00] [INFO] [balance/score_based_balancer.go:582] ["node channel workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 10.000000, CurrentScore: 10.000000, Priority: 0}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:18.996800547Z [2025/01/07 05:02:18.996 +00:00] [INFO] [balance/score_based_balancer.go:514] ["node segment workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 4604.000000, CurrentScore: 5064.400000, Priority: 460}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:20.722060709Z [2025/01/07 05:02:20.721 +00:00] [INFO] [delegator/exclude_info.go:80] ["remove segment from exclude info"] [segmentID=455141007491797204]
milvus-standalone-test  | 2025-01-07T05:02:20.722100145Z [2025/01/07 05:02:20.721 +00:00] [INFO] [delegator/exclude_info.go:80] ["remove segment from exclude info"] [segmentID=455141007493000458]
milvus-standalone-test  | 2025-01-07T05:02:21.811570720Z [2025/01/07 05:02:21.811 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:21.811660396Z [2025/01/07 05:02:21.811 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=786fbb322b7aeeacaa913a28109c5f42] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:21.811813105Z [2025/01/07 05:02:21.811 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[id,content,embedding,$meta]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:21.997126973Z [2025/01/07 05:02:21.996 +00:00] [INFO] [balance/score_based_balancer.go:582] ["node channel workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 10.000000, CurrentScore: 10.000000, Priority: 0}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:21.997177727Z [2025/01/07 05:02:21.997 +00:00] [INFO] [balance/score_based_balancer.go:514] ["node segment workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 4604.000000, CurrentScore: 5064.400000, Priority: 460}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:24.820680135Z [2025/01/07 05:02:24.820 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:24.820847037Z [2025/01/07 05:02:24.820 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=786fbb322b7aeeacaa913a28109c5f42] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:24.821046349Z [2025/01/07 05:02:24.820 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[id,content,embedding,$meta]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:24.821115936Z [2025/01/07 05:02:24.821 +00:00] [WARN] [retry/retry.go:130] ["retry func failed"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [retried=8] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"]
milvus-standalone-test  | 2025-01-07T05:02:24.996680850Z [2025/01/07 05:02:24.996 +00:00] [INFO] [balance/score_based_balancer.go:582] ["node channel workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 10.000000, CurrentScore: 10.000000, Priority: 0}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:24.996721210Z [2025/01/07 05:02:24.996 +00:00] [INFO] [balance/score_based_balancer.go:514] ["node segment workload status"] [collectionID=455141007491597098] [replicaID=455141007810363396] [nodes="[\"{NodeID: 1, AssignedScore: 4604.000000, CurrentScore: 5064.400000, Priority: 460}\"]"]
milvus-standalone-test  | 2025-01-07T05:02:26.156235146Z [2025/01/07 05:02:26.156 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=265]
milvus-standalone-test  | 2025-01-07T05:02:27.836135450Z [2025/01/07 05:02:27.835 +00:00] [WARN] [proxy/task_search.go:748] ["failed to requery"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:27.836226203Z [2025/01/07 05:02:27.835 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=786fbb322b7aeeacaa913a28109c5f42] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
milvus-standalone-test  | 2025-01-07T05:02:27.836439073Z [2025/01/07 05:02:27.836 +00:00] [WARN] [proxy/impl.go:3173] ["Search failed to WaitToFinish"] [traceID=786fbb322b7aeeacaa913a28109c5f42] [role=proxy] [db=default] [collection=test] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=4108] [OutputFields="[id,content,embedding,$meta]"] [search_params="[{\"key\":\"topk\",\"value\":\"40\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"anns_field\",\"value\":\"embedding\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [nq=1] [error="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result"] [errorVerbose="incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:914\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:746\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id ff843c4d-c8ba-46d7-b054-6c15b12629b2, len(searchIDs) = 40, len(queryIDs) = 36, collection=455141007491597098\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]

I am encountering a similar issue. When I create a collection and run a query, this error occurs intermittently. To resolve it, I have to delete and recreate the collection.
Additionally, when checked with attu, there is a node with the ID ff843c4d-c8ba-46d7-b054-6c15b12629b2 in the collection.

@Andy6132024
Copy link

+1. The issue might be introduced in v2.5.0-beta because it started to happen after we upgraded from v2.4.15 to v2.5.0-beta. The issue still exists in v2.5.2.

This is a major blocker for us and we appreciate the team can expediate the fix.

Here are the logs from v2.5.2,

[2025/01/07 05:50:45.715 +00:00] [WARN] [proxy/task_search.go:767] ["failed to requery"] [traceID=a1c932b3a8a9b2ecb97a0273abaa93f3] [nq=1] [error="incomplete query result, missing id 9c929280-31b3-5398-9ca8-cad2673b52cc, len(searchIDs) = 15, len(queryIDs) = 6, collection=455050997327826341: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 9c929280-31b3-5398-9ca8-cad2673b52cc, len(searchIDs) = 15, len(queryIDs) = 6, collection=455050997327826341: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:941\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:765\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id 9c929280-31b3-5398-9ca8-cad2673b52cc, len(searchIDs) = 15, len(queryIDs) = 6, collection=455050997327826341\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]

@msci-diego-vg
Copy link

msci-diego-vg commented Jan 9, 2025 •
edited
Loading

Hi, I’m encountering the same issue with Milvus v2.5.2 and pymilvus v2.5.3.

Hybrid search works fine for smaller collections, but for larger ones, I get the same logs as mentioned above:

[2025/01/08 15:07:37.783 +00:00] [WARN] [proxy/impl.go:3389] ["HybridSearch failed to WaitToFinish"] [traceID=798cd4453bac1f73aa6dab3becfdc42d] [role=proxy] [db=default] [collection=abcd] [partitions="[]"] [OutputFields="[pk,document_id,issuer_id,version,issuer_name]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id abcd.1.94.3, len(searchIDs) = 5, len(queryIDs) = 0, collection=453940153033158858: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 25c1f6a6-b5a7-4cc4-848e-a7266bad1238.1.94.3, len(searchIDs) = 5, len(queryIDs) = 0, collection=453940153033158858: inconsistent requery result\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n | \t/workspace/source/pkg/util/merr/utils.go:1104\n | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n | \t/workspace/source/internal/proxy/task_search.go:941\n | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n | \t/workspace/source/internal/proxy/task_search.go:765\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n

When I run the hybrid_search() function with output_fields, it hangs, and the log above appears. However, if I remove the output_fields, the function executes as expected—though I need those output fields in my result.

Interestingly, running dense and sparse vector searches independently on the same index with the same query vectors works as expected.

Any insights or workarounds would be greatly appreciated.

@noob9527
Copy link

@congqixia is working on the fix. BTW, if any one can reproduce this issue without any upsert or delete requested, please upload the milvus logs for investigation.

I just encoutered this problem, in brand new collection, I havent call nether upsert nor delete.

milvus version: 2.5.2

single vector search works fine, hybrid search hangs, and reports
client log

io.milvus.v2.exception.MilvusClientException: incomplete query result, missing id 269794:C:251171:0, len(searchIDs) = 200, len(queryIDs) = 10, collection=455231893842317857: inconsistent requery result

server log

[2025/01/11 15:38:51.951 +00:00] [WARN] [proxy/task_search.go:767] ["failed to requery"] [traceID=3804244cf226de5158c8e360990ed66e] [nq=1] [error="incomplete query result, missing id 269794:C:251171:0, len(searchIDs) = 200, len(queryIDs) = 10, collection=455231893842317857: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 269794:C:251171:0, len(searchIDs) = 200, len(queryIDs) = 10, collection=455231893842317857: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:941\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:765\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id 269794:C:251171:0, len(searchIDs) = 200, len(queryIDs) = 10, collection=455231893842317857\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2025/01/11 15:38:51.951 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=3804244cf226de5158c8e360990ed66e] [error="incomplete query result, missing id 269794:C:251171:0, len(searchIDs) = 200, len(queryIDs) = 10, collection=455231893842317857: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 269794:C:251171:0, len(searchIDs) = 200, len(queryIDs) = 10, collection=455231893842317857: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:941\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:765\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id 269794:C:251171:0, len(searchIDs) = 200, len(queryIDs) = 10, collection=455231893842317857\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2025/01/11 15:38:51.951 +00:00] [WARN] [proxy/impl.go:3389] ["HybridSearch failed to WaitToFinish"] [traceID=3804244cf226de5158c8e360990ed66e] [role=proxy] [db=default] [collection=expert_11_23_v1] [partitions="[]"] [OutputFields="[consultant_id,text_chunk,text_id,text_type,text_timestamp,consultant_created_at,$meta]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 269794:C:251171:0, len(searchIDs) = 200, len(queryIDs) = 10, collection=455231893842317857: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 269794:C:251171:0, len(searchIDs) = 200, len(queryIDs) = 10, collection=455231893842317857: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:941\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:765\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) incomplete query result, missing id 269794:C:251171:0, len(searchIDs) = 200, len(queryIDs) = 10, collection=455231893842317857\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]

@noob9527
Copy link

FYI, I just downgraded milvus to 2.4.5, everything works fine

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@congqixia congqixia

Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.3
Development

No branches or pull requests