Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: there is no IndexNode online #39167

Open
1 task done
lhj1304 opened this issue Jan 11, 2025 · 1 comment
Open
1 task done

[Bug]: there is no IndexNode online #39167

lhj1304 opened this issue Jan 11, 2025 · 1 comment
Labels
kind/bug Issues or changes related a bug triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@lhj1304
Copy link

lhj1304 commented Jan 11, 2025

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:milvusdb/milvus:v2.4.3
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):pymilvus:2.4.8
- OS(Ubuntu or CentOS): ubuntu
- CPU/Memory: 4/32Gb
- GPU: 
- Others:

Current Behavior

[2025/01/10 10:13:32.790 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=455160513247170569]
[2025/01/10 10:13:32.824 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:32.825 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:32.888 +00:00] [INFO] [datacoord/index_meta.go:697] ["finish index task success"] [buildID=455160513247219532] [state=Finished] ["fail reason"=] [current_index_version=0]
[2025/01/10 10:13:32.889 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=2]
[2025/01/10 10:13:32.889 +00:00] [INFO] [datacoord/index_builder.go:535] ["IndexNode no longer exist, no need to drop index task"] [buildID=455160513247219532] [nodeID=0]
[2025/01/10 10:13:32.889 +00:00] [INFO] [datacoord/index_builder.go:260] ["segment does not need index really"] [buildID=455160513247219539] [segmentID=455160513246492873] ["num rows"=3]
[2025/01/10 10:13:32.890 +00:00] [INFO] [observers/replica_observer.go:134] ["all segment/channel has been removed from ro node, try to remove it from replica"] [collectionID=454896180733935644] [replicaID=454896180866056193] [removedNodes="[11]"] [roNodes="[11]"] [rwNodes="[]"]
[2025/01/10 10:13:32.890 +00:00] [INFO] [datacoord/index_meta.go:697] ["finish index task success"] [buildID=455160513247219539] [state=Finished] ["fail reason"=] [current_index_version=0]
[2025/01/10 10:13:32.890 +00:00] [INFO] [datacoord/index_meta.go:314] ["meta update: adding segment index"] [collectionID=455160513246232876] [segmentID=455160513246492910] [indexID=455160513246232884] [buildID=455160513247219540]
[2025/01/10 10:13:32.891 +00:00] [INFO] [datacoord/index_meta.go:326] ["meta update: adding segment index success"] [collectionID=455160513246232876] [segmentID=455160513246492910] [indexID=455160513246232884] [buildID=455160513247219540]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:160] ["indexBuilder enqueue task"] [buildID=455160513247219540]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_service.go:58] ["create index for segment"] [segmentID=455160513246492910] [indexID=455160513246232890]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=2]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:535] ["IndexNode no longer exist, no need to drop index task"] [buildID=455160513247219539] [nodeID=0]
[2025/01/10 10:13:32.892 +00:00] [ERROR] [datacoord/indexnode_manager.go:104] ["there is no IndexNode online"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PeekClient\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/indexnode_manager.go:104\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).process\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:277\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).run\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:200\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).schedule\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:177"]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:279] ["index builder peek client error, there is no available"]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:202] ["there is no idle indexing node, wait a minute..."]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_meta.go:314] ["meta update: adding segment index"] [collectionID=455160513246232876] [segmentID=455160513246492910] [indexID=455160513246232890] [buildID=455160513247219542]
[2025/01/10 10:13:32.893 +00:00] [INFO] [datacoord/index_meta.go:326] ["meta update: adding segment index success"] [collectionID=455160513246232876] [segmentID=455160513246492910] [indexID=455160513246232890] [buildID=455160513247219542]
[2025/01/10 10:13:32.894 +00:00] [INFO] [datacoord/index_builder.go:160] ["indexBuilder enqueue task"] [buildID=455160513247219542]
[2025/01/10 10:13:32.894 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=2]
[2025/01/10 10:13:32.894 +00:00] [ERROR] [datacoord/indexnode_manager.go:104] ["there is no IndexNode online"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PeekClient\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/indexnode_manager.go:104\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).process\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:277\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).run\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:200\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).schedule\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:177"]
[2025/01/10 10:13:32.894 +00:00] [INFO] [datacoord/index_builder.go:279] ["index builder peek client error, there is no available"]
[2025/01/10 10:13:32.894 +00:00] [INFO] [datacoord/index_builder.go:202] ["there is no idle indexing node, wait a minute..."]
[2025/01/10 10:13:33.032 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.032 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.209 +00:00] [WARN] [datacoord/services.go:1474] ["node is not matched with channel"] [traceID=36930fd98daa53daa017c1974290e865] [channel=by-dev-rootcoord-dml_0_454896180733935644v0] [sourceID=11] [ts=455214108011659266]
[2025/01/10 10:13:33.209 +00:00] [WARN] [datacoord/services.go:1474] ["node is not matched with channel"] [traceID=36930fd98daa53daa017c1974290e865] [channel=by-dev-rootcoord-dml_1_455160513246232876v0] [sourceID=11] [ts=455214108011659266]
[2025/01/10 10:13:33.222 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.223 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.340 +00:00] [INFO] [datacoord/channel_manager_v2.go:515] ["Reassign standby channels to node"] [nodeID=-9223372036854775808] [channels="[by-dev-rootcoord-dml_1_455160513246232876v0,by-dev-rootcoord-dml_0_454896180733935644v0]"]
[2025/01/10 10:13:33.350 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=2]
[2025/01/10 10:13:33.350 +00:00] [ERROR] [datacoord/indexnode_manager.go:104] ["there is no IndexNode online"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PeekClient\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/indexnode_manager.go:104\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).process\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:277\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).run\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:200\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).schedule\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:181"]
[2025/01/10 10:13:33.350 +00:00] [INFO] [datacoord/index_builder.go:279] ["index builder peek client error, there is no available"]
[2025/01/10 10:13:33.350 +00:00] [INFO] [datacoord/index_builder.go:202] ["there is no idle indexing node, wait a minute..."]
[2025/01/10 10:13:33.422 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.422 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.622 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.622 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.708 +00:00] [WARN] [datacoord/services.go:1474] ["node is not matched with channel"] [traceID=f9454ce6542e3cd2682c521eee95dc4a] [channel=by-dev-rootcoord-dml_0_454896180733935644v0] [sourceID=11] [ts=455214108116516866]
[2025/01/10 10:13:33.708 +00:00] [WARN] [datacoord/services.go:1474] ["node is not matched with channel"] [traceID=f9454ce6542e3cd2682c521eee95dc4a] [channel=by-dev-rootcoord-dml_1_455160513246232876v0] [sourceID=11] [ts=455214108116516866]
[2025/01/10 10:13:33.764 +00:00] [WARN] [sessionutil/session_util.go:553] ["fail to retry keepAliveOnce"] [serverName=querycoord] [LeaseID=7587883964900698493] [error="etcdserver: requested lease not found"]
[2025/01/10 10:13:33.764 +00:00] [WARN] [sessionutil/session_util.go:882] ["connection lost detected, shuting down"]
[2025/01/10 10:13:33.764 +00:00] [ERROR] [querycoordv2/server.go:155] ["QueryCoord disconnected from etcd, process will exit"] [serverID=11] [stack="github.com/milvus-io/milvus/internal/querycoordv2.(*Server).Register.func1.1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/server.go:155"]
(base) ubuntu@VM-0-2-ubuntu:~$ [2025/01/10 10:13:32.824 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:32.825 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:32.888 +00:00] [INFO] [datacoord/index_meta.go:697] ["finish index task success"] [buildID=455160513247219532] [state=Finished] ["fail reason"=] [current_index_version=0]
[2025/01/10 10:13:32.889 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=2]
[2025/01/10 10:13:32.889 +00:00] [INFO] [datacoord/index_builder.go:535] ["IndexNode no longer exist, no need to drop index task"] [buildID=455160513247219532] [nodeID=0]
[2025/01/10 10:13:32.889 +00:00] [INFO] [datacoord/index_builder.go:260] ["segment does not need index really"] [buildID=455160513247219539] [segmentID=455160513246492873] ["num rows"=3]
[2025/01/10 10:13:32.890 +00:00] [INFO] [observers/replica_observer.go:134] ["all segment/channel has been removed from ro node, try to remove it from replica"] [collectionID=454896180733935644] [replicaID=454896180866056193] [removedNodes="[11]"] [roNodes="[11]"] [rwNodes="[]"]
[2025/01/10 10:13:32.890 +00:00] [INFO] [datacoord/index_meta.go:697] ["finish index task success"] [buildID=455160513247219539] [state=Finished] ["fail reason"=] [current_index_version=0]
[2025/01/10 10:13:32.890 +00:00] [INFO] [datacoord/index_meta.go:314] ["meta update: adding segment index"] [collectionID=455160513246232876] [segmentID=455160513246492910] [indexID=455160513246232884] [buildID=455160513247219540]
[2025/01/10 10:13:32.891 +00:00] [INFO] [datacoord/index_meta.go:326] ["meta update: adding segment index success"] [collectionID=455160513246232876] [segmentID=455160513246492910] [indexID=455160513246232884] [buildID=455160513247219540]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:160] ["indexBuilder enqueue task"] [buildID=455160513247219540]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_service.go:58] ["create index for segment"] [segmentID=455160513246492910] [indexID=455160513246232890]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=2]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:535] ["IndexNode no longer exist, no need to drop index task"] [buildID=455160513247219539] [nodeID=0]
[2025/01/10 10:13:32.892 +00:00] [ERROR] [datacoord/indexnode_manager.go:104] ["there is no IndexNode online"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PeekClient\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/indexnode_manager.go:104\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).process\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:277\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).run\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:200\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).schedule\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:177"]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:279] ["index builder peek client error, there is no available"]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:202] ["there is no idle indexing node, wait a minute..."]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_meta.go:314] ["meta update: adding segment index"] [collectionID=455160513246232876] [segmentID=455160513246492910] [indexID=455160513246232890] [buildID=455160513247219542]
[2025/01/10 10:13:32.893 +00:00] [INFO] [datacoord/index_meta.go:326] ["meta update: adding segment index success"] [collectionID=455160513246232876] [segmentID=455160513246492910] [indexID=455160513246232890] [buildID=455160513247219542]
[2025/01/10 10:13:32.894 +00:00] [INFO] [datacoord/index_builder.go:160] ["indexBuilder enqueue task"] [buildID=455160513247219542]
[2025/01/10 10:13:32.894 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=2]
[2025/01/10 10:13:32.894 +00:00] [ERROR] [datacoord/indexnode_manager.go:104] ["there is no IndexNode online"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PeekClient\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/indexnode_manager.go:104\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).process\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:277\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).run\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:200\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).schedule\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:177"]
[2025/01/10 10:13:32.894 +00:00] [INFO] [datacoord/index_builder.go:279] ["index builder peek client error, there is no available"]
[2025/01/10 10:13:32.894 +00:00] [INFO] [datacoord/index_builder.go:202] ["there is no idle indexing node, wait a minute..."]
[2025/01/10 10:13:33.032 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.032 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.209 +00:00] [WARN] [datacoord/services.go:1474] ["node is not matched with channel"] [traceID=36930fd98daa53daa017c1974290e865] [channel=by-dev-rootcoord-dml_0_454896180733935644v0] [sourceID=11] [ts=455214108011659266]
[2025/01/10 10:13:33.209 +00:00] [WARN] [datacoord/services.go:1474] ["node is not matched with channel"] [traceID=36930fd98daa53daa017c1974290e865] [channel=by-dev-rootcoord-dml_1_455160513246232876v0] [sourceID=11] [ts=455214108011659266]
[2025/01/10 10:13:33.222 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.223 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.340 +00:00] [INFO] [datacoord/channel_manager_v2.go:515] ["Reassign standby channels to node"] [nodeID=-9223372036854775808] [channels="[by-dev-rootcoord-dml_1_455160513246232876v0,by-dev-rootcoord-dml_0_454896180733935644v0]"]
[2025/01/10 10:13:33.350 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=2]
[2025/01/10 10:13:33.350 +00:00] [ERROR] [datacoord/indexnode_manager.go:104] ["there is no IndexNode online"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PeekClient\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/indexnode_manager.go:104\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).process\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:277\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).run\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:200\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).schedule\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:181"]
[2025/01/10 10:13:33.350 +00:00] [INFO] [datacoord/index_builder.go:279] ["index builder peek client error, there is no available"]
[2025/01/10 10:13:33.350 +00:00] [INFO] [datacoord/index_builder.go:202] ["there is no idle indexing node, wait a minute..."]
[2025/01/10 10:13:33.422 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.422 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.622 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.622 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]

Expected Behavior

No response

Steps To Reproduce

# milvus
  milvusEtcd:
    container_name: milvusEtcd
    image: quay.io/coreos/etcd:v3.5.5
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    networks:
      - fastgpt
    volumes:
      - ~/hdwb/FastGpt/milvus/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: [ 'CMD', 'etcdctl', 'endpoint', 'health' ]
      interval: 30s
      timeout: 20s
      retries: 3
  milvusStandalone:
    container_name: milvusStandalone
    image: milvusdb/milvus:v2.4.3
    command: [ 'milvus', 'run', 'standalone' ]
    security_opt:
      - seccomp:unconfined
    environment:
      ETCD_ENDPOINTS: milvusEtcd:2379
      MINIO_ADDRESS: minio:9000
    networks:
      - fastgpt
    volumes:
      - ~/hdwb/FastGpt/milvus/data:/var/lib/milvus
    healthcheck:
      test: [ 'CMD', 'curl', '-f', 'http://localhost:9091/healthz' ]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - 'milvusEtcd'
      - 'minio'

Milvus Log

[2025/01/10 10:13:32.790 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=455160513247170569]
[2025/01/10 10:13:32.824 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:32.825 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:32.888 +00:00] [INFO] [datacoord/index_meta.go:697] ["finish index task success"] [buildID=455160513247219532] [state=Finished] ["fail reason"=] [current_index_version=0]
[2025/01/10 10:13:32.889 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=2]
[2025/01/10 10:13:32.889 +00:00] [INFO] [datacoord/index_builder.go:535] ["IndexNode no longer exist, no need to drop index task"] [buildID=455160513247219532] [nodeID=0]
[2025/01/10 10:13:32.889 +00:00] [INFO] [datacoord/index_builder.go:260] ["segment does not need index really"] [buildID=455160513247219539] [segmentID=455160513246492873] ["num rows"=3]
[2025/01/10 10:13:32.890 +00:00] [INFO] [observers/replica_observer.go:134] ["all segment/channel has been removed from ro node, try to remove it from replica"] [collectionID=454896180733935644] [replicaID=454896180866056193] [removedNodes="[11]"] [roNodes="[11]"] [rwNodes="[]"]
[2025/01/10 10:13:32.890 +00:00] [INFO] [datacoord/index_meta.go:697] ["finish index task success"] [buildID=455160513247219539] [state=Finished] ["fail reason"=] [current_index_version=0]
[2025/01/10 10:13:32.890 +00:00] [INFO] [datacoord/index_meta.go:314] ["meta update: adding segment index"] [collectionID=455160513246232876] [segmentID=455160513246492910] [indexID=455160513246232884] [buildID=455160513247219540]
[2025/01/10 10:13:32.891 +00:00] [INFO] [datacoord/index_meta.go:326] ["meta update: adding segment index success"] [collectionID=455160513246232876] [segmentID=455160513246492910] [indexID=455160513246232884] [buildID=455160513247219540]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:160] ["indexBuilder enqueue task"] [buildID=455160513247219540]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_service.go:58] ["create index for segment"] [segmentID=455160513246492910] [indexID=455160513246232890]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=2]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:535] ["IndexNode no longer exist, no need to drop index task"] [buildID=455160513247219539] [nodeID=0]
[2025/01/10 10:13:32.892 +00:00] [ERROR] [datacoord/indexnode_manager.go:104] ["there is no IndexNode online"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PeekClient\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/indexnode_manager.go:104\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).process\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:277\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).run\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:200\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).schedule\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:177"]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:279] ["index builder peek client error, there is no available"]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:202] ["there is no idle indexing node, wait a minute..."]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_meta.go:314] ["meta update: adding segment index"] [collectionID=455160513246232876] [segmentID=455160513246492910] [indexID=455160513246232890] [buildID=455160513247219542]
[2025/01/10 10:13:32.893 +00:00] [INFO] [datacoord/index_meta.go:326] ["meta update: adding segment index success"] [collectionID=455160513246232876] [segmentID=455160513246492910] [indexID=455160513246232890] [buildID=455160513247219542]
[2025/01/10 10:13:32.894 +00:00] [INFO] [datacoord/index_builder.go:160] ["indexBuilder enqueue task"] [buildID=455160513247219542]
[2025/01/10 10:13:32.894 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=2]
[2025/01/10 10:13:32.894 +00:00] [ERROR] [datacoord/indexnode_manager.go:104] ["there is no IndexNode online"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PeekClient\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/indexnode_manager.go:104\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).process\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:277\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).run\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:200\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).schedule\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:177"]
[2025/01/10 10:13:32.894 +00:00] [INFO] [datacoord/index_builder.go:279] ["index builder peek client error, there is no available"]
[2025/01/10 10:13:32.894 +00:00] [INFO] [datacoord/index_builder.go:202] ["there is no idle indexing node, wait a minute..."]
[2025/01/10 10:13:33.032 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.032 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.209 +00:00] [WARN] [datacoord/services.go:1474] ["node is not matched with channel"] [traceID=36930fd98daa53daa017c1974290e865] [channel=by-dev-rootcoord-dml_0_454896180733935644v0] [sourceID=11] [ts=455214108011659266]
[2025/01/10 10:13:33.209 +00:00] [WARN] [datacoord/services.go:1474] ["node is not matched with channel"] [traceID=36930fd98daa53daa017c1974290e865] [channel=by-dev-rootcoord-dml_1_455160513246232876v0] [sourceID=11] [ts=455214108011659266]
[2025/01/10 10:13:33.222 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.223 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.340 +00:00] [INFO] [datacoord/channel_manager_v2.go:515] ["Reassign standby channels to node"] [nodeID=-9223372036854775808] [channels="[by-dev-rootcoord-dml_1_455160513246232876v0,by-dev-rootcoord-dml_0_454896180733935644v0]"]
[2025/01/10 10:13:33.350 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=2]
[2025/01/10 10:13:33.350 +00:00] [ERROR] [datacoord/indexnode_manager.go:104] ["there is no IndexNode online"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PeekClient\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/indexnode_manager.go:104\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).process\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:277\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).run\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:200\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).schedule\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:181"]
[2025/01/10 10:13:33.350 +00:00] [INFO] [datacoord/index_builder.go:279] ["index builder peek client error, there is no available"]
[2025/01/10 10:13:33.350 +00:00] [INFO] [datacoord/index_builder.go:202] ["there is no idle indexing node, wait a minute..."]
[2025/01/10 10:13:33.422 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.422 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.622 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.622 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.708 +00:00] [WARN] [datacoord/services.go:1474] ["node is not matched with channel"] [traceID=f9454ce6542e3cd2682c521eee95dc4a] [channel=by-dev-rootcoord-dml_0_454896180733935644v0] [sourceID=11] [ts=455214108116516866]
[2025/01/10 10:13:33.708 +00:00] [WARN] [datacoord/services.go:1474] ["node is not matched with channel"] [traceID=f9454ce6542e3cd2682c521eee95dc4a] [channel=by-dev-rootcoord-dml_1_455160513246232876v0] [sourceID=11] [ts=455214108116516866]
[2025/01/10 10:13:33.764 +00:00] [WARN] [sessionutil/session_util.go:553] ["fail to retry keepAliveOnce"] [serverName=querycoord] [LeaseID=7587883964900698493] [error="etcdserver: requested lease not found"]
[2025/01/10 10:13:33.764 +00:00] [WARN] [sessionutil/session_util.go:882] ["connection lost detected, shuting down"]
[2025/01/10 10:13:33.764 +00:00] [ERROR] [querycoordv2/server.go:155] ["QueryCoord disconnected from etcd, process will exit"] [serverID=11] [stack="github.com/milvus-io/milvus/internal/querycoordv2.(*Server).Register.func1.1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/server.go:155"]
(base) ubuntu@VM-0-2-ubuntu:~$ [2025/01/10 10:13:32.824 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:32.825 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:32.888 +00:00] [INFO] [datacoord/index_meta.go:697] ["finish index task success"] [buildID=455160513247219532] [state=Finished] ["fail reason"=] [current_index_version=0]
[2025/01/10 10:13:32.889 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=2]
[2025/01/10 10:13:32.889 +00:00] [INFO] [datacoord/index_builder.go:535] ["IndexNode no longer exist, no need to drop index task"] [buildID=455160513247219532] [nodeID=0]
[2025/01/10 10:13:32.889 +00:00] [INFO] [datacoord/index_builder.go:260] ["segment does not need index really"] [buildID=455160513247219539] [segmentID=455160513246492873] ["num rows"=3]
[2025/01/10 10:13:32.890 +00:00] [INFO] [observers/replica_observer.go:134] ["all segment/channel has been removed from ro node, try to remove it from replica"] [collectionID=454896180733935644] [replicaID=454896180866056193] [removedNodes="[11]"] [roNodes="[11]"] [rwNodes="[]"]
[2025/01/10 10:13:32.890 +00:00] [INFO] [datacoord/index_meta.go:697] ["finish index task success"] [buildID=455160513247219539] [state=Finished] ["fail reason"=] [current_index_version=0]
[2025/01/10 10:13:32.890 +00:00] [INFO] [datacoord/index_meta.go:314] ["meta update: adding segment index"] [collectionID=455160513246232876] [segmentID=455160513246492910] [indexID=455160513246232884] [buildID=455160513247219540]
[2025/01/10 10:13:32.891 +00:00] [INFO] [datacoord/index_meta.go:326] ["meta update: adding segment index success"] [collectionID=455160513246232876] [segmentID=455160513246492910] [indexID=455160513246232884] [buildID=455160513247219540]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:160] ["indexBuilder enqueue task"] [buildID=455160513247219540]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_service.go:58] ["create index for segment"] [segmentID=455160513246492910] [indexID=455160513246232890]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=2]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:535] ["IndexNode no longer exist, no need to drop index task"] [buildID=455160513247219539] [nodeID=0]
[2025/01/10 10:13:32.892 +00:00] [ERROR] [datacoord/indexnode_manager.go:104] ["there is no IndexNode online"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PeekClient\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/indexnode_manager.go:104\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).process\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:277\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).run\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:200\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).schedule\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:177"]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:279] ["index builder peek client error, there is no available"]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_builder.go:202] ["there is no idle indexing node, wait a minute..."]
[2025/01/10 10:13:32.892 +00:00] [INFO] [datacoord/index_meta.go:314] ["meta update: adding segment index"] [collectionID=455160513246232876] [segmentID=455160513246492910] [indexID=455160513246232890] [buildID=455160513247219542]
[2025/01/10 10:13:32.893 +00:00] [INFO] [datacoord/index_meta.go:326] ["meta update: adding segment index success"] [collectionID=455160513246232876] [segmentID=455160513246492910] [indexID=455160513246232890] [buildID=455160513247219542]
[2025/01/10 10:13:32.894 +00:00] [INFO] [datacoord/index_builder.go:160] ["indexBuilder enqueue task"] [buildID=455160513247219542]
[2025/01/10 10:13:32.894 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=2]
[2025/01/10 10:13:32.894 +00:00] [ERROR] [datacoord/indexnode_manager.go:104] ["there is no IndexNode online"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PeekClient\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/indexnode_manager.go:104\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).process\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:277\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).run\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:200\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).schedule\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:177"]
[2025/01/10 10:13:32.894 +00:00] [INFO] [datacoord/index_builder.go:279] ["index builder peek client error, there is no available"]
[2025/01/10 10:13:32.894 +00:00] [INFO] [datacoord/index_builder.go:202] ["there is no idle indexing node, wait a minute..."]
[2025/01/10 10:13:33.032 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.032 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.209 +00:00] [WARN] [datacoord/services.go:1474] ["node is not matched with channel"] [traceID=36930fd98daa53daa017c1974290e865] [channel=by-dev-rootcoord-dml_0_454896180733935644v0] [sourceID=11] [ts=455214108011659266]
[2025/01/10 10:13:33.209 +00:00] [WARN] [datacoord/services.go:1474] ["node is not matched with channel"] [traceID=36930fd98daa53daa017c1974290e865] [channel=by-dev-rootcoord-dml_1_455160513246232876v0] [sourceID=11] [ts=455214108011659266]
[2025/01/10 10:13:33.222 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.223 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.340 +00:00] [INFO] [datacoord/channel_manager_v2.go:515] ["Reassign standby channels to node"] [nodeID=-9223372036854775808] [channels="[by-dev-rootcoord-dml_1_455160513246232876v0,by-dev-rootcoord-dml_0_454896180733935644v0]"]
[2025/01/10 10:13:33.350 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=2]
[2025/01/10 10:13:33.350 +00:00] [ERROR] [datacoord/indexnode_manager.go:104] ["there is no IndexNode online"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PeekClient\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/indexnode_manager.go:104\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).process\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:277\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).run\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:200\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexBuilder).schedule\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/index_builder.go:181"]
[2025/01/10 10:13:33.350 +00:00] [INFO] [datacoord/index_builder.go:279] ["index builder peek client error, there is no available"]
[2025/01/10 10:13:33.350 +00:00] [INFO] [datacoord/index_builder.go:202] ["there is no idle indexing node, wait a minute..."]
[2025/01/10 10:13:33.422 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.422 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.622 +00:00] [WARN] [rootcoord/root_coord.go:1641] ["failed to updateTimeTick"] [role=rootcoord] [error="skip ChannelTimeTickMsg from un-recognized session 11"]
[2025/01/10 10:13:33.622 +00:00] [WARN] [proxy/proxy.go:378] [sendChannelsTimeTickLoop.UpdateChannelTimeTick] [ErrorCode=UnexpectedError] [Reason="skip ChannelTimeTickMsg from un-recognized session 11"]

Anything else?

No response
@lhj1304 lhj1304 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 11, 2025
@yanliang567
Copy link
Contributor

/assifn @lhj1304
please attach the milvus logs for investigation. One quick question is what did you you before the issue pops?
/unassign

@yanliang567 yanliang567 added triage/needs-information Indicates an issue needs more information in order to work on it. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Issues or changes related a bug triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lhj1304 @yanliang567