Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: milvus 2.1.1-1 after many search requests e got error IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"] after this cluster doesn't work. #34514

Open
1 task done
martinenkoEduard opened this issue Jul 9, 2024 · 3 comments
Open
1 task done

[Bug]: milvus 2.1.1-1 after many search requests e got error IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"] after this cluster doesn't work. #34514

martinenkoEduard opened this issue Jul 9, 2024 · 3 comments
Assignees
@martinenkoEduard
Labels
help wanted Extra attention is needed

Comments

@martinenkoEduard
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: milvus 2.1.1-1
- Deployment mode(standalone or cluster): standalone. deb pakages.
- MQ type(rocksmq, pulsar or kafka):    rocksmq
- SDK version(e.g. pymilvus v2.0.0rc2): 2.1.1-1
- OS(Ubuntu or CentOS): ubuntu 22
- CPU/Memory: 64gb
- GPU: none
- Others:

Current Behavior

I run a lot of searches and got

Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.706 +00:00] [DEBUG] [indexcoord/index_coord.go:344] ["get IndexCoord component states ..."]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.523 +00:00] [WARN] [rootcoord/timeticksync.go:330] ["SendTimeTickToChannel fail"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.523 +00:00] [ERROR] [rootcoord/dml_channels.go:115] ["Broadcast failed"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"] [chanName=by-dev-rootcoord-dml_0] [stack="github.com/milvus-io/milvus/internal/rootcoord.(*dmlChannels).broadcast\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/dml_channels.go:115\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).sendTimeTickToChannel\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:368\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).startWatch.func1\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:329"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.523 +00:00] [ERROR] [rootcoord/dml_channels.go:115] ["Broadcast failed"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"] [chanName=by-dev-rootcoord-dml_1] [stack="github.com/milvus-io/milvus/internal/rootcoord.(*dmlChannels).broadcast\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/dml_channels.go:115\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).sendTimeTickToChannel\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:368\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).startWatch.func1\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:329"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.323 +00:00] [WARN] [rootcoord/timeticksync.go:330] ["SendTimeTickToChannel fail"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.323 +00:00] [ERROR] [rootcoord/dml_channels.go:115] ["Broadcast failed"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"] [chanName=by-dev-rootcoord-dml_1] [stack="github.com/milvus-io/milvus/internal/rootcoord.(*dmlChannels).broadcast\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/dml_channels.go:115\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).sendTimeTickToChannel\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:368\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).startWatch.func1\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:329"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.323 +00:00] [WARN] [rootcoord/timeticksync.go:330] ["SendTimeTickToChannel fail"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.323 +00:00] [ERROR] [rootcoord/dml_channels.go:115] ["Broadcast failed"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"] [chanName=by-dev-rootcoord-dml_0] [stack="github.com/milvus-io/milvus/internal/rootcoord.(*dmlChannels).broadcast\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/dml_channels.go:115\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).sendTimeTickToChannel\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:368\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).startWatch.func1\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:329"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.123 +00:00] [WARN] [rootcoord/timeticksync.go:330] ["SendTimeTickToChannel fail"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.123 +00:00] [ERROR] [rootcoord/dml_channels.go:115] ["Broadcast failed"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"] [chanName=by-dev-rootcoord-dml_0] [stack="github.com/milvus-io/milvus/internal/rootcoord.(*dmlChannels).broadcast\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/dml_channels.go:115\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).sendTimeTickToChannel\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:368\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).startWatch.func1\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:329"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.123 +00:00] [WARN] [rootcoord/timeticksync.go:330] ["SendTimeTickToChannel fail"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.123 +00:00] [ERROR] [rootcoord/dml_channels.go:115] ["Broadcast failed"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"] [chanName=by-dev-rootcoord-dml_1] [stack="github.com/milvus-io/milvus/internal/rootcoord.(*dmlChannels).broadcast\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/dml_channels.go:115\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).sendTimeTickToChannel\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:368\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).startWatch.func1\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:329"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.110 +00:00] [DEBUG] [client/client.go:106] ["RootCoordClient GetSessions success"] [address=192.168.5.3:53100]

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

got following error:

Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.706 +00:00] [DEBUG] [indexcoord/index_coord.go:344] ["get IndexCoord component states ..."]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.523 +00:00] [WARN] [rootcoord/timeticksync.go:330] ["SendTimeTickToChannel fail"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.523 +00:00] [ERROR] [rootcoord/dml_channels.go:115] ["Broadcast failed"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"] [chanName=by-dev-rootcoord-dml_0] [stack="github.com/milvus-io/milvus/internal/rootcoord.(*dmlChannels).broadcast\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/dml_channels.go:115\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).sendTimeTickToChannel\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:368\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).startWatch.func1\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:329"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.523 +00:00] [ERROR] [rootcoord/dml_channels.go:115] ["Broadcast failed"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"] [chanName=by-dev-rootcoord-dml_1] [stack="github.com/milvus-io/milvus/internal/rootcoord.(*dmlChannels).broadcast\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/dml_channels.go:115\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).sendTimeTickToChannel\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:368\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).startWatch.func1\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:329"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.323 +00:00] [WARN] [rootcoord/timeticksync.go:330] ["SendTimeTickToChannel fail"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.323 +00:00] [ERROR] [rootcoord/dml_channels.go:115] ["Broadcast failed"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"] [chanName=by-dev-rootcoord-dml_1] [stack="github.com/milvus-io/milvus/internal/rootcoord.(*dmlChannels).broadcast\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/dml_channels.go:115\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).sendTimeTickToChannel\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:368\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).startWatch.func1\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:329"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.323 +00:00] [WARN] [rootcoord/timeticksync.go:330] ["SendTimeTickToChannel fail"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.323 +00:00] [ERROR] [rootcoord/dml_channels.go:115] ["Broadcast failed"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"] [chanName=by-dev-rootcoord-dml_0] [stack="github.com/milvus-io/milvus/internal/rootcoord.(*dmlChannels).broadcast\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/dml_channels.go:115\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).sendTimeTickToChannel\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:368\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).startWatch.func1\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:329"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.123 +00:00] [WARN] [rootcoord/timeticksync.go:330] ["SendTimeTickToChannel fail"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.123 +00:00] [ERROR] [rootcoord/dml_channels.go:115] ["Broadcast failed"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"] [chanName=by-dev-rootcoord-dml_0] [stack="github.com/milvus-io/milvus/internal/rootcoord.(*dmlChannels).broadcast\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/dml_channels.go:115\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).sendTimeTickToChannel\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:368\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).startWatch.func1\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:329"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.123 +00:00] [WARN] [rootcoord/timeticksync.go:330] ["SendTimeTickToChannel fail"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.123 +00:00] [ERROR] [rootcoord/dml_channels.go:115] ["Broadcast failed"] [error="IO error: While open a file for random read: /var/lib/milvus/rdb_data/015697.sst: Too many open files"] [chanName=by-dev-rootcoord-dml_1] [stack="github.com/milvus-io/milvus/internal/rootcoord.(*dmlChannels).broadcast\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/dml_channels.go:115\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).sendTimeTickToChannel\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:368\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).startWatch.func1\n\t/root/milvusdeb-2.1/milvus/internal/rootcoord/timeticksync.go:329"]
Jul 08 13:35:24 milvus_2_1_1 milvus[15588]: [2024/07/08 13:35:24.110 +00:00] [DEBUG] [client/client.go:106] ["RootCoordClient GetSessions success"] [address=192.168.5.3:53100]

after restart
-- Logs begin at Thu 2024-06-20 04:22:30 UTC, end at Tue 2024-07-09 05:06:05 UTC. --
Jul 09 05:06:04 milvus_2_1_1 milvus[20589]: [2024/07/09 05:06:04.940 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:06:04 milvus_2_1_1 milvus[20589]: [2024/07/09 05:06:04.940 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:06:01 milvus_2_1_1 milvus[20589]: [2024/07/09 05:06:01.939 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:58 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:58.936 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:58 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:58.936 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:58 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:58.936 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:56 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:56.767 +00:00] [DEBUG] [retry/retry.go:39] ["retry func failed"] ["retry time"=10] [error="function CompareAndSwap error for compare is false for key: indexcoord"]
Jul 09 05:05:56 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:56.573 +00:00] [DEBUG] [retry/retry.go:39] ["retry func failed"] ["retry time"=10] [error="function CompareAndSwap error for compare is false for key: datacoord"]
Jul 09 05:05:55 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:55.994 +00:00] [DEBUG] [retry/retry.go:39] ["retry func failed"] ["retry time"=10] [error="function CompareAndSwap error for compare is false for key: rootcoord"]
Jul 09 05:05:55 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:55.936 +00:00] [DEBUG] [retry/retry.go:39] ["retry func failed"] ["retry time"=10] [error="WaitForComponentStates, not meet, RootCoord current state: Initializing"]
Jul 09 05:05:55 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:55.935 +00:00] [DEBUG] [retry/retry.go:39] ["retry func failed"] ["retry time"=10] [error="WaitForComponentStates, not meet, RootCoord current state: Initializing"]
Jul 09 05:05:55 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:55.935 +00:00] [DEBUG] [retry/retry.go:39] ["retry func failed"] ["retry time"=10] [error="WaitForComponentStates, not meet, RootCoord current state: Initializing"]
Jul 09 05:05:55 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:55.934 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:55 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:55.934 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:55 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:55.934 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:52 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:52.932 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:52 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:52.932 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:52 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:52.441 +00:00] [DEBUG] [datacoord/channel_checker.go:97] ["timer started"] ["watch state"=ToRelease] [nodeID=5842] ["channel name"=by-dev-rootcoord-dml_0_435620910403944449v0] ["timeout time"=2024/07/09 05:06:12.441 +00:00]
Jul 09 05:05:52 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:52.441 +00:00] [INFO] [datacoord/handler.go:54] [GetSegmentsByChannel] [collectionID=435620910403944449] [channel=by-dev-rootcoord-dml_0_435620910403944449v0] [numOfSegments=34]
Jul 09 05:05:52 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:52.441 +00:00] [DEBUG] [datacoord/channel_manager.go:653] ["receive timeout acks from state watcher"] [state=3] [nodeID=5842] ["channel name"=by-dev-rootcoord-dml_0_435620910403944449v0]
Jul 09 05:05:52 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:52.440 +00:00] [DEBUG] [datacoord/channel_checker.go:97] ["timer started"] ["watch state"=ToRelease] [nodeID=5842] ["channel name"=by-dev-rootcoord-dml_1_435620910403944449v1] ["timeout time"=2024/07/09 05:06:12.440 +00:00]
Jul 09 05:05:52 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:52.440 +00:00] [INFO] [datacoord/channel_checker.go:104] ["timeout and stop timer: wait for channel ACK timeout"] ["watch state"=ToWatch] [nodeID=5842] ["channel name"=by-dev-rootcoord-dml_0_435620910403944449v0] ["timeout time"=2024/07/09 05:05:52.439 +00:00]
Jul 09 05:05:52 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:52.440 +00:00] [INFO] [datacoord/handler.go:54] [GetSegmentsByChannel] [collectionID=435620910403944449] [channel=by-dev-rootcoord-dml_1_435620910403944449v1] [numOfSegments=34]
Jul 09 05:05:52 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:52.440 +00:00] [DEBUG] [datacoord/channel_manager.go:653] ["receive timeout acks from state watcher"] [state=3] [nodeID=5842] ["channel name"=by-dev-rootcoord-dml_1_435620910403944449v1]
Jul 09 05:05:52 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:52.439 +00:00] [INFO] [datacoord/channel_checker.go:104] ["timeout and stop timer: wait for channel ACK timeout"] ["watch state"=ToWatch] [nodeID=5842] ["channel name"=by-dev-rootcoord-dml_1_435620910403944449v1] ["timeout time"=2024/07/09 05:05:52.439 +00:00]
Jul 09 05:05:49 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:49.931 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:49 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:49.931 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:49 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:49.930 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:46 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:46.929 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:46 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:46.928 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:43 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:43.926 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:43 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:43.926 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:43 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:43.926 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:40 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:40.925 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:37 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:37.923 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:37 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:37.923 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:36 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:36.322 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:36 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:36.322 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.786 +00:00] [WARN] [tso/tso.go:177] ["clock offset is huge, check network latency and clock skew"] [jet-lag=378.281917ms] [prev-physical=2024/07/09 05:05:35.408 +00:00] [now=2024/07/09 05:05:35.786 +00:00]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.738 +00:00] [DEBUG] [retry/retry.go:39] ["retry func failed"] ["retry time"=0] [error="function CompareAndSwap error for compare is false for key: indexcoord"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.736 +00:00] [DEBUG] [indexcoord/index_coord.go:796] [IndexCoord] ["IndexNode number"=2]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.736 +00:00] [DEBUG] [indexcoord/node_manager.go:70] ["IndexNode NodeManager setclient success"] [nodeID=5844]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.736 +00:00] [DEBUG] [indexcoord/node_manager.go:58] ["IndexCoord NodeManager setClient"] [nodeID=5844]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.736 +00:00] [INFO] [indexcoord/garbage_collector.go:100] ["IndexCoord garbageCollector start recycleUnusedIndexFiles loop"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.736 +00:00] [DEBUG] [indexcoord/node_manager.go:86] ["IndexCoord addNode"] [nodeID=5844] ["node address"=192.168.5.3:21121]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.736 +00:00] [INFO] [indexcoord/garbage_collector.go:68] ["IndexCoord garbageCollector recycleUnusedMetaLoop start"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.736 +00:00] [DEBUG] [indexcoord/index_coord.go:789] ["IndexCoord watchNodeLoop SessionAddEvent"] [serverID=5844] [address=192.168.5.3:21121]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.736 +00:00] [DEBUG] [indexcoord/index_coord.go:785] ["IndexCoord watchNodeLoop event updated"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.736 +00:00] [DEBUG] [indexcoord/index_coord.go:767] ["IndexCoord watchNodeLoop start"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.736 +00:00] [DEBUG] [indexcoord/index_coord.go:817] ["IndexCoord watchMetaLoop start"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.736 +00:00] [DEBUG] [sessionutil/session_util.go:200] ["service begin to register to etcd"] [serverName=indexcoord] [ServerID=5846]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.736 +00:00] [DEBUG] [indexcoord/service.go:155] ["indexCoord started"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.736 +00:00] [DEBUG] [indexcoord/index_coord.go:280] ["IndexCoord start successfully"] [State=Healthy]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.735 +00:00] [DEBUG] [indexcoord/service.go:80] ["IndexCoord init done ..."]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.546 +00:00] [INFO] [datacoord/session_manager.go:201] ["re-collect segment stats call succeeded"] ["DataNode ID"=5842] ["segment stat collected"="[]"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.545 +00:00] [INFO] [datanode/data_node.go:651] ["found segment(s) with stats to resend"] ["segment IDs"="[]"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.545 +00:00] [INFO] [datanode/data_node.go:648] ["start resending segment stats, if any"] ["DataNode ID"=0]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.544 +00:00] [DEBUG] [paramtable/grpc_param.go:244] ["Init back off multiplier"] [role=datanode] [grpc.client.backoffMultiplier=104857600]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.544 +00:00] [DEBUG] [paramtable/grpc_param.go:244] ["Init max back off"] [role=datanode] [grpc.client.maxBackoff=104857600]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.544 +00:00] [DEBUG] [paramtable/grpc_param.go:244] ["Init initial back off"] [role=datanode] [grpc.client.initialBackOff=104857600]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.544 +00:00] [DEBUG] [paramtable/grpc_param.go:244] ["Init max attempts"] [role=datanode] [grpc.client.maxMaxAttempts=104857600]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.544 +00:00] [DEBUG] [paramtable/grpc_param.go:244] ["Init keep alive time"] [role=datanode] [grpc.client.keepAliveTime=104857600]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.544 +00:00] [DEBUG] [paramtable/grpc_param.go:244] ["Init keep alive timeout"] [role=datanode] [grpc.client.keepAliveTimeout=104857600]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.544 +00:00] [DEBUG] [paramtable/grpc_param.go:244] ["Init dial timeout"] [role=datanode] [grpc.client.dialTimeout=104857600]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.544 +00:00] [DEBUG] [paramtable/grpc_param.go:244] ["Init client max recv size"] [role=datanode] [grpc.clientMaxRecvSize=104857600]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.544 +00:00] [DEBUG] [paramtable/grpc_param.go:244] ["Init client max send size"] [role=datanode] [grpc.clientMaxSendSize=104857600]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.544 +00:00] [INFO] [paramtable/base_table.go:459] ["Pulsar config"] ["pulsar url"=pulsar://localhost:6650] ["pulsar web url"=http://localhost:80]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.543 +00:00] [DEBUG] [retry/retry.go:39] ["retry func failed"] ["retry time"=0] [error="function CompareAndSwap error for compare is false for key: datacoord"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.542 +00:00] [DEBUG] [sessionutil/session_util.go:200] ["service begin to register to etcd"] [serverName=datacoord] [ServerID=5845]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.542 +00:00] [INFO] [datacoord/server.go:962] ["re-collecting segment stats from DataNodes"] ["DataNode IDs"="[5842]"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.542 +00:00] [INFO] [datacoord/server.go:311] ["DataNode (re)starts successfully and re-collecting segment stats from DataNodes"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.542 +00:00] [DEBUG] [datacoord/service.go:162] ["startup success"] [module=DataCoord]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.542 +00:00] [INFO] [datacoord/server.go:512] ["DataCoord creates the timetick channel consumer"] [timeTickChannel=by-dev-datacoord-timetick-channel] [subscription=by-dev-dataCoord]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.542 +00:00] [INFO] [msgstream/mq_msgstream.go:176] ["Successfully create consumer"] [channel=by-dev-datacoord-timetick-channel] [subname=by-dev-dataCoord]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.542 +00:00] [DEBUG] [server/rocksmq_impl.go:466] ["Rocksmq register consumer successfully "] [topic=by-dev-datacoord-timetick-channel] [elapsed=0]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.542 +00:00] [DEBUG] [server/rocksmq_impl.go:823] ["successfully seek to latest"] [topic=by-dev-datacoord-timetick-channel] [group=by-dev-dataCoord] [latest=451017701674940019]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.542 +00:00] [DEBUG] [server/rocksmq_impl.go:440] ["Rocksmq create consumer group successfully "] [topic=by-dev-datacoord-timetick-channel] [group=by-dev-dataCoord] [elapsed=0]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.542 +00:00] [INFO] [datacoord/garbage_collector.go:64] ["GC with option"] [enabled=true] [interval=1h0m0s] [missingTolerance=24h0m0s] [dropTolerance=24h0m0s]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.542 +00:00] [INFO] [datacoord/server.go:374] ["minio chunk manager init success"] [bucketname=a-bucket]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.542 +00:00] [INFO] [storage/minio_chunk_manager.go:107] ["minio chunk manager init success."] [bucketname=a-bucket] [root=/var/lib/milvus/data/]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.535 +00:00] [DEBUG] [components/query_node.go:54] ["QueryNode successfully started"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.535 +00:00] [DEBUG] [querynode/service.go:204] ["QueryNode start done ..."]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.535 +00:00] [INFO] [sessionutil/session_util.go:241] ["Service registered successfully"] [ServerName=querynode] [serverID=5847]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.534 +00:00] [DEBUG] [retry/retry.go:39] ["retry func failed"] ["retry time"=0] [error="WaitForComponentStates, not meet, DataCoord current state: Initializing"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.533 +00:00] [DEBUG] [sessionutil/session_util.go:299] ["SessionUtil GetSessions "] [prefix=datacoord] [key=datacoord] [address=192.168.5.3:13333]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.533 +00:00] [DEBUG] [sessionutil/session_util.go:200] ["service begin to register to etcd"] [serverName=querynode] [ServerID=5847]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.533 +00:00] [INFO] [querynode/query_node.go:305] ["query node start successfully"] [queryNodeID=5847] [IP=192.168.5.3] [Port=21123]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.533 +00:00] [INFO] [storage/minio_chunk_manager.go:107] ["minio chunk manager init success."] [bucketname=a-bucket] [root=/var/lib/milvus/data/]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.532 +00:00] [DEBUG] [indexcoord/service.go:136] ["IndexCoord try to wait for DataCoord ready"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.532 +00:00] [DEBUG] [sessionutil/session_util.go:101] ["Session connect to etcd success"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.531 +00:00] [DEBUG] [sessionutil/session_util.go:86] ["Session try to connect to etcd"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.531 +00:00] [DEBUG] [indexcoord/index_coord.go:246] ["IndexCoord init finished"] []
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.531 +00:00] [DEBUG] [indexcoord/index_coord.go:241] ["IndexCoord new task scheduler success"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.531 +00:00] [DEBUG] [indexcoord/index_coord.go:231] ["IndexCoord new minio chunkManager success"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.531 +00:00] [INFO] [storage/minio_chunk_manager.go:107] ["minio chunk manager init success."] [bucketname=a-bucket] [root=/var/lib/milvus/data/]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.530 +00:00] [INFO] [querynode/query_node.go:348] ["query node watchChangeInfo start"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.530 +00:00] [DEBUG] [querynode/service.go:199] ["QueryNode init done ..."]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.530 +00:00] [INFO] [querynode/query_node.go:276] ["query node init successfully"] [queryNodeID=5847] [IP=192.168.5.3] [Port=21123]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: 2024-07-09 05:05:35,530 DEBUG [default] [SEGCORE][SegcoreSetIndexSliceSize][milvus] set config index slice size: 16
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: 2024-07-09 05:05:35,530 DEBUG [default] [KNOWHERE][SetIndexFileSliceSize][milvus] Set knowhere::index_file_slice_size to 16
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: 2024-07-09 05:05:35,530 DEBUG [default] [KNOWHERE][SetSimdType][milvus] FAISS hook AVX2
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: 2024-07-09 05:05:35,530 DEBUG [default] [KNOWHERE][SetSimdType][milvus] FAISS expect simdType::AUTO
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: 2024-07-09 05:05:35,530 DEBUG [default] [SEGCORE][SegcoreSetSimdType][milvus] set config simd_type: auto
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.530 +00:00] [INFO] [querynode/segment_loader.go:843] ["SegmentLoader created"] [cpu-pool-size=6] [io-pool-size=32]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.529 +00:00] [INFO] [querynode/query_node.go:242] ["queryNode try to connect etcd success"] [MetaRootPath=by-dev/meta]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.529 +00:00] [INFO] [storage/minio_chunk_manager.go:107] ["minio chunk manager init success."] [bucketname=a-bucket] [root=/var/lib/milvus/data/]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.526 +00:00] [DEBUG] [sessionutil/session_util.go:393] [WatchService] ["event type"=1]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.526 +00:00] [DEBUG] [sessionutil/session_util.go:375] ["watch services"] ["add kv"="key:"by-dev/meta/session/indexnode-5844" create_revision:71029624 mod_revision:71029624 version:1 value:"{\"ServerID\":5844,\"ServerName\":\"indexnode\",\"Address\":\"192.168.5.3:21121\",\"TriggerKill\":true}" lease:7587879920605197378 "]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.526 +00:00] [DEBUG] [components/index_node.go:52] ["IndexNode successfully started"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.526 +00:00] [DEBUG] [indexnode/service.go:76] ["IndexNode start done ..."]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.526 +00:00] [DEBUG] [indexnode/service.go:181] ["IndexNode Register etcd success"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.526 +00:00] [INFO] [sessionutil/session_util.go:241] ["Service registered successfully"] [ServerName=indexnode] [serverID=5844]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.523 +00:00] [DEBUG] [indexnode/task_scheduler.go:295] ["IndexNode TaskScheduler start build loop ..."]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.523 +00:00] [DEBUG] [sessionutil/session_util.go:200] ["service begin to register to etcd"] [serverName=indexnode] [ServerID=5844]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.523 +00:00] [DEBUG] [indexnode/indexnode.go:236] ["IndexNode start finished"] []
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.523 +00:00] [DEBUG] [indexnode/indexnode.go:229] [IndexNode] [State=Healthy]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.523 +00:00] [DEBUG] [indexnode/service.go:72] ["IndexNode init done ..."]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.523 +00:00] [DEBUG] [indexnode/indexnode.go:214] ["Init IndexNode finished"] []
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: 2024-07-09 05:05:35,523 DEBUG [default] [KNOWHERE][SetIndexFileSliceSize][milvus] Set knowhere::index_file_slice_size to 16
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: 2024-07-09 05:05:35,523 DEBUG [default] [KNOWHERE][SetSimdType][milvus] FAISS hook AVX2
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: 2024-07-09 05:05:35,523 DEBUG [default] [KNOWHERE][SetSimdType][milvus] FAISS expect simdType::AUTO
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: 2024-07-09 05:05:35,522 DEBUG [default] [SERVER][operator()][milvus] Config easylogging with yaml file: /etc/milvus/configs/easylogging.yaml
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: ASSERTION FAILURE FROM EASYLOGGING++ (LINE: 307) [(assertionPassed = base::utils::File::pathExists(configurationFile.c_str(), true)) == true] WITH MESSAGE "Configuration file [/etc/milvus/configs/easylogging.yaml] does not exist!"
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.522 +00:00] [DEBUG] [indexnode/indexnode.go:208] ["IndexNode NewMinIOKV succeeded"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.522 +00:00] [INFO] [storage/minio_chunk_manager.go:107] ["minio chunk manager init success."] [bucketname=a-bucket] [root=/var/lib/milvus/data/]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.520 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.520 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.139 +00:00] [WARN] [storage/minio_chunk_manager.go:79] ["failed to check blob bucket exist"] [bucket=a-bucket] [requestID=] [error="Get "http://localhost:9000/a-bucket/?location=": dial tcp 127.0.0.1:9000: connect: connection refused"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.128 +00:00] [WARN] [storage/minio_chunk_manager.go:79] ["failed to check blob bucket exist"] [bucket=a-bucket] [requestID=] [error="Get "http://localhost:9000/a-bucket/?location=": dial tcp 127.0.0.1:9000: connect: connection refused"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.125 +00:00] [WARN] [storage/minio_chunk_manager.go:79] ["failed to check blob bucket exist"] [bucket=a-bucket] [requestID=] [error="Get "http://localhost:9000/a-bucket/?location=": dial tcp 127.0.0.1:9000: connect: connection refused"]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.119 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.119 +00:00] [DEBUG] [rootcoord/root_coord.go:1450] [GetComponentStates] ["State Code"=Initializing]
Jul 09 05:05:35 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:35.117 +00:00] [WARN] [storage/minio_chunk_manager.go:79] ["failed to check blob bucket exist"] [bucket=a-bucket] [requestID=] [error="Get "http://localhost:9000/a-bucket/?location=": dial tcp 127.0.0.1:9000: connect: connection refused"]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.949 +00:00] [DEBUG] [retry/retry.go:39] ["retry func failed"] ["retry time"=0] [error="function CompareAndSwap error for compare is false for key: rootcoord"]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.948 +00:00] [DEBUG] [sessionutil/session_util.go:200] ["service begin to register to etcd"] [serverName=rootcoord] [ServerID=5848]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.948 +00:00] [DEBUG] [rootcoord/root_coord.go:1275] ["skip reSendDdMsg with dd-msg-send set to true"]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.947 +00:00] [DEBUG] [rootcoord/proxy_manager.go:105] ["start to watch etcd"]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.947 +00:00] [DEBUG] [rootcoord/proxy_manager.go:86] ["succeed to init sessions on etcd"] [sessions=null] [revision=71029623]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.946 +00:00] [DEBUG] [rootcoord/root_coord.go:1407] ["starting service"] ["service role"=rootcoord] ["node id"=5848]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.946 +00:00] [DEBUG] [rootcoord/service.go:272] ["RootCoord Core start ..."]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.946 +00:00] [DEBUG] [rootcoord/service.go:144] ["RootCoord init done ..."]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.946 +00:00] [DEBUG] [rootcoord/root_coord.go:1243] ["RootCoord init done"]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.945 +00:00] [INFO] [rootcoord/import_manager.go:538] ["import manager starts loading from Etcd"]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.945 +00:00] [DEBUG] [rootcoord/root_coord.go:1208] ["RootCoord, set proxy manager"]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.945 +00:00] [DEBUG] [rootcoord/timeticksync.go:263] ["Add session for timeticksync"] [serverID=5848]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.945 +00:00] [DEBUG] [rootcoord/timeticksync.go:110] ["recover delta channels"] [collID=435620910403944449] ["delta channels"="[by-dev-rootcoord-delta_0,by-dev-rootcoord-delta_1]"]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.945 +00:00] [DEBUG] [rootcoord/timeticksync.go:98] ["recover physical channels"] [collID=435620910403944449] ["physical channels"="[by-dev-rootcoord-dml_0,by-dev-rootcoord-dml_1]"]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.945 +00:00] [DEBUG] [rootcoord/dml_channels.go:72] ["init dml channels"] [num=256]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.945 +00:00] [WARN] [server/rocksmq_impl.go:318] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-delta_255]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.945 +00:00] [WARN] [server/rocksmq_impl.go:318] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-delta_254]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.945 +00:00] [WARN] [server/rocksmq_impl.go:318] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-delta_253]
Jul 09 05:05:34 milvus_2_1_1 milvus[20589]: [2024/07/09 05:05:34.945 +00:00] [WARN] [server/rocksmq_impl.go:318] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-delta_252]

Anything else?

What are the possible ways to repair cluster?
Does : Too many open files could happen on docker? Should I change options?
@martinenkoEduard martinenkoEduard added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 9, 2024
@yanliang567
Copy link
Contributor

@martinenkoEduard I'd suggest you upgrade to miluvs latest v2.3.18 or v2.4.5 immediately. please try solutions here: #21454
/assign @martinenkoEduard
/unassign
@yanliang567 yanliang567 added help wanted Extra attention is needed and removed kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 10, 2024
@martinenkoEduard
Copy link
Author

martinenkoEduard commented Jul 10, 2024 •
edited
Loading

@martinenkoEduard I'd suggest you upgrade to miluvs latest v2.3.18 or v2.4.5 immediately. please try solutions here: #21454 /assign @martinenkoEduard /unassign

Linux Number of Open Files.
Is there a need to increase them for standalone docker compose installation?
Is there a need to increase limit on the host system and in docker containers?
What limit should I set?
@xiaofan-luan
Copy link
Contributor

you need to tune ulimit.

Strongly recommend you to use 2.3.18 or above.
2.1.1 is an way older version and highly recommend not to put this into production
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
3 participants
@yanliang567 @xiaofan-luan @martinenkoEduard