[Bug]: Loading collection stuck at 0% after reboot

[wjddd]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.3.5
- Deployment mode(standalone or cluster): standalone
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus v2.3.5
- OS(Ubuntu or CentOS): centos7
- CPU/Memory: 64c128g

Current Behavior

65 collection in total but only 4 of them successfully loaded, others just keep loading.

Milvus Log

milvus.zip

[yanliang567]

/assign @longjiquan
/unassign

[wjddd]

@longjiquan Hi, is there any solution for this?

[xiaofan-luan]

/assign @bigsheeper

[xiaofan-luan]

@longjiquan Hi, is there any solution for this?

What happened after reboot?
Did your disk ever be full?

[xiaofan-luan]

what's the reason of it's reboot?
please describe what happens to this cluster

[wjddd]

what's the reason of it's reboot? please describe what happens to this cluster

It's just a routine maintenance, we have enough free space for milvus.
Everything is ok except milvus got crashed.

[bigsheeper]

what's the reason of it's reboot? please describe what happens to this cluster

It's just a routine maintenance, we have enough free space for milvus. Everything is ok except milvus got crashed.

@wjddd Could you provide the logs from just before the Milvus crash? Were there any unusual occurrences at the time of the crash?

[wjddd]

what's the reason of it's reboot? please describe what happens to this cluster

It's just a routine maintenance, we have enough free space for milvus. Everything is ok except milvus got crashed.

@wjddd Could you provide the logs from just before the Milvus crash? Were there any unusual occurrences at the time of the crash?

We don't keep them for so long. All the logs that we can find are provided in milvus.zip. Everything was fine before reboot. After reboot, the loading got stuck, so we deleted more than 5000 empty collections and restarted all the containers.

[xiaofan-luan]

Let's try the reproduce like this:
create 10000 collections,
each collection write 100 entities.
kill the cluster and see if it can recovery @wjddd any suggestions?

BTW, we don't recommend to create so much collections

[wjddd]

Let's try the reproduce like this: create 10000 collections, each collection write 100 entities. kill the cluster and see if it can recovery @wjddd any suggestions?

BTW, we don't recommend to create so much collections

OK, I'll give it a try.