[Bug]: Not able to reduce segment size in V2.4.5

[sumanthhuddar]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.4.5
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    kafka
- SDK version(e.g. pymilvus v2.0.0rc2):java
- OS(Ubuntu or CentOS): ubuntu
- CPU/Memory: 32 CPU/120GB 
- GPU: 
- Others:

Current Behavior

I loaded collection with maxsegment size set to 1024 MB , after loading ~4 M , i had ~26 segments. I Changed maxsegment to 4GB , and restarted datacoord node, and nunber of segments reduced from 26 to 8 . Now when i try to increase number of segments by setting maxsegmentsize to 2GB , it is not effective. tried restarting data coord and data nodes , load\unload collection , compact , also tuned with sealProportion with values 0.12,.0.24,.75 but there is no change in segmets\segment size.

Expected Behavior

Decreasing maxsegment should increase number of segments.

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[yanliang567]

increasing the value of maxsegmentsize is a one-way street for the existing segments. Milvus only has compaction policy to compact small segments to larger ones, but has no policy to split the large segments to smaller ones. So for now this is expected. @sumanthhuddar

/assign @sumanthhuddar
/unassign

[xiaofan-luan]

I thought this is actaully doable.
With the new clustering compaction, segment can be split into the size limtied.

@wayblink could you please take a look at it

[wayblink]

@sumanthhuddar As @yanliang567 mentioned, it is currently expected. We will think about adding policy to split the large segments to smaller ones. Clustering compaction can compact segments to smaller size. However, it needs alter the collection, set clustering key field first, it is not supported now.

I wonder what's your purpose for trying different maxSegmentSize? We can find out best solution for you.

[sumanthhuddar]

@wayblink I am doing some perf test , and started with maxSegmentSize = 1024 MB , and after loading data , we had 26 sealed segments , with this we didn't saw good performance , so we reduced number of segments to 8 and got good latency . for testing purpose we further reduced segment count to 2. I thought we can increase\decrease maxSegmentSize to right size the segments. It would be nice if we have this option to compact bigger segments to smaller segments in future release.

[xiaofan-luan]

again we don't recommend to do so, PLEASE keep segment size to be less than 8G, unless you never delete, update and insert

[xiaofan-luan]

Build a huge index is not really useful in production environment. You have many other way to optimize performance, and scale or use zilliz cloud is usaully the easiest way.

We have so much pain to use larger index so my suggesion is simply "Don't do that!"

[sumanthhuddar]

@xiaofan-luan thanks for the suggestions.

[xiaofan-luan]

Segment size of 2GB or 4GG is reasonable.
We recomend user with 16GB memory or more to use 2GB segment size. 32GB or more to use 4GB segment szie.
The maximum segment size we recommend is 8GB.

The reason is build index a 8GB or more segment is very expensieve (Takes hours to do so with 8core32G index, might even OOM). also frequent update, insert will bring trouble as well