how to know that segments compaction done and why this process takes so much time

[rongilaad]

I have switched from segment size 7.6GB to 9.6GB, restarted milvus and reloaded the wiki50G collection. After 6 hours there is no change in the number of segments

[yhmo]

Compaction machinery compacts segments according to pre-defined configurations and algorithms.
The configurations about compaction is defined in the milvus.yaml:

dataCoord:
 segment:
    maxSize: 1024 # Maximum size of a segment in MB
    diskSegmentMaxSize: 2048 # Maximun size of a segment in MB for collection which has Disk index
    sealProportion: 0.12
    assignmentExpiration: 2000 # The time of the assignment expiration in ms
    allocLatestExpireAttempt: 200 # The time attempting to alloc latest lastExpire from rootCoord after restart
    maxLife: 86400 # The max lifetime of segment in seconds, 24*60*60
    # If a segment didn't accept dml records in maxIdleTime and the size of segment is greater than
    # minSizeFromIdleToSealed, Milvus will automatically seal it.
    # The max idle time of segment in seconds, 10*60.
    maxIdleTime: 600
    minSizeFromIdleToSealed: 16 # The min size in MB of segment which can be idle from sealed.
    # The max number of binlog file for one segment, the segment will be sealed if
    # the number of binlog file reaches to max value.
    maxBinlogFileNumber: 32
    smallProportion: 0.5 # The segment is considered as "small segment" when its # of rows is smaller than
    # (smallProportion * segment max # of rows).
    # A compaction will happen on small segments if the segment after compaction will have
    compactableProportion: 0.85

The "smallProportion" is a threshold that to determine whether a segment can be merged.
If segment.maxSize = 7.6GB, only the segments(size < 3.8GB) are treated as "small segment" and can be merged.
If segment.maxSize = 9.6GB, only the segments(size < 4.8GB) are treated as "small segment" and can be merged.

So, although you change the segment.maxSize to 9.6GB, the number of "small segment"(size < 4.8GB) is less than 2, compaction will not work.

[rongilaad]

Thanks yhmo!
I have created the wiki50M collection with segment size of 6.4GB and had 27 segments.
Then I have reconfigured maxSize to 7.7GB and got 21 segments.
Now I am trying to perform another compaction with maxSize 9.6GB, but it doesn't work.
The majority of my segments are around 6.08 to 6.09 GB in size. A few segments are larger, around 7.97 to 9.12 GB. I have changed the small proportion from 0.5 to 0.75. Since maxSize is 9.6GB - All segments smaller than 7.2 GB will be considered small and eligible for compaction: Still no compaction occurs.
There are no errors In the log, after loading is done garbage collector recycle task started and ended - at the end of the log.
Here is my configuration:

segment:
maxSize: 9844
diskSegmentMaxSize: 9844
sealProportion: 0.9
# maxSize: 1024 # Maximum size of a segment in MB
# diskSegmentMaxSize: 2048 # Maximun size of a segment in MB for collection which has Disk index
# sealProportion: 0.12
assignmentExpiration: 2000 # The time of the assignment expiration in ms
allocLatestExpireAttempt: 200 # The time attempting to alloc latest lastExpire from rootCoord after restart
maxLife: 86400 # The max lifetime of segment in seconds, 246060
# If a segment didn't accept dml records in maxIdleTime and the size of segment is greater than
# minSizeFromIdleToSealed, Milvus will automatically seal it.
# The max idle time of segment in seconds, 10*60.
maxIdleTime: 600
minSizeFromIdleToSealed: 16 # The min size in MB of segment which can be idle from sealed.
# The max number of binlog file for one segment, the segment will be sealed if
# the number of binlog file reaches to max value.
maxBinlogFileNumber: 32
smallProportion: 0.75 # 0.5 # The segment is considered as "small segment" when its # of rows is smaller than
# (smallProportion * segment max # of rows).
# A compaction will happen on small segments if the segment after compaction will have
compactableProportion: 0.85
# over (compactableProportion * segment max # of rows) rows.
# MUST BE GREATER THAN OR EQUAL TO !!!
# During compaction, the size of segment # of rows is able to exceed segment max # of rows by (expansionRate-1) * 100%.
expansionRate: 1.25
autoUpgradeSegmentIndex: false # whether auto upgrade segment index to index engine's version
enableCompaction: true # Enable data segment compaction
compaction:
enableAutoCompaction: true
indexBasedCompaction: true
rpcTimeout: 180 #10
timeout: 5600
maxParallelTaskNum: 10
workerMaxParallelTaskNum: 2
levelzero:
forceTrigger:
minSize: 8388608 # The minmum size in bytes to force trigger a LevelZero Compaction, default as 8MB
maxSize: 85899345920
# maxSize: 67108864 # The maxmum size in bytes to force trigger a LevelZero Compaction, default as 64MB
deltalogMinNum: 10 # The minimum number of deltalog files to force trigger a LevelZero Compaction
deltalogMaxNum: 30 # The maxmum number of deltalog files to force trigger a LevelZero Compaction, default as 30
enableGarbageCollection: true

[xiaofan-luan]

I have switched from segment size 7.6GB to 9.6GB, restarted milvus and reloaded the wiki50G collection. After 6 hours there is no change in the number of segments

could you show your segment size distribution?
You 7G+6G can not be merged because it passed 9.6GB limit.
Also, 7.6G seems to be a large enough. We don't recommend to increase to more than 8G because this might cause other serious problems