Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement]: Concurrency control of clustering compaction #34544

Closed
1 task done
wayblink opened this issue Jul 9, 2024 · 1 comment
Closed
1 task done

[Enhancement]: Concurrency control of clustering compaction #34544

wayblink opened this issue Jul 9, 2024 · 1 comment
Labels
kind/enhancement Issues or changes related to enhancement

Comments

@wayblink
Copy link
Collaborator

wayblink commented Jul 9, 2024

Is there an existing issue for this?

  • I have searched the existing issues

What would you like to be added?

#34495

Clustering compaction is much bigger task than normal compaction. And milvus doesn't have a strict memory control. Hence, we need to control the concurrency of clustering compactions.

Why is this needed?

No response

Anything else?

No response
@wayblink wayblink added the kind/enhancement Issues or changes related to enhancement label Jul 9, 2024
This was referenced Jul 9, 2024
Closed
Merged
Merged
@xiaofan-luan
Copy link
Contributor

what is the memory estimation of a clustering compaction? is it related to segment number * bin log size?

what is the expected memory usage and concurrency?
sre-ci-robot pushed a commit that referenced this issue Jul 18, 2024
@wayblink
enhance:[cherry-pick] Add compaction task slot usage logic (#34625)
a26e965 
issue: #34544
pr: #34581

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
sre-ci-robot pushed a commit that referenced this issue Jul 18, 2024
@wayblink
enhance: Add compaction task slot usage logic (#34581)
c79d1af 
#34544

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement Issues or changes related to enhancement
2 participants
@wayblink @xiaofan-luan