Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Channel-cp meta still exists after the collection GC #34545

Closed
1 task done
jaime0815 opened this issue Jul 9, 2024 · 3 comments
Closed
1 task done

[Bug]: Channel-cp meta still exists after the collection GC #34545

jaime0815 opened this issue Jul 9, 2024 · 3 comments
Assignees
@jaime0815
Labels
kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.

Comments

@jaime0815
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: 2.4 
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

We couldn't find any meta data about collection id: 449684528748778322 in etcd except for 06b84fe16780ed1/meta/datacoord-meta/channel-cp/in01-06b84fe16780ed1-rootcoord-dml_3_449684528748778322v0

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response
@jaime0815 jaime0815 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 9, 2024
@jaime0815 jaime0815 self-assigned this Jul 9, 2024
This was referenced Jul 10, 2024
Merged
Merged
sre-ci-robot pushed a commit that referenced this issue Jul 11, 2024
@jaime0815
enhance: skip orphan channel cp meta when checking cp lag (#34556)
073a501 
issue: #34545
pr: #34555

Signed-off-by: jaime <yun.zhang@zilliz.com>
sre-ci-robot pushed a commit that referenced this issue Jul 11, 2024
@jaime0815
enhance: skip orphan channel cp meta when checking cp lag (#34555)
c332f69 
issue: # #34545

Print warn log instead of check health fail if orphan channel cp meta is
found in health check request.

Signed-off-by: jaime <yun.zhang@zilliz.com>
@xiaofan-luan
Copy link
Contributor

image
@xiaofan-luan
Copy link
Contributor

this has to be moved out of segment gc loop
@jaime0815
Copy link
Contributor Author

The channel cp meta can not be removed when no dropped segment GC happens, such as:

  1. Drop an empty collection that never inserts any data
  2. Import task fails will remove segment meta directly in the older version
This was referenced Jul 11, 2024
Merged
Merged
sre-ci-robot pushed a commit that referenced this issue Jul 11, 2024
@jaime0815
fix: encountering orphan channel-cp meta after GC (#34609)
1d2062a 
issue: #34545
pr: #34612

Signed-off-by: jaime <yun.zhang@zilliz.com>
sre-ci-robot pushed a commit that referenced this issue Jul 11, 2024
@jaime0815
fix: encountering orphan channel-cp meta after DataCoord GC (#34612)
a08a0c8 
issue: #34545

Signed-off-by: jaime <yun.zhang@zilliz.com>
jaime0815 added a commit to jaime0815/milvus that referenced this issue Jul 12, 2024
@jaime0815
fix: encountering orphan channel-cp meta after GC (milvus-io#34609)
7017ddb 
issue: milvus-io#34545
pr: milvus-io#34612

Signed-off-by: jaime <yun.zhang@zilliz.com>
@jaime0815 jaime0815 mentioned this issue Jul 12, 2024
Merged
jaime0815 added a commit to jaime0815/milvus that referenced this issue Jul 12, 2024
@jaime0815
fix: encountering orphan channel-cp meta after GC (milvus-io#34609)
2fc6999 
issue: milvus-io#34545
pr: milvus-io#34612

Signed-off-by: jaime <yun.zhang@zilliz.com>
sre-ci-robot pushed a commit that referenced this issue Jul 12, 2024
@jaime0815 @LoveEachDay
fix: cherry pick bug fix for orphan meta and metrics (#34643)
9a1fabb 
issue: #34545 #34041
pr: #34609 #34010 #34592

---------

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: edward.zeng <jie.zeng@zilliz.com>
chyezh pushed a commit to chyezh/milvus that referenced this issue Jul 15, 2024
@jaime0815 @chyezh
fix: encountering orphan channel-cp meta after DataCoord GC (milvus-i…
8161ad1 
…o#34612)

issue: milvus-io#34545

Signed-off-by: jaime <yun.zhang@zilliz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.
2 participants
@jaime0815 @xiaofan-luan