-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Fix](group commit) Fix group commit wal back pressure case #37569
[Fix](group commit) Fix group commit wal back pressure case #37569
Conversation
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
run buildall |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
PR approved by at least one committer and no changes requested. |
PR approved by anyone and no changes requested. |
run buildall |
run p0 |
run p0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
…7569) ## Proposed changes <!--Describe your changes.--> Problem: The group commit WAL back pressure case failed. Reason: Previously, the group commit was restructured from **regular load** to **pipeline architecture load**, which improved the concurrency and speed of load. Consequently, the loads in the test case were completed much faster. The default batching time for a group commit is 10 seconds. Therefore, if all loads are finished in less than this time(10 seconds), the group commit data is not yet committed to the disk and visible, resulting in no data being returned by the final select statement. Solution: To resolve the issue, wait for one batching period (10 seconds) after completing all imports before executing the query. Plus, due to the faster speed of load, we move this case from p2 to p0.
## Proposed changes <!--Describe your changes.--> Problem: The group commit WAL back pressure case failed. Reason: Previously, the group commit was restructured from **regular load** to **pipeline architecture load**, which improved the concurrency and speed of load. Consequently, the loads in the test case were completed much faster. The default batching time for a group commit is 10 seconds. Therefore, if all loads are finished in less than this time(10 seconds), the group commit data is not yet committed to the disk and visible, resulting in no data being returned by the final select statement. Solution: To resolve the issue, wait for one batching period (10 seconds) after completing all imports before executing the query. Plus, due to the faster speed of load, we move this case from p2 to p0.
Proposed changes
Problem: The group commit WAL back pressure case failed.
Reason: Previously, the group commit was restructured from regular load to pipeline architecture load, which improved the concurrency and speed of load. Consequently, the loads in the test case were completed much faster. The default batching time for a group commit is 10 seconds. Therefore, if all loads are finished in less than this time(10 seconds), the group commit data is not yet committed to the disk and visible, resulting in no data being returned by the final select statement.
Solution: To resolve the issue, wait for one batching period (10 seconds) after completing all imports before executing the query.
Plus, due to the faster speed of load, we move this case from p2 to p0.