[DSIP-55][Master] Separate the waiting dispatched task into different queue by worker group

[ruanwenjun]

Search before asking

• I had searched in the DSIP and found no similar DSIP.

Motivation

Right now, in master DS will use one single thread to dispatch the tasks. The problem is all tasks are stored at one waiting queue, if a task dispatch failed then it will be moved to the end of waiting queue.

There exist a case, if a worker is overload at one worker group, and A, B ,C are using the worker, the priority is A > B > C, if the A dispatched failed, then begin to dispatch B, but the worker changed from overload to normal, then B will be dispatched successfully, this will caused problem, since priority of A is higher than B.

Design Detail

Different worker group use different waiting queue.

We need to make one waiting queue should only be handled by one dispatch thread. Once a task dispatched failed it should be put back to its worker group queue.

Compatibility, Deprecation, and Migration Plan

No response

Test Plan

No response

Code of Conduct

• I agree to follow this project's Code of Conduct