Thread wakening may be bottom neck for large core systems about inferllm

[chenqy4933]

Can you show the trace result? or you can comment the thread::yield() function, and trace again

[xhebox]

Can you show the trace result? or you can comment the thread::yield() function, and trace again

Later. I've modified the base code a lot. It is like so:

main      | add_task                                         | add_task
thread1 | matmul_int8 
thread2 | ................. matmul_int8 ( this is problematic part, if it can start much earlier, the total time will be reduced a lot)
...
threadx | ..matmul_int8 

[chenqy4933]

It seems when dispatching a task， not all threads will be scheduled right now， some threads yield out. if so, comment the thread::yield and test again, it should be faster, but I test with the same result.

[xhebox]

It seems when dispatching a task， not all threads will be scheduled right now， some threads yield out. if so, comment the thread::yield and test again, it should be faster, but I test with the same result.

It could be an advantage of x64 processors. I mean these processors may be more responsive anyway.

And yes, removing yield works. But not always. It makes processors too busy on atomic operations sometimes. Racing will slow down the whole system. I am still experimenting on how to make the loop better.

I've got a pretty good result so far. With some other new optimizations, I got 5token/s.

[chenqy4933]

great ！！！， Looking forward to your optimizations

[xhebox]

great ！！！， Looking forward to your optimizations

#62

Check out this demo. Add ZoneScopedNS() in the lambda function of matmul will give you the trace result.

[xhebox]

@chenqy4933 Got 4.4 tokens with the master. I guess the optimization works. Though I did not get as high as 5 token/s.

[chenqy4933]

@chenqy4933 Got 4.4 tokens with the master. I guess the optimization works. Though I did not get as high as 5 token/s.

you can optimize it continue， I just optimized it with the CPU level yield.