My question is how and when one thread wakes up other thread(s)? I tried to look at Linux kernel code, but didn't find what I was looking for. For example, there is one thread waiting on mutex, conditional variable, file descriptor (event fd, for example).
What work is performed by thread that releases mutex, and work is performed by other cpu core that is about to run sleeping thread?
I have searched existing answers, but did not find details. I have have read that scheduler can usually be called:
- after a system call before returning to userspace
- after interrupt processing
- after timer interrupt processing - for example, every 1ms (hz = 1000) ок 4ms (hz = 250)
I believe that thread that releases some resource, it calls through some system call kernel function try_to_wake_up. This function picks some task(s) and sets its state to RUNNABLE. This work is performed by signaling thread and takes some time. But how actually task is started to run? If system is buzy, there may be no free cpus to run this task. Some time in the future, for example, after call by timer when some other thread goes to sleep or exhausted its quantum, scheduler is called on some cpu and takes runnable task for running. Maybe this task will be preferably run on that cpu where it ran previously.
But there must be some other scenario. When there are idle cpus, I believe that task is awakened immediately, without waiting at most 1ms or even 4ms (wake-up latency is always around several microseconds, not milleseconds).
Also, for example, imagine situation when some thread is running exclusively on some cpu core. This cpu core may be isolated from kernel threads and interrupts handlers and only one user thread has affinity set to run on this and only on this core. I believe that if there are enough free cpu cores, no other threads will be normally scheduled to run on that core (am I wrong?) Also this cpu core may have nohz_full option enabled. So when user thread goes to sleep, this core goes to sleep too. No irqs from devices, no timer irqs are processed.
So there must be some way for one cpu to tell other cpu to start running (throug interrupt), call scheduler and run user thread that is ready to awake. Scheduler must run not on the cpu that releases resource, but on some other cpu, that should be awakened. Maybe this is performed somehow via IPI interrupt? Can you help me to find corresponding code in the kernel or describe how it works? Thank you.