I have a problem firstly on Java 11 (OpenJdk, Debian).
All I need is just to interrupt a thread after a 20ms deadline. For example like in google GRPC java client library. So, common way to achieve this is to create a ScheduledThreadPoolExecutor and submit a Watchdog task with a delay (20 ms in my case). After that we just need to send the interrupt to the worker thread that is waiting. Of course, watchdog threads should have maximum priority.
The watchdog thread triggers almost exactly after 20ms in case of low system load. And this is expected. But in case of 100% system load by other threads (number of threads is the same as CPU core number or twice as big) ScheduledThreadPoolExecutor.schedule with delay 20 ms can awake sometimes after 25ms or 40 and more milliseconds!
How my single highest priority thread can't get CPU time for more than 40 ms?
I've tried it on my laptop with core i7 6 cores with HT (12 logical) / nix 5.3.0-40-generic #32~18.04.1-Ubuntu SMP
After that I wrote some C++ code to test accuracy for sleep and awake high priority thread and this happens again! The watchdog thread just sleeps for 1 ms (want to check state 1000 times in second). Behaviour is somewhat crazy.
Moreover - at my friend's laptop it works OK with 100+ worker threads showing always under 10 us delay. The program runs under sudo to allow set thread priority.
g++ -pthread -o threads threads.cpp
#include <iostream>
#include <thread>
#include <vector>
#include <algorithm>
#include <chrono>
#include <cstring>
#include <pthread.h>
void watchdog() {
long i = 9000000000;
auto last = std::chrono::high_resolution_clock::now();
int sleep_duration_us = 1000;
while (i-- > 0) {
std::this_thread::sleep_for (std::chrono::microseconds(sleep_duration_us));
auto now = std::chrono::high_resolution_clock::now();
auto delay = std::chrono::duration_cast<std::chrono::microseconds>(now-last).count() - sleep_duration_us;
if (i % 100 == 0)
std::cout << "Awakening delay: " << delay << " us\n";
if (delay > 1000)
std::cerr << "LONG DELAY!!! " << delay << " us" << std::endl;
// if (sleep_time > 1000000) {
// std::cout << "\n watchdog fail - " << sleep_time ;
// }
// if (i % 10000000 == 0)
// std::cout << "\n watchdog - " << std::chrono::duration_cast<std::chrono::nanoseconds>(now-last).count() ;
last = now;
}
}
int main()
{
// vector container stores threads
std::vector<std::thread> workers;
for (int i = 0; i < 10; i++) {
auto t = std::thread([]()
{
int i = 0, j = i;
while (true) {
if (j % 2 == 0)
j /= 2;
else
j = 3 * j + 1;
if (j == 123456789)
std::cout << j << std::endl;
}
});
workers.push_back(std::move(t));
}
std::cout << "main thread\n";
std::thread tw(watchdog);
sched_param sch;
int policy;
pthread_getschedparam(tw.native_handle(), &policy, &sch);
std::cout << "default priority " << sch.sched_priority << std::endl;
sch.sched_priority = 99;
if(pthread_setschedparam(tw.native_handle(), SCHED_RR, &sch)) {
std::cout << "Failed to setschedparam: " << std::strerror(errno) << '\n';
}
pthread_getschedparam(tw.native_handle(), &policy, &sch);
std::cout << "new priority " << sch.sched_priority << std::endl;
std::for_each(workers.begin(), workers.end(), [](std::thread &t)
{
std::cout << t.get_id() << "\n";
});
std::this_thread::sleep_for(std::chrono::seconds(15));
std::cout << "finish\n";
return 0;
}
My output is:
root@openminder-nix:/source/scheduler_test# sudo ./threads
main thread
default priority 0
new priority 99
140018767865600
140018759472896
140018751080192
140018742687488
140018734294784
140018725902080
140018717509376
140018709116672
140018700723968
140018692331264
Awakening delay: 2 us
Awakening delay: 2 us
Awakening delay: 2 us
Awakening delay: 2 us
Awakening delay: 2 us
Awakening delay: 3 us
Awakening delay: 4 us
Awakening delay: 2 us
Awakening delay: 2 us
Awakening delay: 2 us
Awakening delay: 2 us
Awakening delay: 3 us
Awakening delay: 2 us
Awakening delay: 3 us
Awakening delay: 2 us
LONG DELAY!!! 2048 us
Awakening delay: 7 us
Awakening delay: 4 us
Awakening delay: 5 us
LONG DELAY!!! 2199 us
Awakening delay: 5 us
Awakening delay: 4 us
Awakening delay: 6 us
Awakening delay: 5 us
Awakening delay: 8 us
Awakening delay: 5 us
Awakening delay: 5 us
Awakening delay: 4 us
Awakening delay: 6 us
LONG DELAY!!! 1195 us
So please help me to understand linux threads scheduling mechanics and write code with watchdog accuracy in 1ms.