I have a Linux C++ multi threaded program which has lower average latency. But it has latency spikes. All latency critical threads are running in isolated CPUs with continuous polling mode. All maskable interruptions are bound to different CPUs so that they do not interrupt latency critical threads.
Profiling tools such as callgrind, gprof, vtune all show only the average cost of a function call (not the max values or percentiles).
How can I identify the pieces of code which is causing these latency spikes ?