How to record a timestamped trace of every memory access?

If you are on Intel, I think the Intel PT feature mentioned in the other answers combined with post-processing and analysis is most likely to get you what you want at high speed (i.e., something like a single digit regression in performance).

If you don't care about performance, you could use any number of binary instrumentation frameworks to get this information. For example, the valgrind framework has a cachegrind tool which captures every memory access and uses them to estimate cache behavior based on an idealized caching model.

You could pretty much modify the cachegrind tool to spit out the list of accesses you are after, along with a timestamp. Of course, the problem is that cachegrind probably runs something like 10 times slower than the native application, so your timestamps will both be "stretched out" and distorted (i.e., because various parts of the program might have different instrumentation overheads).

Whether that matters for your application is up to you.

The nice thing about Valgrind is that it doesn't rely on any particular hardware and works across different hardware architectures. It is probably also easier than getting an Intel PT-based analysis working - although I'm not 100% sure since I have tried either myself.

If you don't care about the total runtime of the actual process while you are recording, but need mostly accurate timing figures, you could also consider running your process under a CPU simulator, such as the Sniper x86 simulator or gem5 that Peter mentions in the comments.

This site which describes the CMP$im tool may be very useful for you. It is able to produce a trace of accesses using Intel's PIN technology, which @Leeor also mentioned in the comments below. I recommend taking a look at the author's associated papers, linked from that site.

The closest hardware capability I can think of is Intel PT (processor trace), which can record timestamps on every (taken?) branch, so you can reconstruct execution down to the block containing the loads. I haven't used pt, and I'm not sure if perf can use it, or if you need different programs.

(Not exactly a "basic block" because there's no record when executing past the target of a branch somewhere else)

That's probably only when the load instructions were issued, not when out-of-order execution actually ran them or when the data arrived from memory/L1d cache.

I don't think any existing x86 chips can record accurate timestamps for every load completion; that would be too much data.

If you're looking for memory hotspots, I'd suggest profiling with perf record -e mem_load_retired.l3_miss,mem_load_retired.l2_miss or similar counters, to look for loads that miss often in different levels of cache. There are some store events, but mostly for loads because the CPU has to wait for load data to arrive before it can use it.

Maybe also dtlb_load_misses.miss_causes_a_walk or other TLB-miss events.

There's also an event for cycle_activity.stalls_l3_miss which counts every cycle when stalled, to look for cases where OoO exec couldn't hide cache-miss latency.

Use perf list to see the events perf knows about. If your perf is old, you might need the ocperf.py wrapper for it. https://github.com/andikleen/pmu-tools

IntelPT will record timestamps and track the control flow information of a running application with various packets that will be logged into the hardware. This information from IntelPT can then act as input to decoders which will help to obtain the disassembled trace of instructions. And IntelPT has also been integrated into perf.

You can use perf with IntelPT as events as below -

perf record -e intel_pt//[uk]  /bin/ls

[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.384 MB perf.data ]

However, what I would suggest is using PEBS (Precise Event Based Sampling). PEBS (Precise Event Based Sampling) is a feature available to a subset of events which allows the hardware to collect additional information very close to the exact time the configured event overflowed. You can use PEBS with perf as well.

Say you want to record information related to a memory load. The PEBS counters will be initialized to a certain maximum value (which is actually the period of sampling). These counters will then decrement by one with each memory load. As soon as the counter hits zero, the PEBS hardware gets armed. The next memory load event will then cause a PEBS record to be written into the PEBS buffer. Once this happens, the PEBS counters are automatically reset to their previous value. This is how, a sample period of 2 will cause the system to record memory loads after an interval of 2.

Anyway, one benefit of using PEBS is that it is very precise, which can be guessed from the way it works. Unlike most other recording mechanisms, where you have to essentially wait for software interrupts to record the event details and the recording happens hundreds of CPU cycles later.

Use PEBS in conjuction with perf to record memory loads like this -

perf record -e r81d0:pp -c 1 -d <application_name> <application_params>

r81d0:pp represents the event memory loads amongst retired instructions in numeric form. In certain cases, certain CPU architectures will not support some events and one is forced to use numeric events like this.

However like Peter said and as has been highlighted in many other questions and answers here, it is absolutely impossible to record 100% memory load or memory store addresses without external hardware mechanisms and/or causing significant overheads to the runtime.

Want to read about PEBS ?. Intel's software developer manual will be your best friend.