In general, in CPUs, memory request passes through memory hierarchy.
However, when OS or something decides to evict a page from the memory,
the page related to the memory request that is still being processed in CPU must not be evicted from the memory.
How to guarantee that such pages are not evicted from the memory?
Are there something like page reference counter to prevent the page from being evicted when the request is being handled in CPU?
Speaking from a Linux perspective, there are a few mechanisms that help make sure this scenario can be handled safely.
1) The way Linux handles page replacement policies. Linux uses several LRU lists for its page cache. For file-backed and anonymous pages, each has a respective active and inactive list of pages. When a page is used, the CPU will mark the page as used through its PAGE_BIT_ACCESSED flag in the corresponding page table entry (pte). When this happens, Linux can put the page in the active list. After some time, the page replacement handler tries to kick the page out, but the page must go through a few rounds and steps before being kicked out. It's accessed bit must be turned off, then it must be put on the inactive list. Each time the handler executes one of these steps, it lets the page live and moves on to the rest of the list looking for pages to evict, perhaps until it reaches our page again. Since each of these steps involves several iterations through the LRU lists, presumably any CPU operations that made our page initially active will have ended by the time the handler comes back around to our page to evict it. If, on the other hand, the CPU accessed the page again while our page was on the inactive list waiting to be evicted, its accessed bit will be turned on and it can again be placed on the active list. The key here is that Linux generally tries to keep a certain number of pages in the inactive list so that when it's time to evict, only inactive pages are chosen. This is a way to prevent your situation from occurring, where an active page is kicked out.
2) Swapping. Suppose your scenario actually happened. The way you describe eviction makes it seem like active memory that's evicted is a fatal error. In reality, when a page is evicted either it is a file-backed page that is written back to disk or an anonymous page that is written to swap space (assuming you have swap space). Thus, barring any power failures, memory won't be lost during eviction. Additionally any cache entries that hold data corresponding to this page will be invalidated. So when the CPU tries to read/write to a page, it will see the cache entries as being invalid raising a page fault. As with any other page fault, the page would be brought back in to memory and up through the cache/memory hierarchy. (A good explanation of CPU's behavior during a page fault)
3) Page locking. If you truly needed to lock pages, Linux has a way to do that. From within the Linux kernel, there are page locking functions (like lock_page()) that are used during I/O operations or page/page_table operations. There is also the mlock() family of functions that can be called from user space to lock specific memory into RAM. Notice that mlock doesn't guarantee that the memory stays in the same physical address, just that it stays in RAM (guaranteeing at worst a soft page fault).
