/dev/mem or user space burst transfer; how to get faster /dev/mem access

user_3827277 0
Viewed 782

setup

I have a bunch of RAM on the PL (programmable logic / FPGA) side of a zync-7000 chip. This memory can be accessed both via the PL and PS (processing system / CPU) side. The plan is for the CPU to load a large GiB buffer and hand it off to the PL.

Linux bursts to / from the RAM when device tree is modified

When I modify the device tree so linux can see the ram I observe fast read/write speeds; the hardware/firmware is capable of burst read/write.

    memory {
        device_type = "memory";
        // The 512 MiB memory at 0x60000000
        reg = <0x0 0x40000000 0x60000000 0x20000000>;
    };

mmap device tree memory

The device tree is modified to prevent linux from using the RAM (so it can be used as a buffer for the PL instead)

    memory {
        device_type = "memory";
        reg = <0x0 0x40000000>;
    };

mmap is slow even after playing around with flags

I have tried several ways of setting up mmap()

int* addr_start = mmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, address);
int* addr_start = mmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_POPULATE, fd, address);

While reliable, none of them give fast results when running an iterate - write / read test

// words_per_page is on the order of 2**20/4
case TEST_WRITE:
    for( int ii=0; ii < words_per_page; ii++)
        *waddr++=count++;
    break;
case TEST_READ:
    for( int ii=0; ii < words_per_page; ii++)
        sum += *raddr++;
    break;

question

Are there any user space ways of creating direct burst transactions to / from memory? If not, relevant linux kernel links would be appreciated.

zynq xilinx embedded-linux
1 Answers

You definitely need to map the region as bufferable in order to maximize transfer speeds. You might need to use a different device driver than /dev/mem.

It is easier to control transfers from the programmable logic side if you use DMA to the Zynq host memory. On Zynq, I found I needed 8 maximum-length read requests in flight at a time to maximize throughput on a link.

If you need cache coherence with user space, you will need to use the ACP port so that the processor's cache will snoop on the memory writes from the programmable logic (PL).

user_2309250
Related