Low PCI bus read performance despite write-combined

Question

I have a test system consisting of a normal x86 platform with Skylake CPU and a Debian Linux 4.9.x with RT Patch. In a PCI slot there is a kind of network card to which I send data or read from there. The card is a 32Bit PCI card, which does not support bus mastering but allows arbitrary long bursts (latency=0). The OS recognizes the memory region of the card as 32bit - non-prefetchable. The CPU chipset on the motherboard has no integrated PCI bus controller, but there is a separate PCIe to PCI bridge connected to a PCIe lane.

Because the card does not support bus mastering, the complete data transfer must be handled by the CPU. I use a mmap which I access from the user space. Although the card reports itself as non-prefetchable I know that a part of the internal memory area is still prefetchable. Therefore I have divided the memory area into 2 segments. One segment is uncachable and the other segment is defined as Write-Combined (WC).

My problem is that reading from the card is very slow. The two variants below to read data from the card do not differ so much in its reading performance: Variant 1 about 14kByte/ms, Variant 2 about 10kByte/ms. Version 1 works with a burst length of 16 (64Byte) and version 2 with a burst length of 8 (32Byte). The primary problem with both variants is that both have a time lag of about 1.9µs between two consecutive bursts, which significantly reduces the read performance. The bursts themselves have a length of about 0.25µs or 0.5µs.

Especially for version 1 I would have expected the time gap between consecutive bursts to be much smaller by using the WC memory and "non-temporal" memory operations. Version 1 also does not work reliably. Occasionally there are unexpected additional bursts. I would have expected that when executing "x0 = _mm_stream_load_si128( pSrc +0 )" a burst is triggered and the whole cacheline of 64Byte is filled. That happens too. The following read operations (x1-x3) then reads the cached data without another burst. Occasionally however, additional bursts occur when reading to X1, x2 or x3. In this case the gap between the bursts is also 1.9µs again, which unnecessarily prolongs the reading process. Version 2 is reliable but even slower than version 1.

Variant 1:

for( i = 0; i < length; i +=64 )
{
  x0 = _mm_stream_load_si128( pSrc +0 );
  x1 = _mm_stream_load_si128( pSrc +1 );
  x2 = _mm_stream_load_si128( pSrc +2 );
  x3 = _mm_stream_load_si128( pSrc +3 );
  _mm_store_si128( pDst +0, x0 );
  _mm_store_si128( pDst +1, x1 );
  _mm_store_si128( pDst +2, x2 );
  _mm_store_si128( pDst +3, x3 );
  pSrc += 4;
  pDst += 4;
}
_mm_mfence();

Version 2:

for( i = 0; i < length; i +=64 )
{
  x0 = _mm256_load_si256( pSrc +0 );
  x1 = _mm256_load_si256( pSrc +1 );
  _mm256_store_si256 ( pDst +0, x0 );
  _mm256_store_si256 ( pDst +1, x1 );
  pSrc += 2;
  pDst += 2;
}

I have the following questions:

Can a 32Bit PCI card in a 64Bit Linux OS even have prefetchable I/O memory?
Does the memory area have to be prefetchable to use Write-Combined?
During bootup the configuration data of the card are read and the I/O memory is configured as non-prefetchable based on the data in the BAR register. Can I still configure this memory as Write Combined in the device driver after the boot process?
Why do read accesses to the PCI card take so long in both variants? Why is the gap between bursts so long even with consecutive bursts, no matter which variant I use?
What is the explanation for the random additional bursts when using the "non-temporal" memory operations (_mm_stream_load_si128)? How can I suppress this?
What other ways are there to improve the reading performance?
On a x86 system with native PCI support in the chipset, is a better read performance to be expected?

Low PCI bus read performance despite write-combined

0 Answers