[knem-devel] The unexpected performance of DMA memcopy with knem

Brice Goglin Brice.Goglin at inria.fr
Mon Jul 8 22:52:30 CEST 2013

Le 08/07/2013 22:29, Zheng Da a écrit :
> Hello,
> On Mon, Jul 8, 2013 at 4:00 PM, Brice Goglin <Brice.Goglin at inria.fr> wrote:
>> Le 08/07/2013 20:57, Zheng Da a écrit :
>>> lstopo doesn't show DMA channels
>> (I assume you don't have PCI devices at all then, DMA channels are
>> supposed to show up as soon as lstopo has PCI support)
> There are PCI devices. Here is the output of lstopo on my machine.

Ah sorry. You need to add --whole-io to get all I/O devices. Only the
usually-useful ones are shown by default.
>>> but cat /sys/class/dma/*/device/numa_node shows all devices are in
>>> NUMA node 0 and 1 (each processor is a NUMA node).
>> Interesting. I wonder why Intel didn't any device in the other sockets.
>> Or it could be the BIOS failing to initialize these devices properly
>> (I've seen some Dell boxes with no IOAT device at all until the BIOS was
>> fixed).
> It's a Dell box. After I enabled IOAT explicitly, I saw 16 DMA
> devices. I don't know if there are other configurations to get 32 DMA
> devices.

I wouldn't be surprised if you had to upgrade your BIOS at some point
(or even fill a bug report to the Dell support). Assuming it's a
Poweredge R820, I see that IOAT support was indeed added to the BIOS
last year (1.1.5), but no interesting change is listed in latest
releases (up to 1.5.0).

IOAT channels are inside the CPU sockets, so there's no reason why two
sockets have them and two others don't. Unfortunately, servers with
E5-4600 processors are rare, so I can't find lspci outputs online to
check how many DMA devices those have.


More information about the knem-devel mailing list