[knem-devel] (no subject)

neuerrechner at gmx.de neuerrechner at gmx.de
Sat Feb 5 15:40:29 CET 2011


Hello Brice,
thank you for your explanations. 

> Are you sure that I/OAT DMA is enabled in your hardware? Sometimes it is
> disabled in the BIOS. If it's enabled, you will see something like this
> in lspci:
> 
> 00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset
> QuickData Technology Device (rev 22)
> 00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset
> QuickData Technology Device (rev 22)
> 
> right after these "standard lines":

I guess that indeed I/OAT is not enabled as there are no such lines in lspci. But I could not find a BIOS switch to enable this although I guess the Chipset and CPU should feature it.

> Anyway, I/OAT isn't very efficient compared to a standard memory copy in
> recent processors. So it's probably better to not use it anyway. That's
> why Open MPI doesn't tell KNEM to use I/OAT by default.

Okay, I see, so maybe it is not that important to further investigate why this I/OAT thing is not working.
But the point is, that with large MPI messages, there is still some speedup when using knem? I will make some test with and w/o it...

So thanks again,

Florian


-------- Original-Nachricht --------
> Datum: Fri, 04 Feb 2011 17:43:33 +0100
> Von: Brice Goglin <Brice.Goglin at inria.fr>
> An: knem-devel at lists.gforge.inria.fr
> Betreff: Re: [knem-devel] (no subject)

> Le 04/02/2011 17:15, neuerrechner at gmx.de a écrit :
> > knem 0.9.5
> >  Driver ABI=0xd
> >  Flags: forcing 0x0, ignoring 0x0
> >  DMAEngine: KernelSupported Enabled NoChannelAvailable
> >  Debug: NotBuilt
> >  Requests Submitted          : 12240
> >  Requests Processed/DMA      : 0
> >  Requests Processed/Thread   : 0
> >  Requests Processed/PinLocal : 0
> >  Requests Failed/NoMemory    : 0
> >  Requests Failed/ReadCmd     : 0
> >  Requests Failed/FindRegion  : 0
> >  Requests Failed/Pin         : 0
> >  Requests Failed/MemcpyToUser: 0
> >  Requests Failed/MemcpyPinned: 0
> >  Requests Failed/DMACopy     : 0
> >  Dmacpy Cleanup Timeout      : 0
> >
> > So it seems no requests have been processed. 
> > What could be the problem here and why does it state
> "NoChannelAvailable"? 
> >   
> 
> Hello Florian,
> 
> All your requests have been processed. As long as they don't fail (which
> means they appear in "Failed/something"), all submitted requests are
> processed fine. But there is no counter for requests that are processed
> in the standard manner (no DMA, no offload to a thread, no local
> pinning). I'll see if I can clarify this.
> 
> Are you sure that I/OAT DMA is enabled in your hardware? Sometimes it is
> disabled in the BIOS. If it's enabled, you will see something like this
> in lspci:
> 
> 00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset
> QuickData Technology Device (rev 22)
> 00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset
> QuickData Technology Device (rev 22)
> 
> right after these "standard lines":
> 
> 00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and
> RAS Registers (rev 22)
> 00:14.3 PIC: Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers
> (rev 22)
> 
> Anyway, I/OAT isn't very efficient compared to a standard memory copy in
> recent processors. So it's probably better to not use it anyway. That's
> why Open MPI doesn't tell KNEM to use I/OAT by default.
> 
> Brice
> 
> 
> _______________________________________________
> knem-devel mailing list
> knem-devel at lists.gforge.inria.fr
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/knem-devel

-- 
GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit 
gratis Handy-Flat! http://portal.gmx.net/de/go/dsl



More information about the knem-devel mailing list