[knem-devel] how to write INT value with KNEM "atomically"

Brice Goglin Brice.Goglin at inria.fr
Tue Sep 27 22:38:01 CEST 2011

Le 27/09/2011 22:24, George Bosilca a écrit :
> On Sep 26, 2011, at 07:39 , Brice Goglin wrote:
>> By the way, would anybody be interested in having a new KNEM copy flag saying:
>>     "add a memory barrier between iovec writes during a single copy"
>> The idea would be that if you post a KNEM request to copy buffer A + flag B (two iovecs) at the same time, you have the guarantee that A is done when B gets actually copied.
> Depend on how you look at this. From a usability point of view, such a flag might be interesting. But from a performance point of view, I have a doubt (you will have to have a second iovec with a very small message to take advantage of such a feature, and this will be costly). Moreover, for small messages, going over knem is too expensive, so most of the time there will be an equivalent mechanism for small messages (usually over shared memory), mechanism that can then be used for synchronization purposes.

That's true if you copy this very small message as another knem copy
request. But we're talking about adding a single byte copy to a larger
copy that you will have anyway. No additional syscall, just some cycles
to handle the additional iovec loop and copy one more byte.

I am not sure I've ever benchmarked the overhead of multiple iovecs in
KNEM. I should look at this one day or another.

> Now from a practical point of view, isn't the kernel supposed to flush the memory before coming back from an interrupt? If this is the case, then there is an automatic memory flush after the memcpy done in knem …

If the remote-side is writing on your own memory, you can't know for
sure if it already came back from the syscall when you see the data it
wrote. If you're reading from the remote side into your memory (and not
in async mode), there is likely no need for a flush indeed.

IIRC, the original question was about adding a notification flag during
a put. It might be the only case where all this can be interesting.


More information about the knem-devel mailing list