[knem-devel] how to write INT value with KNEM "atomically"

George Bosilca bosilca at eecs.utk.edu
Tue Sep 27 23:05:11 CEST 2011

On Sep 27, 2011, at 16:38 , Brice Goglin wrote:

> Le 27/09/2011 22:24, George Bosilca a écrit :
>> On Sep 26, 2011, at 07:39 , Brice Goglin wrote:
>>> By the way, would anybody be interested in having a new KNEM copy flag saying:
>>>    "add a memory barrier between iovec writes during a single copy"
>>> The idea would be that if you post a KNEM request to copy buffer A + flag B (two iovecs) at the same time, you have the guarantee that A is done when B gets actually copied.
>> Depend on how you look at this. From a usability point of view, such a flag might be interesting. But from a performance point of view, I have a doubt (you will have to have a second iovec with a very small message to take advantage of such a feature, and this will be costly). Moreover, for small messages, going over knem is too expensive, so most of the time there will be an equivalent mechanism for small messages (usually over shared memory), mechanism that can then be used for synchronization purposes.
> That's true if you copy this very small message as another knem copy
> request. But we're talking about adding a single byte copy to a larger
> copy that you will have anyway. No additional syscall, just some cycles
> to handle the additional iovec loop and copy one more byte.
> I am not sure I've ever benchmarked the overhead of multiple iovecs in
> KNEM. I should look at this one day or another.
>> Now from a practical point of view, isn't the kernel supposed to flush the memory before coming back from an interrupt? If this is the case, then there is an automatic memory flush after the memcpy done in knem …
> If the remote-side is writing on your own memory, you can't know for
> sure if it already came back from the syscall when you see the data it
> wrote.

This was always the most unsafe way of remote notification. My point was that even for a put, you need some completion to be able to remove the knem registration. This is done through a syscall, which has to ensure that the memory is flushed before.

I mean a similar issue can happened to a normal read from a file. If the effective read is done on another thread that the one calling the syscall, just returning from the syscall doesn't means your process memory has been updated yet … The memory has to be flushed by the kernel before returning from a syscall…


> If you're reading from the remote side into your memory (and not
> in async mode), there is likely no need for a flush indeed.
> IIRC, the original question was about adding a notification flag during
> a put. It might be the only case where all this can be interesting.
> Brice
> _______________________________________________
> knem-devel mailing list
> knem-devel at lists.gforge.inria.fr
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/knem-devel

More information about the knem-devel mailing list