[knem-devel] how to write INT value with KNEM "atomically"

George Bosilca bosilca at eecs.utk.edu
Tue Sep 27 22:24:29 CEST 2011


On Sep 26, 2011, at 07:39 , Brice Goglin wrote:

> By the way, would anybody be interested in having a new KNEM copy flag saying:
>     "add a memory barrier between iovec writes during a single copy"
> The idea would be that if you post a KNEM request to copy buffer A + flag B (two iovecs) at the same time, you have the guarantee that A is done when B gets actually copied.

Depend on how you look at this. From a usability point of view, such a flag might be interesting. But from a performance point of view, I have a doubt (you will have to have a second iovec with a very small message to take advantage of such a feature, and this will be costly). Moreover, for small messages, going over knem is too expensive, so most of the time there will be an equivalent mechanism for small messages (usually over shared memory), mechanism that can then be used for synchronization purposes.

Now from a practical point of view, isn't the kernel supposed to flush the memory before coming back from an interrupt? If this is the case, then there is an automatic memory flush after the memcpy done in knem …

  george.

> 
> Brice
> 
> 
> 
> 
> Le 26/09/2011 13:35, George Bosilca a écrit :
>> Valentin,
>> 
>> You can extend your knem-based communication framework with a shared memory region. You do the large data transfers with knem, everything else will go through the shared memory region (coordination messages as well as short data transfers). You can order the memory writes using the memory flush instructions, and this work globally, including the areas copied using knem.
>> 
>>   George.
>> 
>> 
>> 
>> On Sep 26, 2011, at 5:04, valentin <valentin.petrov at itseez.com> wrote:
>> 
>>> Ok. In fact, that's the answer i'd expect. Nevertheless, thanks a lot for you help!
>>> 
>>> 26.09.2011 12:31, Brice Goglin пишет:
>>>> Hello Valentin,
>>>> KNEM writes data using memcpy-like functions. The way those functions actually move bytes depend a lot on the kernel version, architecture and whether we're in the beginning/middle/end of the actual copy. It may copy from 1 to 16bytes at once, and maybe 32bytes with AVX. So it's very hard to us to have any idea of what's going on underneath (without loosing performance). I am not even sure that memcpy is guaranteed to copy in order (endianness would also be a problem for INTs anyway). Memory barriers may be needed. So I am afraid I cannot help you much here.
>>>> 
>>>> You could ask KNEM to write your INT + a special flag byte for notification. When using vectorial memory regions, KNEM writes iovecs in order, but again I am not sure whether memory barriers may be needed to guarantee that you're target process will actually see those writes in order.
>>>> 
>>>> Brice
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> Le 26/09/2011 10:17, valentin a écrit :
>>>>> Hello All,
>>>>> i'm trying to use KNEM to perform a one-sided put operation. A process A puts an integer value to Process B. Process B waits actively while the int value at the appropriate address is changed and uses this value after that. My problem is that KNEM writes data by bytes (as far as i see it now). So the Process B continues to work (leaves active wait) once the first byte in the int variable is changed, despite the fact that the whole new value has not been entirely written  yet. So, is there some way to enable KNEM to perform such kind of task, i.e. to write data corresponding to the predefined types (like INT) "atomically"?
>>>>> I would greatly appreciate any help!
>>>>> 
>>>>> -- 
>>>>> BR, Valentin Petrov
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> knem-devel mailing list
>>>>> 
>>>>> knem-devel at lists.gforge.inria.fr
>>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/knem-devel
>>>> 
>>> 
>>> -- 
>>> BR, Valentin Petrov
>>> 
>>> _______________________________________________
>>> knem-devel mailing list
>>> knem-devel at lists.gforge.inria.fr
>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/knem-devel
> 




More information about the knem-devel mailing list