[Simgrid-user] simulating a map-reduce system

jdd dhok jddhok at gmail.com
Wed Nov 12 20:43:53 CET 2008


Hello all,
I am trying to simulate a map-reduce system as described in the
original Google
paper(http://www.google.co.in/url?sa=t&source=web&ct=res&cd=2&url=http%3A%2F%2Flabs.google.com%2Fpapers%2Fmapreduce-osdi04.pdf&ei=zzEbSa70NYvO6gP59LjEDg&usg=AFQjCNGR2uEfpCUiHvw6I876kTPeiv-mUA&sig2=WBwLevKyUyDK8uhfKmb7sA)
using SimGrid. Its not clear to me how the following can be done using
SimGrid. I couldn't find any API for it, and searching the list
archives didn't help either.
Basically at the minimum I want to do this
1. Have a catalogue of files (file -> replica locations) at a central location.
2. Start a map task for each file on the node that contains the file.
3. Collect output of map tasks into files on a subset of the entire
cluster, and start reduce tasks for each of these files.
4. Simulate node failures
5. Create nodes with more than one processors (multicore machines)

I have never used SimGrid before, so any pointers in getting started
will be really helpful. Another doubt is that which framework should I
use (MSG/GRAS/SURF)?

Thanks,
-- 
- JDD




More information about the Simgrid-user mailing list