[Simgrid-user] [ANN] Hubble, a simulator of the Hydra/Nix build system

Ludovic Courtès ludovic.courtes at inria.fr
Mon Sep 13 18:36:39 CEST 2010


I’ve been using SimGrid in Hubble, a simulator of the Hydra/Nix build
tools [0], as part of my work in the Cépage team at INRIA Bordeaux
Sud-Ouest [1].

Nix is a build tool that forces users to provide a complete description
of the dependencies required for a given build action.  It is used
notably to build a GNU/Linux distribution, where the dependencies are
described down to the compiler’s compiler.

Hydra is a continuous build tool built on top of Nix.  It is currently
deployed on a homogeneous cluster at TU Delft, which can optionally use
an Amazon EC2 instance [1].

Hubble aims to simulate scheduling algorithms of build tasks with
Hydra/Nix using SimGrid.  Its input is a large DAG of build tasks
(currently 2,619 nodes and 17,328 edges) along with their execution time
and the size of the build output, collected on the TU Delft cluster.

We hope to use it mainly to:

  * Compare the efficiency of several scheduling algorithms in terms of
    total build time and/or amount of data transferred.

  * Compare these algorithms on different platforms, ranging from small
    homogeneous clusters to large heterogeneous nodes.

  * Devise specific scheduling heuristics, if the need arises.

Hubble is written in Guile Scheme [3] and uses the SimDAG API.  It
implements a dynamic greedy scheduling algorithm.  It simulates the “Nix
store”, which is a build cache local to each compute node.
Communication tasks are created on-the-fly when the output of a build
task is needed and not available in the local cache.

The greedy scheduling algorithm is parametrized by a function to select
the next task to schedule among a set of ready tasks.  There are
currently two options available: select the first ready task, or select
the ready task that has the highest HEFT upward rank.  We plan to
experiment with other heuristics.

Hubble is available under the GPLv3 at:


Comments welcome!


[0] http://nixos.org/
[1] http://www.inria.fr/bordeaux/scientific-research/research-teams/CEPAGE_page
[2] The Hydra instance at TU Delft is accessible at <http://hydra.nixos.org/>.
[3] http://www.gnu.org/software/guile/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.gforge.inria.fr/pipermail/simgrid-user/attachments/20100913/c5aff932/attachment.pgp>

More information about the Simgrid-user mailing list