Název projektu:     Will Been (Future Benchmarking Environment Development)
Vedoucí projektu:   Petr Tůma <petr.tuma<at>dsrg.mff.cuni.cz>

Počet řešitelů:     4
Termín dokončení:   jaro 2009


1. Overview

The Been framework is a student project, successfully completed during the
2004-2006 period, which provides a framework for executing benchmark
experiments in a heterogeneous networked environment.

The basic architecture of the Been framework consists of a host runtime
capable of executing arbitrary tasks, a task manager that relies on the
host runtime to distribute and execute scheduled sequences of tasks, and
a benchmark manager that creates sequences of tasks to perform benchmarks.
Other components include a software repository, a results repository, and
a graphical user interface.

For a detailed overview of the Been framework, see a paper describing
its main components [1] or the homepage of the Been framework [2].


2. Goals

The goal of this project is to extend the Been framework further, namely:

 - To integrate support for writing tasks as scripts.

   The current version of the Been framework requires that tasks are
   implemented as Java classes, which can be cumbersome especially
   for tasks that involve spawning multiple external processes.
   The support for writing tasks as scripts would make it possible
   to implement tasks in some common scripting language, making the
   entire framework more flexible when introducing new tasks.

 - To create an interface that would allow driving the Been framework
   from other places than the user interface, especially from scripts.

   The current version of the Been framework requires that experiments
   are controlled from a web interface, making it difficult to create
   experiments that are not entirely managed by the Been framework.
   The external interface would allow creating such experiments.

 - To create a modular results repository structure that would allow
   plugging in different results repositories, and to create an alternative
   results repository that would integrate with the newly developed Drupal
   Plugin for presenting regression benchmarking results, as a proof of
   concept of the modular results repository structure.

   The current version of the Been framework provides a repository
   that integrates Java and R using legacy libraries that are no
   longer maintained, making it difficult to deploy Been.

 - To perform basic maintenance related to upgrades of packages that Been
   relies on for its functionality, making as many such packages as possible
   strictly optional.

Optional tasks depending on the success of the previous steps include:

 - Adding support for regression benchmarking calculations with additional
   experiment planning to improve regression benchmarking precision.
   The necessary calculations are described in [3].

 - Adding support for other benchmarks, such as DaCapo or SpecJVM.


3. Planning

The project is planned as a high intensity task starting with detailed
requirements specification using the standard software engineering
approaches (UML use cases, formalized requirements), and followed
by an intense pair coding period (one pair focusing on the
scripting features, another pair focusing on the repository).


4. References

[1] Been Overview, http://dsrg.mff.cuni.cz/publications/Submitted_1404_BEEN.pdf
[2] Been Homepage, http://been.ow2.org
[3] Kalibera T., Tuma P.: Precise Regression Benchmarking with Random Effects: Improving Mono Benchmark Results,
    http://dsrg.mff.cuni.cz/publications/KaliberaTuma-PreciseRegressionBenchmarking.pdf