Název projektu:     Ever Been (Continuous Benchmarking Environment Development)
Vedoucí projektu:   Andrej Podzimek <podzimek@d3s.mff.cuni.cz>

Počet řešitelů:     4
Termín dokončení:   léto 2011


1. Overview

The Been framework automatically executes software performance measurements
in a heterogeneous networked environment.  The basic architecture of the
Been framework consists of a host runtime capable of executing arbitrary
tasks, a task manager that relies on the host runtime to distribute and
execute scheduled sequences of tasks, and a benchmark manager that creates
the sequences of tasks to execute and measure benchmarks.  Other components
include a software repository, a results repository, and a graphical user
interface.

The Been framework has been developed as a part of a student project
between 2004-2006, and substantially modified as a part of another student
project between 2009-2010.

For a detailed overview of the Been framework, see a paper describing its
main components [1] or the homepage of the Been framework [2].


2. Project Goals (Student Perspective)

The overall goal of this project is to modify the Been framework to
facilitate truly continuous execution. The defining features of the
project are:

 - Code review and code refactoring rather than coding from zero.

   This means the team members will get to experience a more
   practical kind of coding work than starting from scratch.
   Having to read and refactor existing code, as well as to
   add new features, is a practically necessary experience.

 - Agile development with frequent internal releases.

   The team members will be expected to keep the Been framework
   in mostly working shape throughout the project. This has the
   benefit of seeing the effect of changes immediately, which,
   among other things, eases debugging.

 - Emphasis on delivering usable code.

   There are clear applications that the Been framework should be
   able to handle at the end of the project. The team will thus
   have a clear feedback on whether their output is good enough.


3. Project Goals (Framework Perspective)

The overall goal of this project is to modify the Been framework to
facilitate truly continuous execution. In particular, this means:

 - Reviewing the code responsible for communication between hosts, setting
   up rules that prevent the communication from creating orphan references
   (and therefore memory leaks), and rules that make the communication
   robust in face of network and host failures.

 - Reviewing the code responsible for logging, setting up rules that
   govern all log storage (and prevent uncontrolled growth of logs).

 - Reviewing the code responsible for temporary data storage, setting
   up rules that enforce reliable temporary data storage cleanup
   while preserving enough data for post mortem inspection of
   failed tasks and hosts.

 - Reviewing the code responsible for measurement result storage,
   setting up rules for archival and cleanup that would make it
   possible to store recent results in detail and older results
   for overview purposes.

 - Generally clean up any reliability related bugs.


3. Planning

The project is planned as a high intensity task following
the agile software engineering approaches.

The project will likely require significant changes to the
Host Manager and the Task Manager components of Been. Those
components now total around 20000 LoC. Other changes might be
required, but of significantly smaller size per change.


4. References

[1] Been Overview, http://d3s.mff.cuni.cz/publications/download/BEEN_VT06.pdf
[2] Been Homepage, http://been.ow2.org
[3] Kalibera T., Tuma P.: Precise Regression Benchmarking with Random Effects: Improving Mono Benchmark Results,
    http://d3s.mff.cuni.cz/publications/download/KaliberaTuma-PreciseRegressionBenchmarking.pdf