Short description
The purpose of this course is to outfit a
practical systems designer with the tools of the trade. I plan to
cover practical concerns such as basic experiment design, workload
selection, measurement tools, simulation techniques, and perturbation
effects as well as basic statistics and probability theory that
underlie all practical evaluation.
A large part of this course will be an evaluation
project. Projects will be individual. You can propose your own
projects, possibly something from your research domain or another
class you are taking. In addition, I have a set of potential projects
in several areas (compilers, operating systems, networking, web
servers, architecture, memory management etc). Other important parts
are critiques of evaluation in published papers and class
participation.
Topics covered include: system metrics, workload selection, monitoring
techniques, benchmarking, data manipulation and presentation, testing
and comparing data, statistical metrics, factor experiments, designing
and controlling simulations and distribution generation.
Project reports