\documentclass[11pt]{article}
\newcommand{\m}[1]{{\bf{#1}}} % for matrices and vectors
\newcommand{\tr}{^{\sf T}} % transpose
\topmargin 0in
\textheight 9in
\oddsidemargin 0pt
\evensidemargin 0pt
\textwidth 6.5in
\begin{document}
\title{CS 219: Sparse matrix algorithms: Homework 4}
\author{Assigned May 1, 2013}
\date{Due by class time Wednesday, May 8}
\maketitle
{\bf Problem 1a.}
Find a 2-by-2 matrix $A$ that is symmetric and nonsingular,
but for which neither $A$ nor $-A$ is positive definite.
What are the eigenvalues of $A$?
Find a 2-vector $y$ such that $y^TAy < 0$.
\par\bigskip
{\bf Problem 1b.}
For $A$ as above,
find a 2-vector $b$ such that the conjugate gradient algorithm,
when started with the zero vector as an initial guess,
does not converge to the solution of $Ax=b$.
Show what happens on the first two iterations of CG,
as in the October 28 class slides.
How do you know it won't converge to the right answer?
\par\bigskip
{\bf Problem 2.}
In this problem you'll actually prove that CG works in at most $n$ steps,
assuming that real numbers are represented exactly.
(This is not a realistic assumption in floating-point arithmetic,
or on any computer with a finite amount of hardware,
but it gives a solid theoretical underpinning to CG.)
Let $A$ be an $n$-by-$n$ symmetric, positive definite matrix,
and let $b$ be an $n$-vector.
We start with the idea of searching through $n$-dimensional
space for the value of $x$ that minimizes $f(x)={1\over2}x^TAx-b^Tx$,
which is the $x$ that satisfies $Ax=b$.
We begin by picking a set of $n$ linearly independent search
directions, called $d_0$, $d_1$, $\ldots,$ $d_{n-1}$.
(Actually we don't know them in advance, but that's a detail.)
At each iteration we proceed along the next direction until
we are ``lined up'' with the final answer,
the value of $x$ at which $Ax=b$.
In $n$-space, once we are lined up with the answer from $n$
independent directions, we will be exactly on the answer.
The {\bf first magic of CG} is that for the right kind of
search directions, there is a way to define ``lined up''
for which we can actually compute how far to go along each
search direction.
The key definition uses {\em $A$-conjugate} vectors.
Then ``lined up'' means that the {\em error} $e_i=x_i-x$ is
exactly crossways to the search direction $d_{i-1}$,
not in the sense of being perpendicular
(which would mean $e_i^Td_{i-1}=0$),
but in the sense of being $A$-conjugate: $e_i^TAd_{i-1}=0$.
An informal way to say that is, we proceed along the search direction
until we are lined up with the solution as seen through $A$-glasses.
The reason for lining up through $A$-glasses rather than bare eyes
is that we can compute where to stop without knowing where the
final answer is.
We can't see and compute with $x$-space directly,
but we can see the space where $Ax$ and $b$ live.
And after lining up each of $n$ independent directions
in an $n$-dimensional space we are guaranteed to be sitting on
top of the right answer, whether the independent directions
are the conventional coordinate axes or the $A$-conjugate axes
we see through our $A$-glasses.
To go along with this, we need to choose the search directions
themselves to be mutually $A$-conjugate:
we will require each $d_i$ to be $A$-conjugate to all the earlier
$d_j$'s, so $d_i^TAd_j=0$ if $i\ne j$.
\par\bigskip
{\bf 2(a)}
Suppose we are given $i$ mutually $A$-conjugate vectors
$d_0$, \dots, $d_{i-1}$.
Suppose $x_0=0$,
and for each $j