a) Let
$$ \begin{bmatrix} 3 & 2 & -2 \\ 2 & 3 & -2 \\ -2 & -2 & 5 \\ \end{bmatrix} $$
be a quadratic form. Write explicitly an orthogonal transformation, $O$, which transforms A into a diagonal form $a_1y_1^2 + a_2y_2^2 + a_3y_3^2$ with coefficients $a_i$ >0.
b) Whether such orthogonal transformation exists for $A_1$, $A_2$?
$$ A_1= \begin{bmatrix} 3 & 4 & 3 \\ 4 & 3 & -3 \\ 3 & -3 & 5 \\ \end{bmatrix} $$ and $$ A_2= \begin{bmatrix} 4 & 2 & -2 \\ 2 & 4 & -2 \\ -2 & -2 & 6 \\ \end{bmatrix} $$
If yes find such transform. If not provide the proof.
Edit: My main difficulty for this problem was relating matrices to their quadratic and diagonal forms, though most of this has already been addressed and cleared up in the comments and answers below. Thanks...
For a), you want to find the spectral-decomposition of that matrix. That is, find the eigen-decomposition, but select the eigenvectors to be orthonormal, so that the resulting matrix of eigenvectors gives you the orthogonal matrix $O$.
For b), it suffices to check whether $A_1$ and $A_2$ are positive definite. The easiest approach to use here is Sylvester's criterion.
Because the matrices are symmetric, we can put them into a diagonal form with an orthogonal transformation. However, the $a_i$ (i.e. the eigenvalues) might not all be positive.
Clarification of the question:
When we think of $A$ as a quadratic form, we consider the map $x \mapsto x^TAx$ rather than the usual linear transformation $x \mapsto Ax$.
An important distinction here is that when we transform $A$ as a quadratic form, we get a new matrix $S^TAS$ (for some invertible matrix $S$). When we transform $A$ as a linear transformation, we get a new matrix $S^{-1}AS$ (for some invertible matrix $S$).
If we can choose $S$ to be orthogonal, then $S^T = S^{-1}$, so that we can simultaneously transform it both as a linear map and as a quadratic form.
In this case, we're looking for an orthogonal matrix $O$ so that the matrix $$ OAO^T = OAO^{-1} $$ is a diagonal matrix (consisting of the eigenvalues of $A$).