Let $V=\mathbb{R^3}$. Find an orthonormal basis in which the bilinear form with matrix $A$:
$\begin{pmatrix} 2 & -2 & 0 \\ -2 & 1 & -2 \\ 0 & -2 & 0\end{pmatrix}$
has a canonical diagonal form
Should you do $C^{-1}AC=D$ or $C^tAC=D$? And how would know that the basis is orthonormal. I am confused in general with this type of question as to why sometimes $C^{-1}$ or $C^t$ is used.
In general, when you use $C^{-1}AC$, it means that you are changing basis. For example, to diagonalise a matrix or to exhibit some other property.
However, in case of symetric matrices, one can show that this matrix has real eigenvalues and that there exists an orthonormal basis of eigenvectors of this matrix. When you pass from one orthonormal basis (canonical basis in $\Bbb R^n$) to another (basis of eigenvectors), then the matrix of passage is orthogonal, and hence you can use $C^tAC$ (because $C^t=C^{-1}$).
In order to find if a basis $\{v_j\}$ is orthogonal, just check that scalar products satisfy $$v_i\cdot v_j=\begin{cases}1,&i=j,\\0,&i\ne j.\end{cases}$$