Show that the matrix $$A = \begin{bmatrix}a&h\\h&b\end{bmatrix} ,\quad a \ne b$$ is transformed to diagonal matrix $D = P^{-1}AP$, where $$P = \begin{bmatrix}\cos x& -\sin x\\\sin x& \cos x\end{bmatrix}$$ and $$\tan2x=\frac{2h}{(a-b)}$$
I understand that a $n \times n$ matrix $A$ is diagonalizable if there is a diagonal matrix $D$ such that $A$ is similar to $D$, that is, if there is an invertible matrix $P$ such that $P^{-1}AP= D$. Also, columns of $P$ are $n $ linearly independent eigenvectors of $A$ and the diagonal entries of $D$ are the eigenvalues of $A$ corresponding to the eigenvectors in $P$ in the same order. I am unable to proceed solving the above problem with these leads.
You seem to know the theory behind the problem. So there is no reason you cannot find the eigenvalues of $A$ and the corresponding eigenvectors required for the diagonalisation. Note that since $A$ is symmetric, it is orthogonally diagonalisable.
To do this in a slightly different setting but essentially the same method, consider the quadratic form
$$Q(X,Y)=(X\quad Y)\begin{pmatrix}a&h\\h&b\end{pmatrix}\begin{pmatrix}X\\Y\end{pmatrix}=aX^2+2hXY+bY^2\quad,\,\,a\ne b$$
Let $x$ be the angle by which the coordinate axes $X,Y$ should be rotated about the origin so that $Q$ is transformed to another quadratic form in which the product term is absent. The vanishing of the product term is equivalent to diagonalising the matrix $A$ or the quadratic form $Q$.
Suppose $u,v$ is our set of coordinate axes.
The required transformation is given by
$$\begin{pmatrix}u\\v\end{pmatrix}=\begin{pmatrix}\cos x&-\sin x\\\sin x&\cos x\end{pmatrix}\begin{pmatrix}X\\Y\end{pmatrix}$$
Or, $$\begin{pmatrix}X\\Y\end{pmatrix}=\begin{pmatrix}\cos x&\sin x\\-\sin x&\cos x\end{pmatrix}\begin{pmatrix}u\\v\end{pmatrix}$$
So, $$Q(X,Y)=Q'(u,v)=Au^2+2Huv+Bv^2$$
, where
$$A=a\cos^2x+2h\sin x\cos x+b\sin^2 x,$$
$$B=a\sin^2x-2h\sin x\cos x+b\cos^2 x,$$
$$H=(b-a)\sin x\cos x+h(\cos^2x-\sin^2 x)$$
The product term $uv$ in $Q'$ vanishes iff
\begin{align} H=0&\implies(b-a)\sin x\cos x+h(\cos^2 x-\sin^2 x)=0 \\&\implies (b-a)\sin 2x+2h\cos 2x=0 \\&\implies \tan 2x=\frac{2h}{a-b} \end{align}
Thus using the rotation matrix $P$, we have transformed the matrix $A$ to a diagonal matrix $D$ which is nothing but the matrix associated with the quadratic form $Au^2+Bv^2$.