Given is the following matrix $$A := \begin{pmatrix} 3x & 4x & 0 \\ 4x & 3x & 0 \\ 0 & 0 & 5x \end{pmatrix} $$ whereas $x \in \mathbb{R}$ is a constant. This matrix can be diagonalized by a rotation of $\alpha$ about the $z$-axis. The problem is to find $\alpha$.
My general solution:
First I have diagonalized $A$. That is I have written it in the form: $$A = S D S^{-1} $$ with $$S = \begin{pmatrix} 1 & -1 & 0 \\ 1 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix}$$ $$D = \begin{pmatrix} 0 & 0 & 5x \\ 0 & -x & 0 \\ 7x & 0 & 0 \end{pmatrix}$$
This is as far as I have come. While $S$ does indeed have the general structure of a rotation matrix about the $z$-axis, it is contradictory in a sense that it requires the existence of angle $\alpha$ that simultaneously needs to satisfy $$\cos\alpha = 1 $$ and $$\sin\alpha = 1 $$
Your matrix is$$x\begin{bmatrix}3&4&0\\4&3&0\\0&0&5\end{bmatrix}.$$So, all you have to do is to diagonalize the matrix$$B=\begin{bmatrix}3&4&0\\4&3&0\\0&0&5\end{bmatrix}.$$Its characteristic polynomial is $-x^3+11 x^2-23 x-35$ and, from the third column of $B$, you know that $5$ is a root of this polynomial. Dividing the characteristic polynomial by $x-5$, what you get is $-x^2+6x+7$, whose roots are $-1$ and $7$. Now, $(1,1,0)$ is an eigenvector of $B$ corresponding to the eigenvalue $7$, and $(-1,1,0)$ is an eigenvector of $B$ corresponding to the eigenvalue $-1$. Of course, $(0,0,1)$ is an eigenvector of $B$ corresponding to the eigenvalue $5$. Dividing these vectors by their normms you get $\frac1{\sqrt2}(1,1,0)$, $\frac1{\sqrt2}(-1,1,0)$, and $(0,0,1)$ respectively. So, consider the matrix$$S=\begin{bmatrix}\frac1{\sqrt2}&-\frac1{\sqrt2}&0\\\frac1{\sqrt2}&\frac1{\sqrt2}&0\\0&0&1\end{bmatrix}$$and you're done:$$S^{-1}BS=\begin{bmatrix}7&0&0\\0&-1&0\\0&0&5\end{bmatrix}.$$