Can the following matrix $A=\begin{pmatrix} 3 & 1 & -5\\ 0 & 2 & 3\\ 0 & 0 & -1 \end{pmatrix}$ be a matrix of self-adjoint operator (for some euclidean inner product space in $\mathbb{R}^3$)? If yes then find corresponding Gram matrix.
My solution: It is easy to compute that this matrix has $3$ eigenvalues, namely $\{-1,2,3\}$. We can find each corresponding eigenspace $V_{3}=\langle (1,0,0)\rangle,V_{2}=\langle (-1,1,0)\rangle, V_{-1}=\langle (3,-2,2)\rangle$. Hence this matrix is diagonalizable. And we have to find such inner product in $\mathbb{R}^3$ such that those eigenvectors are orthogonal. More precisely, the function $$\langle \cdot,\cdot\rangle:\mathbb{R}^3\times\mathbb{R}^3\to \mathbb{R}, \quad \langle x,y\rangle=(x_1,x_2,x_3)A\begin{pmatrix} y_1 \\ y_2\\ y_3 \end{pmatrix}$$ where $A$ should be symmetric and positive definite, i.e. $A=\begin{pmatrix} a & b & c\\ b & d & e\\ c & e & f \end{pmatrix}$. Let''s denote vectors $(1,0,0), (-1,1,0), (3,-2,2)$ by $a_1,a_2,a_3$, respectively. Since we want $a_i\perp a_j$ for $i\neq j$.
Then condition $a_1\perp a_2$ implies that $b=a$. Condition $a_1\perp a_3$ implies $c=-\frac{a}{2}$. And condition $a_2\perp a_3$ implies that $e=d-\frac{3a}{2}$. So our matrix looks like $$A=\begin{pmatrix} a & a & -\frac{a}{2}\\ a & d & d-\frac{3a}{2}\\ -\frac{a}{2} & d-\frac{3a}{2} & f \end{pmatrix}.$$ Since $A$ is positive definite then all upper-left submatrices should have positive determinants, i.e. it means that $$a>0, ad-a^2>0, \det A=\frac{a}{2}(d-a)(2f-2d+\frac{3a}{2})>0$$ which is equivalent to $$a>0, d>a,f>d-\frac{3a}{4}.$$
In particular If I take $a=2, d=4$ then $f>\frac{5}{2}$ and one can take $f=4$. So It means that Gram matrix can be taken as $$\begin{pmatrix} 2 & 2 & -1\\ 2 & 4 & 1\\ -1 & 1 & 4 \end{pmatrix}$$ which makes our operator to be self-adjoint.
As you’ve written, you’re looking for an inner product for which your eigenvectors are orthogonal. Strengthening this to having them be an orthonormal set, if $P$ is the matrix of eigenvectors, then this is equivalent to finding a symmetric matrix $G$ such that $P^TGP$ is diagonal, with positive diagonal entries. A simple choice for this diagonal matrix is the identity, and multiplying by inverses on both sides yields $G=P^{-T}P^{-1}$. Other choices for the diagonal matrix amount to scaling the eigenvectors.
For your choice of eigenvectors, this produces $$G=\begin{bmatrix}1&1&-\frac12\\1&2&\frac12\\-\frac12&\frac12&\frac32\end{bmatrix}.$$ For the $G$ that you came up with, we have $P^TGP=\operatorname{diag}(2,2,6)$.