I'm working on the following problem in my quantum mechanics course:
Consider the following matrix form of the angular momentum operator for a system with $l=1$: $$\hat{L}_z = \begin{bmatrix}1&0&0\\0&0&0\\0&0&-1\end{bmatrix}.$$ 1) Is this operator Hermetian, unitary, a projection operator?
2) Give the operators $\hat{L}_x$, $\hat{L}_y$ and calculate $\hat{L}^2$ in matrix form.
My solution to part 1: $\hat{L}_z$ is hermitian if and only if $\hat{L}_z=\hat{L}_z^H$. This is clearly the case (real diagonal, zero everywhere else). Unitary if and only if $\hat{L}_z^*\hat{L}_z = \hat{L}_z\hat{L}_z^*=I_3$, but $\hat{L}_z^*\hat{L}_z\neq I_3$ (Middle $0$ remains as is.) A projection operator if and only if $\hat{L}_z^2=\hat{L}_z$ which is also clearly not the case here.
But I'm not sure how I can use this knowledge (Hermitian but not unitary or a projection) to find $\hat{L}_x, \hat{L}_y$.
I assume you are given that $L_x$, $L_y$ and $L_z$ should follow the angular momentum algebra, that is, $$[L_i,L_j]=\mathrm i\epsilon_{ijk}L_k \tag{1}$$ Now from this alone, the $L_x$ and $L_y$ are not uniquely defined.
From $(1)$ we get $L_y=\mathrm i[L_x,L_z]$. Let's for the moment write $$L_x=\pmatrix{a&b&c\\d&e&f\\g&h&j} \tag{2}$$ where by assumption all numbers are real. Then we get: $$L_y = \pmatrix{0 & -\mathrm ib & -2\mathrm i c\\ \mathrm id & 0 & -\mathrm if\\ 2\mathrm ig & \mathrm ih & 0}$$ Now again using $(1)$, we have $L_x = i[L_z, L_y]$, so inserting our previous result, we get $$L_x = \pmatrix{0 & b & 4c\\ d & 0 & f\\ 4g & h & 0} \tag{3}$$ Now comparing eqs. $(2)$ and $(3)$, we find that $a=c=e=g=j=0$. Finally we use $(1)$ again in the instance $[L_x,L_y] = \mathrm iL_z$. This gives $$\pmatrix{\mathrm ibd & 0 & -\mathrm ibf\\ 0 & \mathrm i(-db + fh) & 0\\ \mathrm ihd & 0 & -\mathrm ihf} - \pmatrix{-\mathrm ibd & 0 & -\mathrm ibf\\ 0 & \mathrm i(bd-hf) & 0\\ \mathrm idh & 0 & \mathrm ifh} = \pmatrix{\mathrm i & 0 & 0\\ 0 & 0 & 0 \\ 0 & 0 & -\mathrm i}$$ From this we can conclude that $$bd=hf=\frac12 \tag{4}$$
To go further, we need another condition: Namely that like $L_z$, also $L_x$ and $L_y$ should be Hermitean. Thus $d=b^*$ and $h=f^*$. Then $(4)$ reduces to $|b|^2=|f|^2=\frac12$. Writing $$b = \frac{1}{\sqrt{2}}\mathrm e^{\mathrm i\phi}, f = \frac{1}{\sqrt{2}}\mathrm e^{\mathrm i\chi}$$ we get $$L_x = \frac{1}{\sqrt{2}}\pmatrix{0 & \mathrm e^{\mathrm i\phi} & 0\\ \mathrm e^{-\mathrm i\phi} & 0 & \mathrm e^{\mathrm i\chi} \\ 0 & \mathrm e^{-i\chi} & 0},\quad L_y = \frac{1}{\sqrt{2}}\pmatrix{0 & -\mathrm i\mathrm e^{\mathrm i\phi} & 0\\ \mathrm i\mathrm e^{-\mathrm i\phi} & 0 & -\mathrm i\mathrm e^{-\mathrm i\chi} \\ 0 & \mathrm i\mathrm e^{i\chi} & 0} $$ Now let's calculate $L^2 = L_x^2 + L_y^2 + L_z^2$. This gives $$L^2 = \pmatrix{2 & 0 & 0\\ 0 & 2 & 0\\ 0 & 0 & 2} = 2I$$ where $I$ is the $3\times 3$ unit matrix.
The usual representation is obtained by choosing $\phi=\chi=0$. This results in $L_x$ only having real entries, and $L_y$ only having imaginary entries.
Actually I'm surprised that you apparently cannot derive $\phi=\chi$ (one free angle can be physically explained by the fact that fixing only the $z$ axis still allows rotations in the $x$-$y$ plane; however I don't know what I should make of the second angle).