I found three $5 \times 5$ matrices that fulfil the defining Lie algebra relation of $\mathfrak{so}(3)$:
However, these matrices are not antisymmetric, which implies that when we put them into the exponential map, the corresponding group-element matrices are not orthogonal. This seems strange because $SO(3)$ is defined as a set of orthogonal elements.
Is my explicit representation wrong? Or am I wrong to assume that $SO(3)$ elements have to be orthogonal?
Any link to a reference the displays the five-dimensional representation of $\mathfrak{so}(3)$ would be greatly appreciated.
There is nothing wrong with the representation you found. The eigenvalues of your matrices are, indeed, ±2i,±i,0, the ones required for the "spin-2" quintet of physics.
You may check directly that the quadratic Casimir is an invariant, $$ T_x^2+ T_y^2+T_z^2= -6 ~ 1\!\! 1, $$ as expected, so that it must be equivalent by some similarity transformation (for you to find) to the spherical basis representation linked, as well as the tasteful antisymmetric matrix representation proffered by @user71769 's blog,
\begin{equation} t_y = \begin{pmatrix} 0 & 0 & 0 & -1 & 0 \\ 0 & 0 & \sqrt{3} & 0 & 1 \\ 0 & -\sqrt{3} & 0 & 0 & 0 \\ 1 & 0 & 0 & 0 & 0 \\ 0 & -1 & 0 & 0 & 0 \end{pmatrix}\\ t_z = \begin{pmatrix} 0 & 0 & 0 & 0 & -2 \\ 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & -1 & 0 & 0 & 0 \\ 2 & 0 & 0 & 0 & 0 \end{pmatrix}\\ t_x = \begin{pmatrix} 0 & -1 & 0 & 0 & 0 \\ 1 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & -\sqrt{3} & 0 \\ 0 & 0 & \sqrt{3} & 0 & -1 \\ 0 & 0 & 0 & 1 & 0 \end{pmatrix} ~. \end{equation}
Such basis changes are routine for the triplet representation, and are detailed in Wikipedia, but I know of no pressing applications for your quintet one.
Just because the geometrical definition of SO(3) involves orthogonal matrices, there is no good reason for all representation matrices to be orthogonal.