I have looked at another page which explains how to rotate a point around the origin for three axis but I don't understand how you do it for just two axis.link
I would like to rotate a point at (x,y,z) around the origin with a rotation around the X-axis and Y-axis. However I would like to go against the right hand rule and define the z-as as being depth, and x and y being the same as on a 2d coordinate plane. Also as Z goes in, the z value increases.
So, the new coordinate would be (x',y',z')
What will the rotation matrix look like, when there is no rotation on the z axis.

An easy way to find what a Transformation does to a set, is to find out what the transformation does to each axis.
Now, let $C=\{e_1,e_2\}$ be the Canonial Basis for $\mathbb{R}^2$. ($e_i$ is the vector with 1 in the $ith$ position and 0 in the rest)
$C=\begin{pmatrix} 1 & 0\\ 0 & 1 \end{pmatrix}$ would be the representation of this.
Now suppose you want to rotate the whole system about $\theta$. The new coordinates of $e_1$ will be: $(cos\theta,sin\theta)$ and given $e_2$ is the orthogonal vector to $e_1$ with both having positive direction, the new coordinates of $e_2$ will be $(-sin\theta,cos\theta)$. (If you have any doubt about how this happens, take a look on the coordinates of a vector over the unitary circumference)
In a more formal way: Let $T:\mathbb{R}^2\longrightarrow\mathbb{R}^2$ the rotation about $\theta$, hence:
$[T]=\begin{pmatrix} cos\theta & -sin\theta\\ sin\theta & cos\theta \end{pmatrix}$,
where $[T]$ is the matrix associated to the transformation T.
Now, a result in Geommetry and Linear Algebra gives us that:
$T(x)=[T][x]$, where $x$ is a vector on the space over where the Transformation is applied.
Therefore, if $x=(x_1,x_2) \in \mathbb{R}^2$:
$T(x)=[T][x]=\begin{pmatrix} cos\theta & -sin\theta\\ sin\theta & cos\theta \end{pmatrix}\begin{pmatrix} x_1\\ x_2 \end{pmatrix}$.
Your question is about a generalization of what was said before:
Let $B=\{e_1,e_2,e_3\}$ be the Canonial Basis for $\mathbb{R}^3$.
$B=\begin{pmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & 0 & 1 \end{pmatrix}$,
and $S:\mathbb{R}^3\longrightarrow\mathbb{R}^3$ the rotation about $\theta$ without $Z$ axis rotating (i.e, rotation only in XY plane), hence:
$[S]=\begin{pmatrix} cos\theta & -sin\theta & 0\\ sin\theta & cos\theta & 0 \\ 0 & 0 & 1 \end{pmatrix}$
and if $y=(y_1,y_2,y_3) \in \mathbb{R}^3$, then
$S(y)=[S][y]=\begin{pmatrix} cos\theta & -sin\theta & 0\\ sin\theta & cos\theta & 0 \\ 0 & 0 & 1 \end{pmatrix}\begin{pmatrix} y_1\\ y_2 \\ y_3 \end{pmatrix}$.
¿can you deduce why is $[S]$ equal to this now that I've given you the details of how this works in $\mathbb{R}^2$?
Hint: think of this as only a rotation in $\mathbb{R}^2$.