I'm trying to rotate 3 points around x- axis.
and three points are (1.33, 2.49, 0),(2.5, 4.33, 0),(0, 5, 0). which is on the xy plane
rotation angle is pi/4. then, I followed this cacultation.

but I think z position should not be zero and x position should be fixed as it rotated around x-axis, right? but the result is different. Could you give me any comment about this? Is there anything I missing now?
Are you trying to do three points at once? That's not how matrix multiplication works. You appear to have a rotation matrix given by
$$M=\begin{bmatrix}1&0&0\\0&\sqrt2&-\sqrt2\\0&\sqrt2&\sqrt2\end{bmatrix}$$ and you want to use that to rotate your point. So do your matrix multiplication on the first point: $$\begin{bmatrix}1&0&0\\0&\sqrt2&-\sqrt2\\0&\sqrt2&\sqrt2\end{bmatrix} \begin{bmatrix}1.33\\2.49\\0\end{bmatrix} = \begin{bmatrix}1\times1.33+0\times2.49+0\times2.49\\ 0\times1.33+\sqrt2\times2.49-\sqrt2\times0\\ 0\times1.33+\sqrt2\times2.49+\sqrt2\times0\end{bmatrix}$$ by taking each member of a row times each member of a column and adding. Then you can do the same for the other points.
EDIT based on comment below: And if you do want to do them all at once, you need to put each point as a column in your right-hand matrix, not a row. Apologies for the confusion.