How to calculate new coordinate of a 2d triangle rotated by Q degrees?
We confused that
x = old X - center of mass X
y = old Y - center of mass Y
x = x * cos(Q) - y * sin(Q)
y = x * sin(Q) + y * cos(Q)
Here x which is multiplicated sin is confused us. Will x in second calculation be old X cordinate or new x cordinate?
$(X, Y)$ are the coordinates of points of the original triangle in some arbitrary coordinate system.
$(x, y)$ are the coordinates of the points of the original triangle about the centroid (called somewhat inaccurately the "center of mass" here - they are only the same in the presence of a constant density). That is, the origin of the coordinate system for $(x, y)$ is at the centroid. Apparently you are rotating your triangle about the centroid (please don't leave important details off of your question).
What is confusing you is that your equations are also using $(x, y)$ to denote the coordinates after the rotation. And that is just plain REALLY BAD notation, so it is no wonder that you find it confusing. Instead, define:
$(x', y')$ are the coordinates of the point after rotation.
Then the equations can be accurately expressed as:
$$x' = x\cos Q - y\sin Q$$ $$y' = x \sin Q + y \cos Q$$
As you can now see, the $x$ and $y$ on the left-hand side are different values than the $x$ and $y$ on the right-hand side (and thus should be denoted differently).