For some experimental and practical reason, I have created a new coordinate system in the form
$$x^\prime_i=T_{ij}x_j$$
where $T_{ij}$ isn't a square matrix. $x_i$ is standard Cartesian coordinates, and $x^\prime_j$ is a point in the new system. I have to mention that the new system's axes are not linearly independent. So the last relation can be written as
$$\left(\matrix{x_0^\prime\\x_1^\prime\\x_2^\prime\\x_3^\prime}\right)=\left( \matrix{T_{11} & T_{12} & T_{13} \\ T_{21} & T_{22} & T_{23} \\ T_{31} & T_{32} & T_{33} \\ T_{41} & T_{42} & T_{43} } \right)\cdot \left(\matrix{x\\y\\z}\right)$$
The matrix $T_{ij}$ is well defined.
What I need is a rotation operator that will transform a point in the primed system, as the standard rotation operator does. So say I have the standard rotation matrix in Cartesian coordinates around the z-axis:
$$R_{ij}= \left( \matrix{\cos{\theta}&-\sin{\theta}&0\\sin{\theta}&\cos{\theta}&0\\0&0&1} \right)$$
So to rotate a point in Cartesian coordinates, we use the standard operator formula:
$$P^\prime_i=R_{ij}P_j$$
where $P_j$ is the point before rotation, and $P^\prime_i$ is the point after rotation.
How can I write this rotation formula for a point in the new coordinates system that uses 4 points? How will the rotation matrix look like? I expect a rotation matrix that is $4\times4$, but I don't know how to derive it. Please help in that.
Thank you for any efforts.
Why not just view $\Bbb R^3$ as a subspace of $\Bbb R^4$ whose last coordinate is always zero?
Without loss of generality, I'll assume the first three rows of your $T$ matrix are linearly independent. Then you can augment it to
$\left(\matrix{x_0^\prime\\x_1^\prime\\x_2^\prime\\x_3^\prime}\right)=\left( \matrix{T_{11} & T_{12} & T_{13}&0 \\ T_{21} & T_{22} & T_{23}&0 \\ T_{31} & T_{32} & T_{33}&0 \\ T_{41} & T_{42} & T_{43}&1 } \right)\cdot \left(\matrix{x\\y\\z\\0}\right)$ and the augmented coordinate transformation is now an invertible matrix.
The rotation on your "three dimensional space" would now be $R(\theta)= \left( \matrix{\cos{\theta}&-\sin{\theta}&0&0\\\sin{\theta}&\cos{\theta}&0&0\\0&0&1&0\\0&0&0&1} \right)$ in the original system of coordinates, and $TR(\theta)T^{-1}$ in the new system.