I have a point $P$ defined by ($x_1$, $y_1$) and a rectangle $R$ that is defined by a point $M$ ($x_2$, $y_2$), height($h$) and width($w$).
I know I can easily find out if the point is inside it if it's axis-aligned, by calculating minimum and maximum $x$ and $x$ of $R$ and comparing them with the $x$ and $y$ of $P$, however it doesn't work when $R$ is rotated.
Is there a way to somehow transform $P$ without transforming $R$, so I can then compare them as if $R$ is axis-aligned?
I saw a lot of methods where they calculate the points of the rectangle and then apply matrix transformations to them, but that's very computationally expensive. Is there a "minimally invasive" method, like transforming only $P$ to "fit" the rotation of $R$?
I tried using the transformation matrix $M_1$ that represents the position, rotation, scale of $R$ and inverting it:
$M_2 = Inverse(M_1)$
And I transform $P$ with $M_2$, however it doesn't check out. It still only works when $R$ is axis-aligned, not when it rotates.
Is there any other way?
EDIT:
The rectangle $R$ can also be defined by a 4x4 matrix(the $M_1$ matrix) It's the transform matrix that determines the position, rotation and scale of the rectangle, so $M$ is basically the position component of the matrix $M_1$. I'm transforming $P$ by the inverse of $M_1$ and then trying to treat $R$ as just axis aligned, but it's not working.
EDIT2: Here's a concrete example(some of the variables have changed, I had missed to add something):
I have a point $P$ in 3D space and a rectangle $R$ that has its position, scale and rotation represented by the transform matrix $M_1$.
I need to transform $P$ into the "frame" of $R$, so I can then treat them as a 2D point and a 2D axis-aligned rectagle on a 2D plane and perform a simple test to see if the $P$ is inside $R$. So far I've been transforming $P$ by $M_1^-1$, which seems to only work when $R$ is rotated on the $z$ axis(if $M_1$ has a rotation on the $z$ axis). If $R$ is however tilted(rotated on the $x$ or $y$ axis), for some reason the tests of $P$ being in $R$ don't work.
EDIT3:
$R$ center position $x y z$: $(0, 0, 0)$ $R$ rotation angles on the $x y z$ axes: $(30, 30, 30)$ $R$ transformation matrix:
$M_1$ $$ \begin{matrix} 0.875 & 0.433 & -0.216 & 0 \\ -0.216 & 0.75 & 0.625 & 0 \\ 0.433 & -0.5 & 0.75 & 0 \\ 0 & 0 & 0 & 1 \\ \end{matrix} $$
$R$ inverse transform (approximated):
$M_1^{-1}$ $$ \begin{matrix} 0.875 & -0.216 & 0.433 & 0 \\ 0.433 & 0.75 & -0.5 & 0 \\ -0.216 & 0.625 & 0.75 & 0 \\ 0 & 0 & 0 & 1 \\ \end{matrix} $$
point $P$ $xyz$ before transformation: $(0.81, -1.61, 0)$
point $P$ $xyz$ after transformation: $(0.014, -1.38, 1.16)$
I need point $P$ to be as if it's aligned with the transform of $R$, as if it's only 2D and to be able to use the simple 2D algorithm to check if $P$ is inside $R$
Your solution should work, but assuming that the matrix $M_1$ means what it usually means, your rectangle consists of all point $Mu$, where $u = \begin{bmatrix}x\\y\\0\\1 \end{bmatrix}$, with $0 \le x \le 1$.
In that case, letting $M_2 = M_1^{-1}$, you can say that a point $P$ (at least a point $P$ that's in the right plane!) is in the rectangle if
$$v = M_2 P$$ is a point in the unit square (not just in any axis-aligned rectangle).
If you're really sure this doesn't work for you, tell us an explicit example of the matrices $M_1$ and $M_2$ and a point $P$, and perhaps we can help you out.