Given the convex quadrilateral Q. The problem is to determine if $\exists$ a rectangle and a camera perspective projection matrix M (3x4), so that Q = M*R.
My question is similar: Mapping Irregular Quadrilateral to a Rectangle but not exactly duplicate.
Every strictly convex quadrilateral (with angles strictly less than $180^\circ$) is the perspective projection of a rectangle.
In the projective plane, identify the two points where opposite sides of the quadrilateral intersect. It is always possible with a projective transformation to move two different points to points that are $90^\circ$ apart on the line at infinity. This transformation takes the entire original quadrilateral to a rectangle.