This question was asked by a work colleague of mine, but my days as a mathematician are long gone unfortunately. It does sound like a pretty basic geometry problem to me, doesn't it?
I'm not expecting an extremly detailed answer here, but does anyone have any resources on this? I'm pretty sure this was already answered somewhere.
Please let me know if the question is formulated too vaguely. Thanks a lot in advance!
For clarification: I imagine that the original image looks something like this:
And then it gets rotated like this:
Only the red pixels are considered the "original pixels" while the white pixels are not. So the question would be how far do I have to zoom into the second picture in order to only see red pixels again?


The figure above shows the original image in the standard orientation with its edges parallel to the $x$ and $y$ axis, and then it shows the same image but rotated by its center by an angle $\theta$.
What you need to do is find the intersection points between the diagonals and the rotated edges of the image, and choose the one that is closest to the center of the image to construct your zoom rectangle (the blue rectangle) in the figure above.
If the image center is the origin, and the image extends over the rectangle $[-a, a] \times [-b, b ]$, then the equation of the right edge is $ x = a $, and the equation of the top edge is $ y = b$. When rotating the image by an angle $\theta$ counter clockwise, then the point $(x, y)$ is mapped to $(x',y') = R (x,y) $
where
$ R = \begin{bmatrix} \cos \theta && - \sin \theta \\ \sin \theta && \cos \theta \end{bmatrix} $
It follows that $(x, y) = R^T (x', y')$. Therefore, the equation of the right edge is $R^T (x', y') = a$, which is
$ \cos(\theta) x' + \sin(\theta) y' = a $
Similarly the equation of the rotated top edge is
$ -\sin(\theta) x' + \cos(\theta) y' = b $
Now the equations of the red diagonals are $y' = \pm \dfrac{b}{a} x' $
Intersect these diagonals with the two rotated edges to obtain $(x_1, y_1)$ and $(x_2, y_2)$. Choose the point that closer to the origin, i.e. having the smaller $\sqrt{ x_i^2 + y_i^2 } $, and call this point $(x_0, y_0)$
Now the zoom rectangle (shown in blue) extands over $[- |x_0|, |x_0| ] \times [-|y_0| , |y_0| ] $
The zoom factor is the following ratio
$\text{Zoom Factor} = \dfrac{a}{|x_0| } = \dfrac{b}{| y_0 | } $