I take a picture of a square of known dimension (let's say 1x1 units) with the camera at an unknown angle relative to the plane of the rectangle. (The distance to the rectangle is also unknown, but irrelevant since the dimension of the square is known.) Without performing any automated image procession to "find" the square, (I'll be processing "by hand"), I want to estimate the angle "off axis" that the camera was relative to the plane of the square when the image was acquired. I am ignoring lens-based distortion for the time being. In other words I am assuming the camera has been calibrated or there is no distortion.
The square will take the form of a trapezoid in the resulting image, the dimensions of which will be a function of the angle that the camera was "off axis". If the primary axis of the camera (a line drawn through the lens perpendicular to the photosensor), is perpendicular to the square, the square will remain square. If the angle of the primary axis of the camera is NOT perpendicular to the plane of the square, the square will appear trapezoidal in the resulting image. The dimensions of that trapezoid reveal the angle "off axis".
What is the math involved once I have the dimensions of the 'trapezoid'?
You are applying some sort of projection from a 3D world to the 2D screen of a camera.
There are different transformations and camera models (example), I would suggest to consult a good introduction to 3D computer graphics, like Foley-Van Dam.
This will usually result in a transformation matrix using homogeneous coordinates. One can then investigate how a square gets transformed. The goal would be to invert the transformation, to reconstruct the square from the image. Due to the loss of information, this invertation might not be possible in a unique fashion for all geometric configurations.