For an arbitrary polygon, I want to be able to calculate how "rectangular" it is.
Does any work exist which has tried to do this?
I found An Efficiently Computable Metric for Comparing Polygonal Shapes (Arkin et al., 1991), which develops a metric for comparing arbitrary polygons. So one way to do what I want might be to use this metric and compare the polygon to some rectangle.
One way to do it could be to determine the minimum bounding rectangle of the given polygon, then use the "coverage factor" (ratio between the area of the polygon and area of its bounding rectangle) as a measure of similarity. This will be a percentage varying from $0\%$ for a thin star-shaped polygon to $100\%$ for an actual rectangle. For convex polygons, the lower bound is $50\%$ (e.g. triangles) as discussed in Area ratio of a minimum bounding rectangle of a convex polygon on MO. Some links to relevant algorithms can be found at Finding minimum-area-rectangle for given points.