It is well known that the circumcenter of a polygon exists if and only if the polygon is cyclic.
I would like to extend the definition of an circumcenter for noncyclic polygons. Namely, let us define the least squares circumcenter as the point A$(x_0, y_0)$ such that the point A minimizes the sum of the squares of the residuals.
Let us consider the case for a noncyclic quadrilateral with vertices P$(x_1, y_1)$, Q$(x_2, y_2)$, R$(x_3, y_3)$, and S$(x_4, y_4)$. Let us also define the origin O$(0, 0)$.
How would we solve for the point A in this case? I was thinking of using matrices and solving $A^\mathsf{T}A \hat{x} = A^\mathsf{T}b$, although any methods are welcome.
Suppose the polygon has vertices $P_1,\ldots, P_m \in\mathbb{R}^n$. The distance $d_i$ from a point $x$ to $P_i$ is simply $d_i = \lVert{P_i-x}\rVert_2$. We would like to minimize the sum of squares of distances. That is, find $x$ solving: $$ \min_x \sum_{i=1}^{m} d_i^2 = \min_x \sum_{i=1}^{m} \lVert P_i-x\rVert_2^2 $$
Consider the square of the 2-norm of the block matrix of size $mn\times 1$ (tall vector), $$ \begin{bmatrix} P_1-x \\ P_2-x \\ \vdots \\ P_m-x \end{bmatrix} $$
The square of the 2-norm of this matrix is exactly $\sum_i d_i^2$. To see this note that d_i^2 is the sum of squares of the entries in the vector $P_i-x$ so that $\sum_i d_i^2$ is the sum of squares of the entries in $P_i-x$ for all $i$.
We can rewrite this matrix in the form $b-Ax$, $$ \begin{bmatrix} P_1-x \\ P_2-x \\ \vdots \\ P_m-x \end{bmatrix} = \begin{bmatrix} P_1\\ P_2 \\ \vdots \\ P_m \end{bmatrix} - \begin{bmatrix} I \\ I \\ \vdots \\ I \end{bmatrix} x $$ where $I$ is the $n\times n$ identity.
To find $x$ we then solve the least squares problem $\min_x \lVert b-Ax \rVert_2$.
EDIT: to see that this just gives the average of the points, form the normal equations.