I am writing a bilinear interpolation method.
This method can be abstract by solve the equation $Ax = b$, $A$ is a $4\times 4$ matrix below: $$A=\begin{pmatrix} 1 &x_1 &y_1 &x_1y_1\\ 1 & x_2 & y_2 & x_2y_2\\ 1 & x_3 & y_3 & x_3y_3\\ 1 & x_4 & y_4 & x_4y_4\end{pmatrix}$$
Here, $(x_1, y_1), (x_2, y_2), (x_3, y_3)$ and $(x_4, y_4)$ are four points containing the dst interpolation point.
My problem is when $\det(A) = 0$ (then $x \neq A^{-1}b$), what is the quadrangle looks like?
On
Consider the rows of your matrix without the leading $1$. This gives four vectors in $\mathbb{R}^3$, namely $v_1 = (x_1, y_1, x_1 y_1)$, $v_2 = (x_2, y_2, x_2 y_2)$, $v_3 = (x_3, y_3, x_3 y_3)$, and $v_4 = (x_4, y_4, x_4 y_4)$. These vectors are affinely dependent if and only if the determinant of your matrix $A$ is $0$. Put differently, they are all contained in a plane in $\mathbb{R}^3$ if and only if $\det(A) = 0$. Since such a plane cuts the $xy$-coordinate plane in a line (except for the trivial case), all points must be collinear.
The determinant is trivially zero, if the four points are collinear (if $y=mx+n$ then the third row is $m$ times the second plus $n$ times the first column). It is also zero if to points coincide (two equal rows). More generally, $\det A=0$ only if all points are on a curve of the form $$\tag1a+bx+cy+dxy=0$$ with not all of $a,b,c,d$ equal to zero. If $d=0$, we re-obtain the result above about collinerar points; if $d\ne0$, then $(1)$ can be rewritten as $$ \left(x+\frac cd\right)\left(y+\frac bd\right)=\frac{bc-ad}{d^2},$$ that is a hyperbola (or two axe-parallel lines)