The message "iloveyou" has been encrypted as "zghyomcu". I want to find the key, $A=\begin{bmatrix}a&b\\c&d\end{bmatrix}$.
I know that I must have: $$A \begin{bmatrix}8&14&4&14\\11&21&24&20\end{bmatrix} = \begin{bmatrix}25&7&14&2\\6&24&12&20\end{bmatrix}$$
I tried to find equations, but looking at the first row of the product matrix, I have the equations $8a+11b=25$
$14a+21b=7$
$4a+24b=14$
$14a+20b=2$
There are, however, no such values for $a,b$ that satisfy the above. Any ideas how to proceed? Thanks for any help.
If I name the first plain text matrix $$P_1 = \begin{bmatrix} 8 & 14 \\ 11 & 21 \\\end{bmatrix}$$ and the corresponding ciphertext matrix $$C_1 = \begin{bmatrix} 25 & 7 \\ 6 & 24 \end{bmatrix}$$ we have $$A \cdot P_1 = C_1\tag{1}$$
Now if $P_1$ were invertible, we could write $$A = C_1 \cdot P_1^{-1}$$ and we'd have a unique candidate key matrix $A$.
But $$\det(P_1)= 8 \times 21 - 11 \times 14 = 14$$ which is not invertible modulo $26$.
So we try $$P_2 = \begin{bmatrix} 4 & 14 \\ 24 & 20 \\\end{bmatrix}$$
and $$C_2 = \begin{bmatrix} 14 & 2 \\ 12 & 20 \\\end{bmatrix}$$
and that fails too as $\det(P_2)$ is clearly even so also non-invertible.
You can also take the first columns of $P_1$ and $P_2$ and combine those:
$$P = \begin{bmatrix} 8 & 4 \\ 11 & 24\end{bmatrix}$$
with $\det(P)=18$, too bad.. The other combination of columns also fails on evenness. So a dead end via this route...
Back to $P_1$ again. We can take the cofactor matrix $P_1'$ of $P_1$, which for a general $2 \times 2$ matrix $\begin{bmatrix} a & b \\ c& d\end{bmatrix}$ is given by $\begin{bmatrix} d & -b \\ -c& a\end{bmatrix}$ and which has the property that $$P_1 P'_1 = P'_1 P_1 =\det(P_1)I_2$$ as is easily checked.
As $\det(P_1)=14$ as we saw, and $\gcd(14, 26)=2$ we can divide out the $7$ factor of $14$ by multiplying $P'_1$ by $7^{-1}=15 \pmod{26}$ and so we get
$$P'_1 = \begin{bmatrix} 3 & 24 \\ 17& 16 \end{bmatrix}$$
which is an "almost inverse" and multiply $(1)$ on the right by $P'_1$ to get
$$2A = C_1 P'_1 = \begin{bmatrix} 12 & 10 \\ 10 & 8\end{bmatrix}$$
Each of the entries of $2A$ gives two possible values for the corresponding entry for $A$: $10$ gives $5$ and $5+13=18$ and $12$ gives $6$ or $19$ etc. So we have $2^4$ candidates for $A$ now that can be checked on $A\cdot P_2=C_2$ to see which one is actually correct. Note that an extra restraint is that $\det(A)$ should be invertible modulo 26 (so not even or 13 or 0) (for the decryption matrix to be well-defined), so that should rule out most of the candidates quickly, without an extra matrix multiplication.
Turns out, the easiest $A$
$$A=\begin{bmatrix} 6 & 5 \\ 5 & 4\end{bmatrix} \text{ with } \det(A)=-1 \text{ and so } A^{-1}= \begin{bmatrix} 22 & 5 \\ 5 & 20\end{bmatrix}$$
works, for all given cipher and plain texts.
This shows why I prefer having a Hill cipher over an alphabet of prime size: everything non-zero is invertible and we have a proper field. Key space is larger too: more invertible square matrices to pick from.