What family of matrices can make the dot product of 2 non-orthogonal vectors 0?( Multiplying the same matrix to both vectors)

Question

What family of matrices can make the dot product of 2 non-orthogonal vectors 0?( Multiplying the same matrix to both vectors) For an example, vectors <5,1> and <7,-4>.

Answer 1

Since you want a dot product, we need a symmetric $\;2\times2\;$ matrix, so let it be $\;A=\begin{pmatrix}a&b\\c&d\end{pmatrix}\;$, and then

$$0=(5\;1)A\binom7{\!\!-4}=(5\;1)\binom{7a-4b}{7c-4d}=35a-20b+7c-4d$$

Well, now find your subset in $\;M_2(\Bbb R)\;$ fulfilling your condition...but remember: you need $\;A\;$ to be (1) symmetric (and thus $\;b=c\;$), but also (2) to be positive definite, so both $\;a>0\;,\;\;ad-b^2>0\;$ ...

Answer 2

Any matrix, which maps either of the two vectors to $0$ would obviously do it.

For the other cases, we note that a $2\times2$ matrix is uniquely determined by where it maps two linearly independent vectors, hence it must map $(5,1)^T$ and $(7,-4)^T$ to an orthogonal basis $o_1,o_2$. This means that the matrix corresponds to a change of basis (https://en.wikipedia.org/wiki/Change_of_basis) onto an orthogonal basis $o_1,o_2$. In the case where $o_1=(1,0)^T$ and $o_2=(0,1)^T$ the answer is simply $$\begin{pmatrix} 5 & 7 \\ 1 & -4 \end{pmatrix}^{-1}$$

In general an orthogonal basis is given by $(a\cos(\theta)), a\sin(\theta))^T, (-b\sin(\theta)),b\cos(\theta))^T$ for arbitrary $a,b\in \mathbb{R}\setminus\{0\}$ and $\theta \in [0,2\pi)$, so the change of basis becomes $$\begin{pmatrix} a\cos(\theta) & -b\sin(\theta) \\ a\sin(\theta) & b\cos(\theta)\end{pmatrix}\begin{pmatrix} 5 & 7 \\ 1 & -4 \end{pmatrix}^{-1}$$