Given: a triangle ABC, with A=(1,1), B=(3,6), C=(6,2). Points M, N, are respectively on the sides AB and BC.
Find: Coordinates of points P and Q on AC, such that MNPQ is a square (figure below) using vector methods.
Asked in a math contest. Hints and full solutions are welcomed.
This is not the quickest way, but its uses general methods that could be used elsewhere.
Note that if you know $P$, $M$, $N$ or $Q$, you automatically knows the three other vertices by going along orthogonal or parallel lines to the triangle edges. Orthogonal lines are rather easy to find in $\mathbb R^2$ : the orthogonal vector of $(a,b)$ is $(-b,a)$ (or $(b,-a)$). For instance by interesecting the line orthogonal to $[AC]$ starting at $P$ with line segment $[AB]$, you can find $M$. Likewise you can find $N$ by intersection $[BC]$ with the line from $M$ going in the same direction than $[AC]$. And yet with the same trick, you can find $Q$.
By using a test function, you can evaluate whether $P$, $M$, $N$ and $Q$ are the vertices of a square. Squares have orthogonal diagonal of same length that cuts in their middle. So you must check that (where $||.||_2^2$ is the squared euclidean norm and $\langle.|.\rangle$ the scalar product) :
\begin{align} \frac{M+Q}2 &= \frac{N+P}2\\ ||Q-M||_2^2 &= ||P - N||_2^2\\ \langle Q-M | P -N \rangle &= 0 \end{align} I.e. you must find a root to $$ f : \left(\mathbb R^2\right)^4 \to \mathbb R : x \mapsto \begin{cases} \frac{M+Q}2 - \frac{N+P}2\\ ||Q-M||_2^2 - ||P - N||_2^2\\ \langle Q-M | P -N \rangle \end{cases} $$
Now let $P$ travel between $A$ and $C$, that is $P = t \cdot A + (1-t) \cdot C$. Now you have constructed linearly $P$, $M$, $N$ and $Q$ depending on $t \in [0,1]$. Hence by replacing $P$, $M$, $N$ and $Q$ by their expression in $t$ in the system above, you now have a giant second-degree equation system. Solve it to find $t$ hence $P$ and $Q$.
Note: You are already guaranteed to have a rectangle thanks to the method of construction of $P$, $M$, $N$ and $Q$. Therefore you only need to check the diagonals are orthogonal and solve $\langle Q-M | P -N \rangle = 0$.
Note2: Since $P$, $M$, $N$ and $Q$ is a rectangle, instead of checking whether the diangonals are orthogonal, you could simply check whether two adjacent sides have same length. That it, you could solve $||Q - P||_2^2 = ||M - P||_2^2$ to find $t$.