Let $W$ and $V$ be subspaces of $\mathbb{R}^n$ with dimensions $m$ and $p$ respectively. I want to find the unit vector in $W$ whose projection onto $V$ has the minimum Euclidean norm. From geometric intuition I gather that the solution can be unique up to a sign change (such as where $W$ and $V$ are two planes in $\mathbb{R}^3$), but the solutions could also reside in a a hyperplane (such as when $W$ is a plane and $V$ is a perpendicular line).
I've tried to solve this problem by solving the following constrained optimization. Let $\mathbf{w}_1,...,\mathbf{w}_m$ be a basis for $W$, let $M \in \mathbf{R}^{n x m}$ with $M = [\mathbf{w}_1 \cdots \mathbf{w}_m]$, and let $P \in \mathbb{R}^{nxn}$ and be the projection matrix for $V$. The problem is to find
$argmin_\mathbf{x}$ $||P M \mathbf{x}||^2$ subject to $||M \mathbf{x}||^2 = 1$. To solve this I found the critical points of the Lagrangian.
\begin{align*} \mathcal{L}(\mathbf{x}, \lambda) = ||P M \mathbf{x}||^2 - \lambda (||M\mathbf{x}||^2 - 1) \end{align*}
Which gives me
\begin{align*} \frac{\partial}{\partial \mathbf{x}} \mathcal{L}(\mathbf{x}, \lambda) = \mathbf{x}^T M^T P^T P M - \lambda \mathbf{x}^T M^T M &= 0 \\ M^T P^T P M \mathbf{x} = \lambda M^T M \mathbf{x} \end{align*}
I initially computed this derivative incorrectly. This implies that a solution is an eigenvector of $P^TP$. I'm now going to verify this.
It’s a standard result that a quadratic form attains its minimal value on a unit hyperphere in the direction of the eigenvectors of the quadratic form with minimal eigenvalue. Here, the quadratic form in question is $$\|PM\mathbf x\|^2 = \mathbf x^T(M^TP^TPM)\mathbf x.$$ If we’re careful to choose $M$ such that $\|\mathbf x\|=1\implies\|M\mathbf x\|=1$, then the solutions carry over to the hyperspace $M\mathbf x$. For your example, if we take for $M$ the obvious orthonormal coordinate system of $W$, the central matrix ends up being $\operatorname{diag}\left(\frac12,1\right)$ so the minimal-length projection is obtained along $\mathbf x=\lambda(1,0)^T$, which corresponds to $(1,0,0)$ and $(-1,0,0)$ in the standard coordinate system of $\mathbb R^3$, as expected.
So, what went wrong? When you concluded that $(P-\lambda I)M\mathbf x = 0$ implies that $M\mathbf x$ must be an eigenvector of $P$, you ignored the implicit constraint $M\mathbf x\in W$. Eigenvectors of $P$ lie either in $V$ itself (eigenvalue $1$) or in $V^\perp$ (eigenvalue $0$). In your example, $V\cap W$ is nontrivial, which is why you found maximally-projecting unit vectors in $W$ that were eigenvectors of $P$, but $V^\perp\cap W = \{0\}$, so there’s no hope of finding minimally-projecting unit vectors of $W$ among the eigenvectors of $P$.
You’ve since updated your question with a corrected equation derived from the Lagrangian, but you’re still making essentially the same sort of error: you cannot conclude from the last equation that $M\mathbf x$ must be an eigenvector of $P^TP$. In fact, $P$ is symmetric (it’s $I$ minus a projection onto a single vector), therefore $P^TP=P^2=P$, so you’re right back where you were before the update.