We want to maximize the function $f(x_1,x_2,x_3,x_4) = \sum_{i=1}^{4}a_i^2x_i$ over the compact set $\Omega = \{x \in \mathbb R^4: |x| = 1, \langle x,a\rangle = 0\}$ where $a = (a_1,a_2,a_3,a_4)$ is some non zero vector
This is a continuous function over a compact set so it admits minimum and maximum. Thus we need to solve the system $\begin{cases}a_i^2 = 2\lambda_1 x_i +\lambda_2 a_i \\ |x|-1 = 0\\\langle x,a \rangle = 0\end{cases}$
I solved this for the case that $\lambda_1 \neq 0$. But when $\lambda_1 = 0$ I have too many variables and no way to isolate $x$ - how do I solve this?
If you have $\lambda_1=0$, then the first condition becomes
$$a_i^2=\lambda_2a_i, i=1,2,3,4$$
for some $\lambda_2$. That means
$$\sum_{i=1}^4a_i^2x_i=\sum_{i=1}^4\lambda_2a_ix_i=\lambda_2\sum_{i=1}^4a_ix_i=\lambda_2\langle a,x\rangle = 0.$$
In other words, your objective function is a multiple of one of of your conditions, so your objective function is the same ($0$) on the whole permissable set.