Help! I am taking a math course, and I just can't figure out this proof:
Let $\alpha,\beta\in R^n$, $a\in R$, and $A$ be an $n\times n$ nonsingular matrix. Use contraction mapping theorem to prove that for a sufficiently small $\epsilon>0$, the equation $$ \alpha-Ax+\epsilon(a-\beta^\text{T}x)x=0 $$ has unique solution in $$ D=\{x\in R^n: \Vert x-A^{-1}\alpha\Vert\leq 1\}. $$
I'm writing $a$ and $b$ instead of your $\alpha$ and $\beta$, and $\mu$ instead of your $a$.
The equation to be solved for $x$ can then be written as $$Ax=a+\epsilon(\mu-b\cdot x)\>x\ .$$ Apply $A^{-1}$ on both sides to get $$x=A^{-1}a +\epsilon(\mu-b\cdot x)\>A^{-1}x\ .$$ We now put $x:=A^{-1}a+y$ and obtain the following equation for the new unknown $y\>$: $$y=\epsilon\bigl(\mu-b\cdot(A^{-1}a+y)\bigr)\ A^{-1}\bigl(A^{-1}a + y)=:T(y)\ .\tag{1}$$ We have to show that for sufficiently small $\epsilon$ the equation $(1)$ has a unique solution $y\in B$, where $B:=\{y\in{\mathbb R}^n\>|\>|y|\leq1\}$.
Let $\|A^{-1}\|=:\rho$. Then for all $y\in B$ one has $$|T(y)|\leq\epsilon\bigl(|\mu|+|b|(\rho|a|+1)\bigr)\rho(\rho|a|+1)\ .$$ Here the right side is $\leq1$ if $\epsilon$ is sufficiently small. Therefore we have $T:\>B\to B$ for such $\epsilon$.
We now have to deal with $T(y)-T(y')$ for given points $y$, $y'\in B$. Here we make use of the bilinear structure of the expression $T(y)$. When $$q(y)=(a-by)(c+dy)$$ then $$\eqalign{q(y)-q(y')&=(a-by)(c+dy)-(a-by')(c+dy')\cr &=b(y'-y)(c+dy')+(a-by)d(y-y')\ .\cr}$$ By analogy we have $$T(y)-T(y')=\epsilon\bigl(b\cdot(y'-y)\bigr)\ A^{-1}\bigl(A^{-1}a + y')+\epsilon\bigl(\mu-b\cdot(A^{-1}a+y)\bigr)\ A^{-1}(y-y')\ .$$ It follows that one has an estimate of the form $$|T(y)-T(y')|\leq\epsilon\>C\>|y-y'|$$ with a suitable constant $C>0$. When $\epsilon<{1\over C}$ the mapping $T$ is contracting on $B$.