Let $V$ be an inner product space, and let $W$ be a finite-dimensional subspace of $V$. If $x \not\in W$, prove that there exists $y \in V$ such that $y \in W^\perp$, but $\langle x, y \rangle\not= 0$.
Theorem: Let $W$ be a finite-dimensional subspace of an inner product space $V$, and let $u \in V$. Then there exists unique vectors $u \in W$ and $z \in W^\perp$ such that $y = u +z $. Furthermore, if $\{ v_1, v_2, ... , v_k\}$ is an orthonormal basis for $W$, then $$ u = \sum_{i=1}^k \langle y, v_i \rangle v_i.$$
Answer: You can write $x$ as $u+z$, with $u\in W$ and $z\in W^\perp$. Now, take $y=z$. Then $y\in W^\perp$ and$$\langle x,y\rangle=\langle x,z\rangle=\langle u,z\rangle+\langle z,z\rangle=\lVert z\rVert^2\neq0,$$since $z\neq 0$ (otherwise, $x\in W$).
My question: I don't get what's the general idea in proving this problem and how does this theorem apply to that. Also the operations $\langle x,z\rangle=\langle u,z\rangle+\langle z,z\rangle=\lVert z\rVert^2\neq0,$ don't make sense to me, how does $\langle u,z\rangle+\langle z,z\rangle$ lead to $\lVert z\rVert^2$?
The general idea is that, informally speaking, we decompose $x$ into its projections onto $W$ and onto $W^{\perp}$. The projection onto the "perpendicular" (i.e. orthogonal) subspace $W^{\perp}$ is what we want.
To visualize with an easier to imagine analogy: think about the usual two-dimensional cartesian plane $\mathbb{R}^2$ as $V$ and the horizontal axis as the subspace $W$. Then any vector can be decomposed into its horizontal and vertical components. Moreover, if $x\notin W$, i.e. if $x$ is not a horizontal vector, then its vertical component $y$ is nonzero, and the dot product of $x\cdot y$ is also nonzero.
To address your second question: $$\langle x,z\rangle=\langle u+z,z\rangle=\langle u,z\rangle+\langle z,z\rangle=0+\|z\|^2=\|z\|^2\neq0.$$ Here $\langle u,z\rangle=0$ because $u\in W$ and $z\in W^{\perp}$; and $\langle z,z\rangle=\|z\|^z$ by properties of inner products (more precisely, that's how the norm is defined in inner-product spaces).