I recently read a paper using Brouwer fixed point theorem in finite dimensional Hilbert space as follows:
$X$ is a finite dimensional Hilbert space with inner product $(.,.)$ and norm $||.||$. $f:X \to X$ is continuous. If $f$ satisfies that: For a positive constant $C$, the result $(f(x),x) {\ge} 0$ always holds for all $x\in X$ with $||x||=C$. Then, there exists $x_0\in X$ with $||x_0||\le C$, such that, $f(x_0)=0$.
I want to know if this is true. If it is true, can you show me some reference about it?( Not just reference who cite it, but about prove it)
Yes, this is true. You can get this result from the Leray-Schauder Alternative (which also delivers Brouwer's fixed point theorem). Reduced to your situation ($B(0,C)$ a closed ball in a finite dimensional vector space $X$) the Leray-Schauder Alternative reads:
If $F:B(0,C) \to X$ is continuous, then (at least) one of the following two assertions hold:
$F$ has a fixed point.
There exist $x \in \partial B(0,C)$ and $\lambda \in (0,1)$ such that $x=\lambda F(x)$.
Application to $F:B(0,C) \to X$, $F(x)=x-f(x)$ leads to the following contradiction if you assume that $F$ has no fixed point (that is, if you assume that $f$ has no zero in $B(0,C)$): There exist $x \in \partial B(0,C)$ and $\lambda \in (0,1)$ such that $x= \lambda (x- f(x))$. Then $(1-\lambda)x = - \lambda f(x)$, hence $$ 0<(1-\lambda)C^2= (1-\lambda) \|x\|^2 = - \lambda \langle f(x),x \rangle \le 0. $$ Note that the Leray-Schauder Alternative is far from trivial even in this reduced version. You can find it (with proof) for example in the following books:
Dugundji J., Granas A.: Fixed Point Theory, Springer (2003), p.123
Agarwal R.P., Meehan M., ORegan D.: Fixed Point Theory and Applications, CUP (2004), Theorem 5.1