Given $g(x)$ is continuous in $[0,1]$ and $g(x)<\frac{1}{2}\frac{1}{\sqrt{x}}$ in $(0,1]$.
How to prove that the equation $\int_{0}^{x^2}g(t)dt=x$ is true only if $x=0$?
I solve it like this.
If $g(t)=\frac{1}{2}\frac{1}{\sqrt{t}}$, then $\int_{0}^{x^2}\frac{1}{2}\frac{1}{\sqrt{t}}dt=x^2/2$, but $x^2/2=x$ only when $x=0$, and because $g(x)<\frac{1}{2}\frac{1}{\sqrt{x}}$ in $(0,1]$, then in these domain, the integral won't equal $x$ for any other $x$, so in the domain $[0,1]$, $\int_{0}^{x^2}g(t)dt=x$ is possible only if $x=0$.
Does that enough for the proof?
You've made a mistake on your integral. $\int_0^{x^2}\frac1{2\sqrt{t}}dt$ is $x$, not $x^2/2$. And you haven't explained your logic in the rest of the argument well (probably because the integral mistake will keep the rest of the argument from working correctly).
What you want to prove is:
"If $x\in(0,1]$, then $\int_0^{x^2}g(t)dt\neq x$."
So start by assuming $x\in(0,1]$. Then $x^2\in(0,1]$ as well (you should be able to prove this statement carefully if someone asks). So $g(t)<\frac1{2\sqrt{t}}$ for $t\in(0,x^2)$. Since $g$ is smaller than that function on the range you want to integrate, we can say that $$ \int_0^{x^2}g(t)dt<\int_0^{x^2}\frac1{2\sqrt{t}}dt=\left.\left(\frac22\sqrt{t}\right)\right|_{t=0}^{x^2}=x. $$ (If someone asks, you should be able to carefully prove that integrals of functions get the same inequality sign between them as the functions themselves. This might be a theorem in your book.)
So $\int_0^{x^2}g(t)dt$ is less than $x$ and they can't be equal. In other words, $\int_0^{x^2}g(t)dt\neq x$, which is what you wanted to prove.
One more suggestion: try break your proofs into multiple sentences. If you feel like you want to jam the whole thing into once sentence, it may be because there is something that you don't know how to explain and you're trying to rush past it.