Find a function $g$ continuous in $[0,\infty]$ and positive in $(0,\infty)$ satisfying $g(0)=1$ and $$ \tag{1} \frac{1}{2}\int_{0}^{x} g^2(t)dt=\frac{1}{x}f^2(x), $$ where $f(x)=\int_0^x g(t)dt$.
My try:clearly $f'(x)=g(x)$
differentiating original condition we get
$$4f(x)g(x)=xg^2(x)+\int_{0}^{x}g^2(t)dt$$
using (1)
$$4f(x)g(x)=xg^2(x)+\frac{2}{x}f^2(x)$$
now i am stuck
On
Starting from your last line, multiply both sides by x $$4xf(x)g(x)=x^2g^2(x)+2f^2(x)$$ move all in one side $$ {x^2}g{(x)^2} - 4xf(x)g\left( x \right) + 2f{(x)^2} = 0$$ solve for $g(x)$ $$g(x) = \frac{{f(x)\left( {2 \pm \sqrt 2 } \right)}}{x} $$ remember that $f'\left( x \right) = g\left( x \right)$ $$f'(x) = \frac{{f(x)\left( {2 \pm \sqrt 2 } \right)}}{x}$$ and then $$\frac{f'(x)}{f(x)}=\frac{ {2 \pm \sqrt 2 }}{x}$$ integrate both sides $$\log f(x)=\left( {2 \pm \sqrt 2 } \right)\log x+C$$ $$f(x)= kx^{ {2 \pm \sqrt 2 }}$$ and finally $$g(x)=k\left( {2 \pm \sqrt 2 } \right)x^{ {1 \pm \sqrt 2 }}$$ This function doesn't satisfy the condition $g(0)=1$, so there is no solution.
Dividing the relation by $x$ and writing $f$ explicitly, we have that $$ \frac{1}{2} \cdot \frac{1}{x} \int_0^x g^2 = \left(\frac{1}{x}\int_0^x g\right)^2, \qquad \forall x > 0. $$ Passing to the limit as $x\to 0^+$ and taking into account that $g(0) = 1$ we thus obtain that $1/2 = 1$, a contradiction.
Hence, there is no such $g$.