It is said that triangle inequality for the space $L^p(\mathbb{R})$ space doesn't hold if $0<p<1$.
Does anyone know an example for this?
Also, what we can say, for example, about the quantity like $\| f \| \colon =\int_{\mathbb{R}} \sqrt{|f|} dx$?
I think in the space $\{ f ; \|f\| < \infty \}$, triangle inequality $\|f+g\| \le \|f\| + \|g\|$ is valid.
For $0<\alpha<1$ and a measure space $(\Omega,\mathcal E,\mu)$, the function $$f\mapsto\int_\Omega \lvert f\rvert^\alpha\,d\mu$$ is indeed subadditive, because the modulus is and the map $x\mapsto x^\alpha$ is subadditive on $[0,\infty)$. This is a consequence of this lemma:
Thus $\int_\Omega \lvert f+g\rvert^\alpha\,dx\stackrel{x^\alpha\text{ increasing}}\le\int_\Omega (\lvert f\rvert+\lvert g\rvert)^\alpha\,dx\stackrel{\text{lemma}}\le\int_\Omega\lvert f\rvert^\alpha+\lvert g\rvert^\alpha\,d\mu=\int_\Omega\lvert f\rvert^\alpha\,d\mu+\int_\Omega\lvert g\rvert^\alpha\,d\mu$
As for an instance where $\lVert f+g\rVert_\alpha>\lVert f\rVert_\alpha+\lVert g\rVert_\alpha$ for $\alpha\in(0,1)$, you can shape it from the counterexample to $v_\alpha(x,y):=(x^\alpha+y^\alpha)^{1/\alpha}$ being a norm on $\Bbb R^2$.
In that case, one notices that, since $\frac1\alpha>1$, $$v_\alpha(1,1)=2^{1/\alpha}>v_\alpha(1,0)+v_\alpha(0,1)=2$$
In the case of functions $f=1_{[0,1]}$ and $g=1_{[1,2]}$ will do just fine.