I am trying to prove that for any $s > 0, F_s(x) = |x|^{s-1}x$ defines a homeomorphism from $\mathbb{B}^n$ to itself, which is a diffeomorphism iff $s = 1$.
My attempt so far: Let $F_s(x) = |x|^{s-1}x = |x|^s \widehat{x}$, where $\widehat{x} = \frac{x}{|x|}$ is the normalisation of $x$. Hence, we see that $|F_s(x)| < 1$ if $|x| < 1$, and thus that $F_s(\mathbb{B}^n) \subset \mathbb{B}^n$. Also, for $s,t > 0$ we see that $$ F_t \circ F_s (x) = F_t(|x|^s \widehat{x}) = ||x|^s \widehat{x}|^t \widehat{|x|^s \widehat{x}} = |x|^{st} \underbrace{|\widehat{x}|^t}_{=1} \widehat{|x|^s \widehat{x}} = |x|^{st} \widehat{x} = F_{st}(x). $$ In particular, this gives that $F_{1/s} \circ F_{s} = F_s \circ F_{1/s} = F_1 = id_{\mathbb{B}^n}$. So $F_s$ is a bijection. It is a homeomorphism because $F_s$ is a composition and quotient of the continuous maps $|\cdot|$ and $(\cdot)^s$ for $x \neq 0$. For $x = 0$ we see that $F_s(x) \to 0$ as $x \to 0$, so it is also continuous in $x =0$. Finally, for $s = 1$ we have the identity map, which is a diffeomorphism. For $s < 1$, then $F_s(x) = |x|^{s-1}x = \frac{x}{|x|^{1-s}}$, I suspect that this is not differentiable in the origin, but I don't know how to show this, that's where I am stuck. For $s > 1$, then the inverse $F_{1/s}$ is not smooth since $\frac{1}{s} < 1$.
First, remark that if $F_s$ is a diffeo, then its differential is invertible at any point by the chain rule: $$F^{-1}\circ F=\mathrm{id}\implies\forall x,D(F^{-1})_{F(x)}\circ DF_x=\mathrm{id}.$$ Thus it has trivial kernel at all points of $\mathbb{B}^n$.
Now take the smooth curve $\gamma:(-1,1)\to\mathbb{B}^n$ defined by $\gamma(t)=(t,0,\dots,0)$. You can check that for $t>0$, $(F_s\circ \gamma)(t)=(t^s,0,\dots,0)$. If $F_s$ was smooth, then $F_s\circ\gamma$ would also be by composition: for $s<1$, this is not the case. Now, for $s>1$, note that by the chain rule
$$(DF_s)_0((1,0,\dots,0))=(DF_s)_{\gamma(0)}(\gamma'(0))=(F_s\circ\gamma)'(0)=(st^{s-1},0,\dots,0)|_{t=0}=0,$$ and this contradicts the triviality of the kernel of $(DF_s)_0$.
You can also conclude for $s>1$ as you did in the post, namely showing that $F_s$ is not a diffeo since $(F_s)^{-1}=F_{\frac{1}{s}}$ is not differentiable at $0$ since$\frac{1}{s}<1$ and this case has already been dealed with.