Let $X, Y$ be positive random variables with densities $f_X=\mathbb{I}_{[0, 1]}$ and $f_Y=f_X*f_X$. To find the density of $Y$ we observe that when $0\le y\le 1$ we have $f_Y(y)=y$ and when $1< y\le 2$ then $f_Y(y)=2-y$.
Now if $S=X+Y$ and $R={X\over S}$ I want to find $f_R(r)$.
By the change of variables we get: $$f_{R,S}(r,s)=sf_X(rs)f_Y(s(1-r))$$ $$f_{R,S}(r,s)=\begin{cases} s^2(1-r)\mathbb{I}_{0\le s\le {1\over r}}\ \mathbb{I}_{0\le s\le {1\over 1-r}} \\ (2s-s^2(1-r))\mathbb{I}_{0\le s\le {1\over r}}\ \mathbb{I}_{{1\over 1-r}< s\le {2\over 1-r}} \end{cases}$$ Now we want to integrate w.r.t $s$.
For the first case when $0\le r\le {1\over 2}$ then $0\le s\le {1\over 1-r}$ and when ${1\over 2}<r\le1$ then $0\le s\le {1\over r}$
For the second case when $0\le r \le{1\over 3}$ then ${1\over 1-r}\le s\le {2\over 1-r}$ and when ${1\over 3}<r \le{1\over 2}$ then ${1\over 1-r}\le s\le {1\over r}$. which gives: $$f_R(r)=\begin{cases} {1\over 3(1-r)^2}\ \mathbb{I}_{0\le r\le {1\over 2}} \\ {1-r\over 3r^3}\ \mathbb{I}_{{1\over 2}< r \le 1} \\ {2\over 3(1-r)^2} \mathbb{I}_{0\le r \le {1\over 3}} \\ {2r^3-9r^2+6r-1\over 3r^3(1-r)^2}\mathbb{I}_{{1\over 3}< r \le {1\over 2}} \end{cases}$$
But this is not a density function. Where did I go wrong?
First of all, I'd like to write $$f(x)=\begin{cases} g(x)1_A\\ h(x)1_B \end{cases}$$ where $A\cap B=\emptyset$ as $$f(x)=g(x)1_A+h(x)1_B.$$ Secondly, your mistakes appeared in your discussions before you write down the display of $f_R(r)$. And here is the right track to follow.
I guess you already konwn that $X$ and $Y$ are independent, otherwise it is not right to have $f_{X,Y}=f_Xf_Y$.
Rewrite the display of $f_{R,S}(r,s)$ as $$f_{R,S}(r,s)= s^2(1-r)1_{\{0\le s\le\min\{ {1\over r},{1\over 1-r}\}\}}+ (2s-s^2(1-r))1_{\{0\le s\le {1\over r}\}\cap\{{1\over 1-r}< s\le {2\over 1-r}\}}.$$ When $\frac12\leq r<1$, we have $\frac1r\leq \frac1{1-r}<\frac2{1-r}$, and then $\{0\le s\le\min\{ {1\over r},{1\over 1-r}\}\}=\{1\leq s\leq \frac1r\}$ and $\{0\le s\le {1\over r}\}\cap\{{1\over 1-r}< s\le {2\over 1-r}\}=\emptyset$. Hence $$f_R(r)=\int_0^{\frac1r} s^2(1-r)\,ds=\frac{1-r}{3r^3}.$$ When $\frac13\leq r<\frac12$, we have $\frac1{1-r}< \frac1{r}leq\frac2{1-r}$, and then $\{0\le s\le\min\{ {1\over r},{1\over 1-r}\}\}=\{1\leq s\leq \frac1{1-r}\}$ and $\{0\le s\le {1\over r}\}\cap\{{1\over 1-r}< s\le {2\over 1-r}\}=\{\frac1{1-r}<s\leq\frac1r\}$. Hence $$f_R(r)=\int_0^{\frac1{1-r}} s^2(1-r)\,ds+\int_{\frac1{1-r}}^{\frac1r}(2s-s^2(1-r))\,ds.$$ Can you move on now?