They ask me to calculate the convolution of the index function $h(x) = \mathbb{1}_{(0,1)}(x)$.
Well $h * h(x) = \int_X h(y) \cdot h(x-y)dy = \int_X \mathbb{1}_{(0,1)}(y) \cdot \mathbb{1}_{(0,1)}(x-y)$.
I use that $0 < x-y < 1 \Rightarrow y < x \text{ and } y > x-1$ so $\mathbb{1}_{(0,1)}(x-y) = \mathbb{1}_{(x-1,x)}(y)$.
Aplying this equality to my equation I have that:
$h * h(x) = \int_X \mathbb{1}_{(0,1) \cap (x-1,x)}(y)dy = \mu((0,1) \cap (x-1,x))$
I don't know if it is correct or not. And also I don't know how can I give an explicit solution.
What you have done is correct. The explicit solution is the triangular function.
What remains to do is to break it into three cases:
$$ (0,1)\cap(x-1,x)= \begin{cases} (0,x) & \text{ if } 0\le x<1 \\ (x-1,1) & \text{ if } 1\le x<2 \\ \emptyset & \text{ otherwise } \end{cases}$$
so you get:
$$h*h(x)= \begin{cases} x & \text{ if } 0\le x<1 \\ 2-x & \text{ if } 1\le x<2 \\ 0 & \text{ otherwise } \end{cases}$$
Engineers call this flip and slide.