Let $a, b \in (0,1)$ with $a+b \le 1$.
Find all continuous functions $f:[0,1] \to \mathbb{R}$ having: $$ \int_{0}^{x} f(t) \,\mathrm dt = \int_{0}^{ax} f(t) \,\mathrm dt + \int_{0}^{bx} f(t) \,\mathrm dt \tag1 \label1 $$ for all $x \in [0,1]$.
From \eqref{1}, by derivation we get the following functional equation: $$f(x)=af(ax) +bf(bx) \tag2 \label2$$ By replacing $x$ with $ax$ in \eqref{2}: $$f(ax)=af\left(a^2x\right) +bf(abx) \tag3 \label3$$ By replacing $x$ with $bx$ in \eqref{2}: $$f(bx)=af(abx) +bf\left(b^2x\right) \tag4 \label4$$ Now, using \eqref{3} and \eqref{4} in \eqref{2}: $$f(x)=a^2f\left(a^2x\right) +2abf(abx) + b^2f\left(b^2x\right) \tag5 \label5$$ Because $f$ continuous on a compact interval, cf. Weierstrass theorem $f$ is bounded, therefore: $$|f(x)|=\Bigl|a^2f\left(a^2x\right) +2abf(abx) + b^2f\left(b^2x\right)\Bigr| \le M (a + b)^2 \tag6 \label6$$ The inequality \eqref{6} can be inductively extended to: $$|f(x)| \le M (a + b)^n , \ n\ge 2, n\in \mathbb{N} \tag7 \label7$$ If $a+b < 1$, from \eqref{7} we get $f(x)=0 \ \forall x$, so $f$ is constant.
The only case I cannot cover is $a+b=1$. Does anyone have an idea?
Regard the case $a+b=1$.
If we suppose that $f$ is continuous we can repeatedly apply your argument to get $(5)$ : $$f(x)=a^2f(a^2x) +2abf(abx) + b^2f(b^2x) \tag5$$ to obtain for example $$f(x)=a^3f(a^3x)+3a^2bf(a^2bx)+3ab^2f(ab^2x)+b^3f(b^3x)$$ noticing that $a^3+3a^2b+3b^2a+b^3=1$. If we keep going like this we'll obtain something like $$f(x)=a_1f(x_1)+a_2f(x_2)+...+a_nf(x_n)$$ where $a_1+...+a_n=1$ and $x_1,..,x_n$ are approaching to $0$: let's say that at the step $N=n+1$ we have $0\le x_1,...,x_n\le\varepsilon_N$. Then we can write $$m(\varepsilon_N)\le f(x)\le M(\varepsilon_N)$$ where $m(\varepsilon_N)$ is the minimum of $f$ in $[0,\varepsilon_N]$ and $M(\varepsilon_N)$ is the maximum. Taking the limit we obtain $$f(x)=f(0).$$