I tried to use the definition of the concave function, but I failed to transform it into the inequality that has to be proved. The original question is:
If $f'' \ge 0, f(0)=0$, and $f$ is differentiable, $\forall a, b \in (0, 1)$, prove that $f(ab)\le a f(b)$
I thought that if $f$ is concave up, the following satisfies: $$\frac{f(b)-f(0)}{b-0}\le\frac{f(b-h)-f(b)}{-h}$$ where $0\le h\le b$. I still don't know how this satisfies mathematically. When we change the inequality by multiplying $-h$: $$f(b-h)\le f(b)-\frac{h}{b}f(b)=\left(1-\frac{h}{b}\right)f(b)$$ When we put $h=(1-a)b$, $$f(ab)\le a f(b)$$
Is this the right procedure? Furthermore, I want to know more precise proof. Could you please give me some key points to this question? Thanks for your advice.
Since $f''$ is nonnegative, $f$ is a convex function. So we can use Jensen's inequality:
To get the desired inequality, take $t=a$, $x=b$ and $y=0$.