$f$ is a continuous real valued function on $[a,b]$ and also differentiable on $(a,b)$ such that $f(a)=0$. If there exists $k\geq 0$ such that $|f'(x)|\leq k|f(x)|$ for all $x \in (a,b)$ then show that $f(x)=0$ for all $x \in [a,b]$.
If i show that $f$ is constant function on $[a,b]$ then it will be done. But according to given condition I think Lagrange's MVT is required, but how to apply?
On
WLOG assume $a \ge 0$.
Notice that $f'$ is bounded because $f$ is bounded and that $f(x) = \int_0^x f'(t)\,dt$ so $$|f'(x)| \le k|f(x)| = k\left|\int_0^x f'(t)\,dt\right| \le k\int_0^x |f'(t)|\,dt \le k\|f'\|_\infty \int_0^x 1\,dt = kx \|f'\|_\infty$$
$$|f'(x)| \le k\int_0^x |f'(t)|\,dt \le k^2\|f'\|_\infty\int_0^x t\,dt = k^2 \frac{x^2}2 \|f'\|_\infty$$ $$|f'(x)| \le k\int_0^x |f'(t)|\,dt \le k^3\|f'\|_\infty\int_0^x \frac{t^2}2\,dt = k^3 \frac{x^3}{6} \|f'\|_\infty$$
Continuing inductively we see $$|f'(x)| \le k^{n} \frac{x^n}{n!}\|f'\|_\infty \xrightarrow{n\to\infty} 0$$ so $f' \equiv 0$.
Therefore $f \equiv f(a) = 0$.
On
Suppose $f(t)\neq 0$ for some $t\in (a,b)$, say $f(t)>0$. Then there is an open interval around of $t$ for which $f>0$, consequently $|(\log f)'|\leq k$ and hence bound $f(x)\geq f(t)\exp(-k|x-t|)>0$ on this interval. Now take limit at the endpoint, repeat this argument, patch to give all of $x\in [a,b]$
On
If $k=0$, then a simply application of MVT shows that $f$ has to be constant on $[a,b]$ and hence constant $0$.
For non-triviality, assume $k\gt0$. Observe that $f$ is not just continuous, but is also uniformly continuous, that is, for any $\varepsilon\gt0$, there exists $\delta\gt0$ such that for any point $x,y\in[a,b]$, $\lvert x-y\rvert\le\delta$ implies $\lvert f(x)-f(y)\rvert\le\varepsilon$. (This is not like the usual strict inequality, but there will be no problem.)
Choose $\varepsilon_1=0.5, \varepsilon_2=\frac{0.5}{k}$. Then choose suitable $\delta_1,\delta_2$ corresponding to $\varepsilon_1,\varepsilon_2$ respectively, satisfying the above uniform continuity condition. Impose also that $\delta_1,\delta_2$ both less than or equal to $\frac{0.5}{k}$ and $0.5$. Define $\delta=\min\{\delta_1,\delta_2\}$, which is less than or equal to both $\frac{0.5}{k}$ and $0.5$. Continuity condition tells us that $f[a,a+\delta]\subseteq [-0.5,0.5]\cap[-\frac{0.5}{k},\frac{0.5}{k}]$, or equivalently, $\lvert f(x)\rvert\le\min\{0.5,\frac{0.5}{k}\}$ on $[a,a+\delta]$. So on the interval $(a,a+\delta)$, we have $\lvert f'(x)\rvert\le k\frac{0.5}{k}=0.5$. This bounded derivative gives us that $\lvert f(x)\rvert\le 0.5\delta$ on $[a,a+\delta]$. This follows of the following theorem:
Theorem: If $f:[a,a+r]\to\Bbb R$ is continuous and differentiable on $(a,a+r)$ with bounded derivative $\lvert f'(x)\rvert\le M$, then $\lvert f(a)-f(a+t)\rvert\le Mt$ for all $t\in[0,r]$.
You can prove this theorem with MVT.
The inequalities $\lvert f(x)\rvert\le 0.5\delta$ on $[a,a+\delta]$ and $\delta\le\min\{0.5,\frac{0.5}{k}\}$ give $\lvert f(x)\rvert\le \min\{0.25,\frac{0.25}{k}\}$ on $[a,a+\delta]$. Then you obtain $\lvert f'(x)\rvert\le k\frac{0.25}{k}=0.25$ on $(a,a+\delta)$, and then $\lvert f(x)\rvert\le 0.25\delta$ on $[a,a+\delta]$, and then $\lvert f(x)\rvert\le\min\{0.125,\frac{0.125}{k}\}$ on $[a,a+\delta]$...
You may notice that an argument is repeated again and again to obtain smaller and smaller upper bound for $\lvert f(x)\rvert$ on $[a,a+\delta]$. In fact, the inequality $\lvert f(x)\rvert\le\min\{0.5^n,\frac{0.5^n}{k}\}$ holds on $[a,a+\delta]$ for all $n\in\Bbb N$. You formally prove this by induction. Hence, $\lvert f(x)\rvert$ is zero on $[a,a+\delta]$.
So now $f(a+\delta)$ is zero. You repeat the same argument above to demonstrate that $\lvert f(x)\rvert$ is zero on $[a+\delta,a+2\delta]$, and then use that $f(a+2\delta)$ is zero to show that $\lvert f(x)\rvert$ is zero on $[a+2\delta,a+3\delta]$, and so on.
On
The function $g: [a, b] \to \Bbb R$, $g(x) = f(x)^2 e^{-2kx}$ is differentiable on $(a, b)$, with $$ g'(x) = 2 \bigl( f(x)f'(x) - k f(x)^2 \bigr) e^{-2kx} \le 0 $$ because $$ f(x)f'(x) \le |f(x)| |f'(x)| \le k f(x)^2 \, . $$ So $g$ is a non-negative and decreasing function with $g(a) = 0$, and therefore identically zero. It follows that $f$ is identically zero.
Remark: The same argument works also if $b=\infty$, i.e. for functions $f:[a, \infty)$ satisfying $f(a) = 0$ and $|f'(x)| \le k |f(x)|$ for $x > a$.
Let us fix some positive $\delta < 1/k$ (or $\delta = b-a$ if $k=0$).
Let us prove that $$ (1) \qquad f(x) = 0\quad \forall x\in [a, a+\delta]. $$ Once (1) is proved, then from the same argument will follow that $f(x) = 0$ for every $x\in [a+\delta, a+2\delta]$ and so on.
For the proof of (1) you can use the MVT. Namely, let $$ M := \max_{x \in [a, a+\delta]} |f(x)|, \quad D := \sup_{x \in (a, a+\delta)} |f'(x)| $$ By assumption, you have that $D \leq k\, M$. On the other hand, by the MVT, for every $x\in (a, a+\delta]$ there exists a point $\xi_x\in (a, x)$ such that $$ |f(x)| = |f(x) - f(a)| = |f'(\xi_x)|\cdot |x-a|\leq D \, \delta\leq k\, M \, \delta. $$ Takin the $\sup$ in this inequality for $x\in (a, a+\delta]$ we get $$ M \leq k\,\delta\, M. $$ Recalling that $k\, \delta < 1$, this inequality can be satisfied if and only if $M=0$, i.e. if and only if (1) holds.