I wanted to prove that $T(n) = 4T(n/2) + n^{1.7} $ is $O(n^2)$ time complexity using induction.
I start with induction step as $T(k) \le c\cdot k^2$ which gives me:
$$T(n/2) \le c\cdot (n/2)^2 $$
$$4T(n/2) \le 4c\cdot(n/2)^2 $$
$$4T(n/2) + n^{1.7} \le 4c\cdot (n/2)^2 + n^{1.7} $$
$$T(n) \le c\cdot n^2 + n^{1.7} $$
$$T(n) \le c\cdot n^2 + n^2 $$
$$ T(n) \le (c+1)\cdot n^2 $$
Using $ T(n) \le (c+1)\cdot n^2 $ (that I got above) with $T(n) \le c\cdot n^2$ (that I will get from substituting $n$ in $T(k) \le c\cdot k^2$ ) I am not able to prove that $ T(n) \le(c+1)\cdot n^2 \le c\cdot n^2$
Hence its false to call it $O(n^2)$.
Can you help here solving this?
If the original hypothesis $T(k) \le ck^2$ was not strong enough, you may try to introduce another term with less complexity to your hypotheses.
In this case, let $P(n)$ be the new proposition $T(n) \le cn^2 +c_2 n^{1.7}$, where $c_2$ is another constant to be determined. The additional $c_2n^{1.7}$ is inspired by the $n^{1.7}$ in the recurrence relation of $T(n)$.
Assume $P(k/2)$ is true, then for $n=k$,
$$\begin{align*} T(k) &= 4T(k/2) +k^{1.7}\\ &\le 4\left[c(k/2)^2+c_2(k/2)^{1.7}\right] + k^{1.7}\\ &= ck^2 + 4c_2\frac{k^{1.7}}{2^{1.7}} + k^{1.7}\\ &= ck^2 + \left(2^{0.3}c_2+1\right)k^{1.7}\\ \end{align*}$$
At this point, if we pick $c_2$ such that $2^{0.3}c_2+1\le c_2$, then this would finish the proof:
$$\begin{align*} 2^{0.3}c_2 + 1 &\le c_2\\ \left(2^{0.3}-1\right) c_2 &\le -1\\ c_2 &\le -\frac{1}{2^{0.3}-1} \end{align*}$$
In other words, it's possible to prove by induction that
$$T(n) \le cn^2 - \frac{n^{1.7}}{2^{0.3}-1}$$
or even (for example) $$T(n) \le cn^2 - 5n^{1.7}$$
for $n$ which is a power of $2$.
After proving that $T(n) \le cn^2 + c_2n^{1.7}$ for all $n$ that are powers of $2$ and for a fixed $c_2<0$, then it's also true that
$$T(n) \le cn^2 + c_2n^{1.7} \le cn^2.$$