I have a confusion in determining big-O, in my book there is an example
Show that $$ 7x^2 = O(x^3)$$
they solved it like this $$x>1\\x^3>x^2\\so, f(7x^2)=O(x^3)$$
and in another example:
Give a big-O estimate for$$ f (x) = (x + 1)\cdot log(x^2 + 1) + 3x^2 $$ here they considered $f(3x^2)=O(x^2)$
Why we didn't use $O(x^3)$ here?
On
Big-O means: $$ \exists c \forall n>n_0 if f(n) <= c*g(n) \rightarrow f(n) \in O(g(n)) $$ so id you can find such $c$ that for all $n$ greater than $n_0$ you can say that $ f(n) \in O(g(n))$. for example: $$ f(n)=4n^2+8n \in O(n^2) $$ because there is a c like 100 that you have: $$ 4n^2+8n <= 100(n^2) $$
I like to think of Big-$O$ notation in terms of limits. Look at this:
In essence we are just saying that the limit as we go to infinity is bounded, and so $g(n)$ has the same or greater asymtotic growth as $f(n)$ does (in this specific sense). In particular, we have that $$\lim_{x \to \infty} \frac{7x^2}{x^3} = 0$$ Thus, we have that $7x^2 \in \mathrm{o}(x^3)$, which is even stronger than $7x^2 \in \mathrm{O}(x^3)$.
In your second example we have that $$\lim_{x \to \infty} \frac{(x+1)\log(x^2+1)+3x^2}{x^2}$$ $$=\lim_{x \to \infty} \frac{(x+1)\log(x^2+1)}{x^2}+\lim_{x \to \infty} \frac{3x^2}{x^2}$$
The first limit yields $0$ by L'Hopital and the second clearly evaluate to $3$, so we have a finite limit, and thus $$\color{red}{(x+1)\log(x^2+1)+3x^2 \in \mathrm{O}(x^2)}$$
Rules you should know for Big-$O$ notation are that constants are irrelevant and you can ignore all terms in $f(n)$ except the fastest growing one. Also, the limits above should really be Limits Superior and Limits Inferior, but that might be a bit above your level.