I wanted to know if the following solution demonstrates that the function $f(x) = (x+1)\mathrm{log}\, (x^2+1) + 3x^2 \in O(x^2)$, because my answer and the book's answer deviate slightly.
Clearly,
$$3x^2 \in O(x^2) \tag{1}$$ $$x+1 \in O(x)\tag{2}$$
The following inequality is where the book and I differ, but I understand how the author obtained his big-O estimate. I said
$$\mathrm{log}(x^2+1) \in O(\mathrm{log}(x^2+1))\tag{3}$$
Therefore, the product of $(2)$ and $(3)$ renders
$$(x+1)\mathrm{log}(x^2+1) \in O(x \mathrm{log}(x^2+1)) \tag{3}$$
Finally, $(1)$ and $(3)$ gives us this big-O estimate
$$(x+1)\mathrm{log}(x^2+1) \in O(\mathrm{max}(x^2,x\mathrm{log}(x^2+1)) = O(x^2) \tag{4}$$
Any problems! Thanks!
If this is as $x \to \infty$, then $\log(x^k+a) =O(\log(x)) $ for any fixed $k$ and $a$.
Also, $\log(x) =O(x^c) $ for any $c > 0$.
This should be enough.