Definition: $o(g(n)) = \{f(n) | \forall c>0, \exists N\in \mathbb{N}:0\leq f(n)<c\cdot g(n)\}$ as in CLRS book
Is the following statement true?
If $f(n) = o(g(n))$ and $f,g,h:\mathbb{N}\rightarrow\mathbb{N}$ are increasing sequences that approach $\infty$, then $h(f(n)) = o(h(g(n))$
No, in fact both of the statements are false:
If $f(n) = o(g(n))$ then $h(f(n)) = o(h(g(n))$
If $f(n) = O(g(n))$ then $h(f(n)) = O(h(g(n))$
(Assume $f,g,h:\mathbb{N}\rightarrow\mathbb{N}$ are increasing sequences that approach $\infty$)
Example for $h(f(n)) \neq o(h(g(n))$:
$f(n) = 2^{n/2}$
$g(n) = 2^n$
$h(n) = lg(n)$
If the statement was true, then we get $n/2 = o(n)$, which is false.
Example for $h(f(n)) \neq O(h(g(n))$:
$f(n) = 2n$
$g(n) = n$
$h(n) = e^n$
If the statement was true, then $e^{2n} = O(e^n)$, which is false.