We have the following theorem. Let $f:\mathbb{R}^{n}\to\mathbb{R}$ be differentiable convex function and $g:\mathbb{R}\to\mathbb{R}$ nondecreasing and pseudoconvex function. Prove that $g(f(x))$ is pseudoconvex function.
I used this idea: We have $x,y\in\mathbb{R}^{n}$ and we need to verify the definition of pseudoconvexity: $$(\nabla {g(f(x))}^{T}(y-x)\ge 0\implies g(f(y))\ge g(f(x)).$$ From the assumption we get $$g{'}(f(x)) (\nabla f(x)(y-x))\ge 0 $$ The first term $g{'}(f(x))\ge 0$, because $g$ is nondecreasing function. So the second term $(\nabla f(x)(y-x))\ge 0$ and because $f$ is differentiable convex function we know that $$f(y)-f(x)\ge \nabla f(x)(y-x)$$ So we get $f(y)\ge f(x)$ and because $g$ is a nondecreasing function we have $g(f(y))\ge g(f(x))$, which we wanted to show.
My question is why do we need the assumption of pseudoconvex function $g$? I didn't use it at all in my idea, but I suppose it's important, otherwise it would not be there. What is wroing in my idea then? Thanks for advice.
We cannot get $\left(\nabla f(x)\right)^T(y-x)\geq 0$ from $g'\left(f(x)\right)\left(\nabla f(x)\right)^T(y-x)\geq 0$ and $g'\left(f(x)\right)\geq 0$ because $\left(\nabla f(x)\right)^T(y-x)$ could be negative if $g'\left(f(x)\right)=0$.
Now, $g$ is a non-decreasing and pseudoconvex function.
Case 1: $g'(f(x))>0$, then $\left(\nabla f(x)\right)^T(y-x)\geq 0$. You have proved it.
Case 2: $g'(f(x))=0$, then $g'(f(x))(f(y)-f(x))=0$ satisfies the condition $g'(f(x))(f(y)-f(x))\geq0$, so, $f(y)\geq f(x)$ due to the pseudoconvexity of $g$.