To check increasing and decreasing function

Question

Can anyone help me in finding the interval for increasing and decreasing of $g(x)$. I know that $tanx$ is increasing in the interval given and $cotx$ is decreasing.I even tried differentiating $g(x)$ but no success.

Answer 1

None of the answers are correct for all choice of $f$ meeting the given condition. Because, as mentioned in comments, on the domain of $g$, the interval $(\pi/2, \pi)$, tangent and cotangent only take negative values.

Consequently, neither invocation of $f$ receives an input in the interval $(\pi/2, \pi)$, the only interval in which $f$ is constrained. Therefore, the values of $f(\tan x)$ and $f(\cot x)$ are completely unconstrained.

For example, let $f(x) = \left| \left( x - \frac{3\pi}{4} \right)^2 - \frac{\pi^2}{16} \right|$. Then plots of $f$ and $g$ in the interval $(\pi/2,\pi)$ are

We can easily see that $f'' < 0$ on the interval $(\pi/2, \pi)$. We also see that the graph of $g$ matches only the second of the four listed answers.

However, a different choice of $f$, which is identical on the interval $(\pi/2, \pi)$ yields different results. Let's see what happens with $f(x) = -\left( x - \frac{3\pi}{4} \right)^2 + \frac{\pi^2}{16}$.

We can easily see that $f'' < 0$ on the interval $(\pi/2, \pi)$. We also see that the graph of $g$ matches only the first, third, and fourth of the four listed answers.

Finally, an example highlighting that the invocations $f(\tan x)$ and $f(\cot x)$ only sample the values of $f$ that are not constrained. Here, $f(x) = \begin{cases} \ln x ,& \pi/2 \leq x \leq \pi \\ U([-1,1]) ,& \text{otherwise} \end{cases}$, where $U[-1,1]$ is a uniformly chosen random number in the interval $[-1,1]$.

Clearly, $f'' < 0$ on $(\pi/2, \pi)$. But that does not matter because $\tan(x)$ and $\cot(x)$ are only ever negative so they only reveal the behaviour of $f$ on the interval $(-\infty, 0]$, which in this example is completely random.

Equally clearly, none of the four provided answers are correct for this $f$.