I want to calculate the area bounded by the curves $f(x) = x \cos^2(x)$ and $g(x) = x$.
To find the points of intersection, I set $f(x)= g(x)$.
$$\begin{aligned}x\cos^2(x)&= x\\ x\cos^2(x)-x&= 0\\ x[cos^2(x)-1] &= 0\\ x[\cos(x)-1]\cdot[\cos(x)+1] &= 0\end{aligned}$$
I find as intersections $x = k \pi$ , with $k \in \mathbb{Z}$.
My issue is that there are no clear boundaries for this function (periodical so with a lot of intersections between curves), so I don't know what the bounds $a$ and $b$ of the integral are supposed to be. (None were given in the questions)
The value of the integral is $$\frac{1}{8} (\cos(2 x) + 2 x (-x + \sin(2 x))) $$
The area you are seeking does not exist as a real number (if even the negative integrals in the $(-\infty,0]$ region are turned to positive), rather the integral which represents the area you are looking for diverges. Note that, if $A$ is the area, \begin{align} A &= \lim_{t \to +\infty}-{\int_{-t}^{0}{x(1-\cos^2{x})\mathrm{d}x}+\int_0^t{x(1-\cos^2{x})\mathrm{d}x}} \\ &= -\int_{-\infty}^{0}{x(1-\cos^2{x})\mathrm{d}x}+\int_0^{+\infty}{x(1-\cos^2{x})\mathrm{d}x} \\ &=2\int_0^{+\infty}{x(1-\cos^2{x})\ \mathrm{d}x} \\ &= \int_0^{+\infty}{x(1-\cos{2x})}\mathrm{d}x \\ &= \int_0^{+\infty}{x \ \mathrm{d}x} \ - \int_0^{+\infty}{x\cos{2x} \ \mathrm{d}x} \\ &= \frac{x^2}{2}-\frac{x\sin{2x}}{2}-\frac{\cos{2x}}{4}\Biggr|_0^{+\infty} \\ &= \lim_{x \to +\infty}\left(\frac{x^2}{2}-\frac{x\sin{2x}}{2}-\frac{\cos{2x}}{4}\right) + 1\rightarrow+\infty\end{align} as $-1 \leq \sin{x}, \cos{x} \leq 1$ so the limit diverges. If you are still not convinced, see that the integral can be written as: \begin{align} A &= 2\int_0^{+\infty}{x(1-\cos^2{x})\mathrm{d}x} \\ &= {\sum_{k =0}^{+\infty}{\int_{k\pi}^{(k+1)\pi}}{x(1-\cos^2{x})\mathrm{d}x}} \\ &=\sum_{k=0}^{+\infty}{\Biggr[\frac{x^2}{2}-\frac{x\sin{2x}}{2}-\frac{\cos{2x}}{4}+1\Biggr]_{k\pi}^{(k+1)\pi}} \\ &= \sum_{k=0}^{+\infty}{\frac{(2k+1)\pi^2}{2}} \end{align} for $k \in \mathbb{Z}$ which also obviously diverges (and now you can even terminate at a partial sum and find the of a section). If the teacher wanted you to just find the integral (which isn't really the area in this case): \begin{align} I=\int_{-\infty}^{+\infty}{x(1-\cos^2{x})\mathrm{d}x}=0 \end{align} since $f(x)=x(1-\cos^2{x})$ is an odd function.