I know $\tan^2(x) \neq \tan(x^2)$ but I can't find an intuitive way to understand $\tan^2(x)$. When I do it in my mind I think: $\tan^2(x) = \left(\frac{\text{opp}}{\text{adj}}\right)^2$ which I think isn't right, that looks more like $\tan(x^2)$.
Thanks for any tips, this has been rolling in my head for a while.
On
Define $f(x):=(\tan x)^2,\,g(x):=\tan(\tan x),\,h(x):=\tan (x^2)$ (i.e. if $y=x^2$ then $h(x)=\tan y$). These are all completely different functions.
Either $f$ or $g$ could be denoted $\tan^2 x$; if it means $f$ we're composing the squaring function with the tangent function, while if it means $g$ we're composing the tangent function with itself. The reason the latter can be denoted $\tan^2 x$ is because functions form an associative algebra under composition, for which composition is like a "multiplication" of functions. And if you "multiply" the tangent function by itself, you're "squaring" it.
However, it's a matter of convention that, unless very clearly stated otherwise, $\tan^2 x$ refers to $f$ rather than $g$, if only because $f$ is far more likely to come up in a problem than $g$ is. In particular, the squared opposite-to-adjacent ratio is indeed $f$.
The traditional notation for the trigonometrc functions is that $\tan^2$ is the function that maps an angle to the square of its tangent. If, for a given $x$, $\mathsf{opp}$ (i.e., $\sin(x)$) and $\mathsf{adj}$ (i.e., $\cos(x)$) denote the lengths of the legs of a right-angled triangle with unit hypotenuse that are opposite and adjacent to the angle $x$, then $\tan^2(x)$ is indeed equal to $(\mathsf{opp}/\mathsf{adj})^2$, i.e., $(\tan(x))^2$. Your thinking that this is more like $\tan(x^2)$ is incorrect: we are holding the right-angle triangle fixed and looking at different functions of the lengths of its edges.
In modern notation, $f^2(x)$ usually means $f(f(x))$. So as $\tan(\tan(x))$ and $(\tan(x))^2$ are certainly not the same function, you need to keep in mind that the convention for trigonometric functions is a special one.