Then which one of the following $T : V \to V$ are NOT linear transformations: for $f(x)$ in $V$, define $T(f(x))$ as:
(a) $f(X^2)$
(b) $f(X)^2$
(c) $X^2f(X)$
(d) $f(X^2+1)$
I was trying to prove the conditions for linear transformations. The addition and scalar multiplication criteria, but I'm lost already.
Let $a,b \in R$, $f,g \in R[x]$.
Let's take a look at part $(b)$, for example. We can show that $T(f(x)) = f(x)^2$ is not linear by picking $a,b,f,g$ such that $T(af + bg) \not= aT(f) + bT(g)$. Indeed if we pick $f = g = 1$, then $T(f+g) = T(2) = 4$ but $T(f) + T(g) = 2$.
For the rest of them, start by experimenting: pick a few combinations of polynomials $f,g$ and scalars $a,b$ and evaluate both $aT(f) + bT(g)$ and $T(af + bg)$. You'll often learn a lot on the first, simplest choice. If you find that the two expressions are agreeing, try to prove directly that the transformation is actually linear.