I have the transformation matrix where the transformation is the derivative operator.
The basis for v is $$\{\sin(x)\cos(x), \sin^2(x), \cos2(x)\}$$ and the transformation is $$\{ \cos^2(x)-\sin^2(x), 2\sin(x)\cos(x), -2\sin(x)\cos(x)\}$$. I keep forgetting the correct alignment of the matrix.
Is $\{0, 2, -2\}$ the first column or the first row? I have it as the first column but it looks like in old notes the professor wrote it as the row.
It's the first row. Think of it like this, the first column has to contain the coefficients of the linear combination of your basis vectors such that it equals the derivative of the first basis vector. There are no $2$s in that linear combination.
Please, also learn to use MathJax properly, powers are created by ^, in your case sin^2.
MathJax basic tutorial and quick reference