Let $f: \mathbb{R}^{3} \rightarrow \mathbb{R}^{2}, f(x)=Ax$ where $$A = \begin{pmatrix} 1 & 2 & 3\\ 4 & 5 & 6 \end{pmatrix}$$
On the vector space $\mathbb{R}_{3}[x]$ of the real polynomials $p$ of grade $n_{p} \leq 3$ we have derivated function $f: \mathbb{R}_{3}[x] \rightarrow \mathbb{R}_{3}[x], f(p) = p'$ and the composition is $g = f \circ f$.
Calculate the dimensions and basis of the kernel $\text{Ker(g)}$ and the image $\text{Im(g)}$
I think to do this we need to derivate $f$ first? So we have
$$f(x) = Ax$$
$$f'(x) = A = \begin{pmatrix} 1 & 2 & 3\\ 4 & 5 & 6 \end{pmatrix}$$
But then I have troubles getting the kernel because there is no variable left because we derivated..?
Because I cannot write that matrix $A$ equals zero-vector. Or I don't have to derivate here at all? :s
The first function $f: \mathbb R^3 \to \mathbb R^2$ has nothing to do with your question and we can disregard it.
You defined function $f: \mathbb R_3[x] \to \mathbb R_3[x]$ by $$f(p)=p', \quad p \in \mathbb R_3[x]$$ that is $f$ takes every polynomial $ax^3+bx^2+cx+d$ into its derivative $3ax^2+2bx+c$ (note that this is a linear function). Then $g=f \circ f$ is another linear function, taking every polynomial $ax^3+bx^2+cx+d$ into it's second derivative $6ax+2b$.
Since, by definition $$\begin{align} \ker(g) &=\{ p \in \mathbb R_3[x] \mid g(p)=0 \} \\ \operatorname{Img}(g) &=\{ g(p) \mid p \in \mathbb R_3[x] \} \end{align}$$ from the above, it is easy to see that $\ker(g)= \operatorname{Img}(g)=\mathbb R_1[x]$.