I got the following exercise:
Let $f$ be the endomorphism in $\mathbb{R}_3[T] := \{p \in \mathbb{R}[T]; \; deg(p) \leq 3\}$ given by $$\;f: \mathbb{R}_3[T] \rightarrow \mathbb{R}_3[T]$$ $$f(p) = T^4p'' + (1-4T^3)p' + (1+6T^2)p$$
where prime ($'$) denotes the derivative. Now, the problem text states that $$\forall{p} \in \mathbb{R}_3[T]: f(p) \in \mathbb{R}_3[T]$$
But let's assume we have a $p$ with $deg(p) = 3 \implies p \in \mathbb{R}_3[T]$. Then $$ deg(f(p)) = deg(T^4 \cdot p'') = 4 + 1 = 5 \notin \mathbb{R}_3[T]$$
Taking the second derivative of this $p$ will give us some polynomial with $T^1$ as the greatest power. Can someone elaborate on what I am doing wrong or if this exercise is nonsense?
Let us proceed with a more detailed reply, taking the cue from @pigeon. Allow me to change the notation a bit and to consider the exercise in a more general setting: let us fix an arbitrary non-null commutative ring $A$, agree to denote derivatives with over-dots and introduce the map $$\varphi: A[X] \to A[X] \\ \varphi(f)=X^4 \ddot{f}+(1_A-4X^3) \dot{f}+(1_A+6X^2)f$$
Since derivations are indeed linear maps, $\varphi$ is easily seen to be an endomorphism of the free $A$-module $A[X]$ (a module is loosely speaking the same kind of a structure as a vector space, satisfying the same list of axioms with the sole difference being that the external scalars come from a ring and not necessarily from a field).
If we now introduce $(A[X])_n=\{f \in A[X]|\ \mathrm{deg}(f) \leqslant n\}$ for any $n \in \mathbb{N} \cup \{-\infty \}$ we automatically have that $(A[X])_n$ is a submodule of $A[X]$ admitting the family $(X^{k-1})_{1 \leqslant k \leqslant n+1}$ as an ordered basis. For simplicity let us abbreviate $(A[X])_3=M$.
Since inverse images of submodules (of the target mdoule) through linear maps remain submodules (of the source module), in order to ascertain that $\varphi(M)\subseteq M$ it will suffice to show that $\varphi^{-1}(M)$ contains the $4$ monomials generating $M$; indeed, by direct computation we have: $$\varphi(1_A)=0_AX^3+6X^2+0_AX+1_A \\ \varphi(X)=2X^3+0_AX^2+X+1_A \\ \varphi(X^2)=0_AX^3+X^2+2X+0_A \\ \varphi(X^3)=X^3+3X^2+0_AX+0_A$$
by which we have made sure that $\varphi(\{1_A, X, X^2, X^3\}) \subseteq M$. Thus, $\varphi$ restricts to an endomorphism of $M$, and this restriction we shall call $\psi$.
Furthermore, the above list of $4$ relations allows us to at once express the matrix of $\psi$ with respect to the canonical ordered basis mentioned above (the one given by the monomials in increasing order of their degrees); we obtain representation matrix:
$$T=\begin{pmatrix} 1_A & 1_A & 0_A & 0_A \\ 0_A & 1_A & 2_A & 0_A \\ 6_A & 0_A & 1_A & 3_A \\ 0_A & 2_A & 0_A & 1_A \end{pmatrix}$$
where for arbitrary $n \in \mathbb{Z}$ we agree to write $n1_A=n_A$ for the multiples of the unity in our ring. Recall that in general when representing an endomorphism $f$ with respect to an ordered basis $a$, the entry at position $(k,l)$ in the representation matrix will be given by the coefficient accompanying $a_k$ in the expansion of $f(a_l)$ as a linear combination of the $a_h$, with $1 \leqslant h \leqslant n$.
At this point, the only task left to fulfill is that of computing the determinant of $T$; by subtracting the first column from the second and consequently expanding along the first row (populated now by just one nonzero entry of $1_A$ at position $(1,1)$) we first obtain
$$\mathrm{det}(T)=\begin{vmatrix} 1_A & 2_A & 0_A \\ -6_A & 1_A & 3_A \\ 2_A & 0_A & 1_A \end{vmatrix}$$
Next, we can subtract twice the first column from the second and once again expand along the first row to further obtain
$$\mathrm{det}(T)=\begin{vmatrix} 13_A & 3_A \\ -4_A & 1_A \end{vmatrix}=25_A$$