I know I do it wrong but where is the mistake???
In E3* are given the points $A(1,0,0,0)$, $B(0,1,0,0)$, $D(0,0,1,0)$, $O(0,0,0,1)$ and $E(1,1,1,2)$.
The linear transformation $\Phi$ operates according to the rules:
$A\to A$; $B\to B$; $D\to D$; $O\to E$;
and $E\to O$.
Find the matrix $C$ of the linear transformation (the analytical representment of $\Phi$).
That means we have $5$ points in $3D$ space with homogeneous coordinates and we have to find the linear transformation. I start with this:
$\Phi(A)=A \implies p_1(1,0,0,0) = C(1,0,0,0) \implies c_{11}=p_1, c_{21}=0, c_{31}=0, c_{41}=0$
(these are elements of matrix $C$);
$\Phi(B)=B \implies p_2(0,1,0,0) = C(0,1,0,0) \implies
c_{12}=0, c_{22}=p_2, c_{32}=0, c_{42}=0$;
$\Phi(D)=D \implies p_3(0,0,1,0) = C(0,0,1,0) \implies
c_{13}=0, c_{23}=0, c_{33}=p_3, c_{43}=0$;
$\Phi(O)=E \implies p_4(0,0,0,1) = C(1,1,1,2) \implies
c_{14}=p_4, c_{24}=p_4, c_{34}=p_4, c_{44}=p_4??? $ ;
$\Phi(E)=O \implies p_5(0,0,0,1) = C(1,1,1,2)$
$c_{11} + c_{12} + c_{13} + c_{14} = 0$;
$c_{21} + c_{22} + c_{23} + c_{24} = 0$;
$c_{31} + c_{32} + c_{33} + c_{34} = 0$;
$c_{41} + c_{42} + c_{43} + c_{44} = 0$;
$\implies$
$p_1 + p_4 = 0$;
$p_2 + p_4 = 0$;
$p_3 + p_4 = 0$;
$p_4 = 1$
$\implies$ $p_1=p_2=p_3=-1$.
Where is the mistake?
I see you are working with arbitrary multiples of vectors (as usual in homogeneous coordinates), so that I withdraw my comment above about dimensions. It has to be done more carefully, as you do it in your work shown, keeping track of possible multipliers.
From your steps:
$Φ(B)=B⟹p_2 (0,1,0,0)=C(0,1,0,0)⟹c_{12} =0,c_{22} =p 2 ,c_{32} =0,c_{42} =0$ ;
$Φ(D)=D⟹p_3 (0,0,1,0)=C(0,0,1,0)⟹c_{13} =0,c_{23} =0,c_{33} =p_ 3 ,c_{43} =0$ ;
$Φ(O)=E⟹p_4 (0,0,0,1)=C(1,1,1,2)⟹c_{14} =p_4 ,c_{24} =p_4 ,c_{34} =p_4 ,c_{44} =p_4$ ??? ;
Here the first two look OK, but since $p_4(0,0,0,1)=(0,0,0,p_4)$ it would seem that $c_{14}=c_{24}=c_{34}=0$ and only $c_{44}=p_4$.
Or maybe I have things backwards and the first three $c_{i4}$ are $p_4$, but in that case the last one should maybe be $2p_4$. Well, something must match the $(1,1,1,2)$ shape of vector $E$...
This correction may affect the solving of the rest of the question; I'll look at it again if you don't respond with something.
ADDED: By monkeying around with the above equations I found the matrix $M$ given by
\begin{bmatrix}2 & 0 & 0 & -1 \\ 0 & 2 & 0 & -1 \\ 0 & 0 & 2 & -1 \\ 0 & 0 & 0 & -2 \end{bmatrix} which, viewing your four coordinate points as column vectors and multiplying on the right, send each point to a multiple of what it should be. It sends the first three vectors $A,B,D$ to twice themselves, and the point $(0,0,0,1)$ maps to $(-1,-1,-1,-2)=-1\cdot(1,1,1,2)$, while on the other hand the point $(1,1,1,2)$ maps to $(0,0,0,-4)=-4(0,0,0,1).$ I assume since you're working in homogeneous coordinates that this matrix (or scalar multiples of it) is what you want.