A transformation such as $P=P^{2}$ is a projection, so tr = 1. This is the system i build in order to calculate the matrix, the problem it's that it misses an equation and i don't know how to find it...
$\begin{bmatrix} a & b\\ c &d \end{bmatrix}$ is the matrix
$\begin{align*}a+d=1\\ a-2b=0\\ c-2d = 0 \end{align*}$
The last 2 equations is calculated using the kernel.
To check the condition
it's enough to check it for the basis vectors: that $P e_1$ and $Pe_2$ lie on the line $y=2x$. Since $Pe_1$ and $Pe_2$ are just the columns of $P$, this gives us the equations \begin{align} c &= 2a, \\ d &= 2b. \end{align}
To check the condition
it's enough to check that one point on this line is mapped to the origin, which as you've found gives the condition \begin{align} a - 2b &= 0, \\ c - 2d &=0. \end{align} Altogether, this tells us that $$ P = \begin{bmatrix} a & b \\ c & d\end{bmatrix} = \begin{bmatrix}a & \frac12a \\ 2a & a\end{bmatrix}. $$ Now we need some way of testing that $P^2 = P$. In fact we can just compute $P^2$ and see for which value of $a$ it equals $P$. You're also right that asking for the trace to be $1$ (and therefore $a = \frac12$) would work. We could also ask that a vector already on the line $y=2x$ is mapped to itself.