$a=e^{\frac{2\pi i}{5}}$ $$A=\begin{bmatrix}1 & a &a^2 &a^3 & a^4 \\ 0 & a &a^2 &a^3 & a^4\\ 0 & 0 &a^2 &a^3 & a^4\\ 0 & 0 &0 &a^3 & a^4\\ 0 & 0 &0 &0 & a^4 \end{bmatrix}$$
find trace of $I+A+A^2$
I wrote $$A=a^{10}\begin{bmatrix}1 & 1 &1 &1 & 1 \\ 0 & 1 &1 &1 & 1\\ 0 & 0 &1 &1 & 1\\ 0 & 0 &0 &1 & 1\\ 0 & 0 &0 &0 & 1 \end{bmatrix}$$
since $a^{10}=1$, so $I+A+A^2$could be found by simple matrix multiplication and addition of A. andd then trace came to be 15,which is wrong.what are other possibilities?
Your transformation is not allowed. This works for determinants, not for matrices. Your second equation is wrong. You changed the matrix by manipulating individual columns (which you can see because the trace changed).
You don't need to square the matrix. Just calculate the diagonal terms of $A^2$. You will see that for a triangular matrix this is a trivial operation (hint: what's above the diagonal doesn't matter at all).