The system
$(p-2)x+y=0$
$x+(p-2)y=0$
has a non-trivial solution. Find $p$.
Now I found the augmented matrix in row echelon form, which is $$ \left[ \begin{array}{cc|c} 1&p-2&0\\ 0&1-(p-2)^2&0 \end{array} \right] $$
However I am now confused as to how to find the non-trivial solution. What does it really mean for a solution to be trivial? Is it when $x=y=0$?
Yes, a non-trivial solution is one for which $\ x\ $ and $\ y\ $ are not both zero. If $\ 1-\left(p-2\right)^2\ne 0\ $ then the second row of your row-echelon form gives you $\ y=0\ $, and then the first row gives you $\ x=0\ $. Therefore $\ 1-\left(p-2\right)^2= 0\ $ is a necessary condition for your original system to have a non-trivial solution. I'll leave it for you to determine whether it's also sufficient.