Given a matrix, $$A=\begin{pmatrix} 1 & 1 & -2\\ p& q & 0\\ 0&0 &p-q \end{pmatrix}$$ where $p$ and $q$ are constants,
Determine all possible values of $p$ and $q$ such that $A^2$ is an upper triangular matrix with det$(A^2)=81$
My attempt: I found $$A^2=\begin{pmatrix} p+1 & q+1 & -2(p-q+1)\\ 0& p+q^2 & -2p\\ 0&0 &(p-q)^2 \end{pmatrix}$$
But I cant find the possible values of $p$ and $q$ such that the determinant is 81.
You have an error for $A^2$. It should read $$A^2 = \begin{bmatrix}p+1 & q+1 & -2(p-q+1) \\ \color{red}{p(q+1)} & p+q^2 & -2p \\ 0 & 0 & (p-q)^2 \end{bmatrix}.$$ This is perhaps why you got stuck.
Since $A^2$ must be upper triangular, this implies $p(q+1) = 0$. Thus we have two cases: either $p = 0$, or $q = -1$. In either case, $\det A^2$ is the product of the entries on the main diagonal, so for the case $p = 0$, we would also require $q$ to satisfy $$\det A^2 = (0+1)(0+q^2)(0-q)^2 = q^4 = 81.$$ Solve for all such $q$, which then gives you solutions of the form $(0,q)$. Then, separately, do the same for the case $q = -1$, which will give you solutions of the form $(p,-1)$.