I have a 4x4 matrix and I want to find the triangular matrix (lower half entries are zero).
$$A= \begin{bmatrix} 2 & -8 & 6 & 8\\ 3 & -9 & 5 & 10\\ -3 & 0 & 1 & -2\\ 1 & -4 & 0 & 6 \end{bmatrix} $$
Here are the elementary row operations I performed to get it into triangular form.
row swap rows 1 and row 4
$r_2 - 3\cdot r_1$ replacing $r_2$
$r_3 + 3\cdot r_1$ replacing $r_3$
$r_4 - 2\cdot r_1$ replacing $r_4$
I get this matrix
$$A= - \begin{bmatrix} 1 & -4 & 0 & 6\\ 0 & 3 & 5 & -8\\ 0 & -12 & 1 & 16\\ 0 & 0 & 6 & -4 \end{bmatrix} $$
I then did $4\cdot r_2 + r_3$ to replace $r_3$ and got
$$A= - \begin{bmatrix} 1 & -4 & 0 & 6\\ 0 & 3 & 5 & -8\\ 0 & 0 & 21 & -16\\ 0 & 0 & 6 & -4 \end{bmatrix} $$
I then did $-21\cdot r_4 + 6\cdot r_3$ to replace $r_4$ and got
$$A= - \begin{bmatrix} 1 & -4 & 0 & 6\\ 0 & 3 & 5 & -8\\ 0 & 0 & 21 & -16\\ 0 & 0 & 0 & -12 \end{bmatrix} $$
I am not sure if I did this correctly but the determinant of the matrix should be -36. When I multiply the diagonal entries it isn't -36. I can't figure out what I am doing wrong.
This is a determinant altering operation and not an elementary operation.
Don't write that $A$ equals something which isn't $A$.
Picking up where you errored and using the same idea you had one gets:
$$\begin{align} \begin{bmatrix} 1 & -4 & 0 & 6\\ 0 & 3 & 5 & -8\\ 0 & 0 & 21 & -16\\ 0 & 0 & 6 & -4 \end{bmatrix}&\leadsto \begin{bmatrix} 1 & -4 & 0 & 6\\ 0 & 3 & 5 & -8\\ 0 & 0 & 6\cdot 21 & -6\cdot 16\\ 0 & 0 & -21\cdot 6 & (-21)\cdot (-4) \end{bmatrix}\\ &\leadsto \begin{bmatrix} 1 & -4 & 0 & 6\\ 0 & 3 & 5 & -8\\ 0 & 0 & 6\cdot 21 & -16\\ 0 & 0 & 0 & -12 \end{bmatrix}_.\end{align}$$
Making the proper compensation yields $$\det(A)=-\dfrac{1\cdot 3\cdot (6\cdot 21)\cdot (-12)}{-21\cdot 6}=-36.$$