The following problem has really troubled me:

I have row reduced it, so now it looks more like this:

How do I really figure out what values of a and b would create infinitely many solutions, no solutions, or a unique solutions?
Copyright © 2021 JogjaFile Inc.
Based on your calculations, the determinant of the coefficient matrix is $a(b-1)(b+1)$. So if $a\neq0$ and $b\neq\pm1$, there is certainly a unique solution.
If $b=-1$, the last row of your reduced matrix says $0=-4$, so there is no solution.
If $b=1$, then the second (and third) row of your reduced matrix says that $x_3=0$. The first row says $ax_1+x_2=1$, so $x_2=1-ax_1$. You can let $x_2$ be any real number, so there are infinitely many solutions.
If $a=0$, and $b\neq\pm1$, then you can reduce the third column to get: $$\begin{bmatrix} 0&-\frac12(b+1)b&0&-\frac12(b+1)+(2b-2)\\ 0&-(b+1)(b-1)&0&2b-2\\ 0&0&b+1&2b-2 \end{bmatrix}$$
$$\begin{bmatrix} 0&-\frac12b^2-\frac12b&0&\frac32b-\frac52\\ 0&-(b+1)(b-1)&0&2b-2\\ 0&0&b+1&2b-2 \end{bmatrix}$$
The top two equations only admit a solution if $$\begin{align} \left(-\frac12b^2-\frac12b\right)(2b-2) &=-(b+1)(b-1)\left(\frac32b-\frac52\right) \\ \left(-b^2-b\right)(b-1) &=-(b+1)(b-1)\left(\frac32b-\frac52\right) \\ -b(b+1) &=-(b+1)\left(\frac32b-\frac52\right)\\ -b &=\frac32b-\frac52\\ -\frac52b &=-\frac52\\ b &=5 \end{align} $$
So when $a=0$, there is no solution unless $b=5$, in which case $x_3$ and $x_2$ are uniquely determined, but $x_1$ can take any value.