I am trying to find the least square solution for Ax=b where A= [(1,3,0),(0,1,1)] and b =(4,1,0) so a 3x2 matrix (I am not sure if I am notating it correctly on here)
I am following the equation [(A^T)(A)]x=Ab
when I compute A^t*A, I got (10,3)(3,2) I found my determinant to be 1/11 so the inverse matrix would be
1/11[(2,-3),(-3,10)]
Now when I multiplied the inverse by A*b, i got:
1/11[(11,-11)=x
This is from a final review and the answer I was supposed to get is (-11,17)=x, so I am not sure if I am making a mistake and if so, where the mistake is. He got a different normal vector than me so I don't understand how he got 4 where I got 10... 
You are right. I don't know where that $4$ comes from either. Also, the normal equations are not $A^tAx = Ay$, are $A^tAx = \color{red}{A^t}y$.