x^2 = 71 mod 77 has four answers as x = 15 mod 77, x = 29 mod 77 , x= −15 mod 77 and x = −29 mod 77. According to the formula when we know the solutions we can try to factorize n as follows:
x = ±a,±b of x2 = y mod n where n = p × q
a = b mod p and a = −b mod q.
However when I plug in a's and b's to the formula it doesnt give me the result ?
You mean $x^2\equiv71\pmod{77}$ etc.
If $a=15$ and $b=29$, then indeed $a\equiv b\pmod 7$ and $a\equiv -b\pmod{11}$ so you can take $p=7$ and $q=11$.
To factor $n$ what you do is find $a$, $b$ with $a^2\equiv b^2\pmod n$ but $a\not\equiv \pm b\pmod n$ and compute the greatest common divisor of $a-b$ and $n$. That will be a proper factor of $n$. Here $\gcd(a-b,n)=\gcd(-14,77)=7$.