I am attempting to find the derivative of $(x-1)^2$ using the definition method for solving derivatives. $$F'(a) = \lim_{h \to 0} \frac{F(a+h)-F(a)}h $$
Would the RHS become $\lim_{h\to0} \dfrac{(a+h-1)^2-(a-1)^2}h$?
I have attempted this previously turning the formula into sqrt rather than exponent 2, but failed miserable. Any advice?
Yes, that's exactly what it would become.
Square roots have nothing to do with this problem. Functions involving square roots are still possible to do with this process but are more difficult to work with than functions involving squares. Don't think about square roots for this particular problem at all.
For this problem, just expand $(a+h-1)^2$ and $(a-1)^2$, simplify while being very careful with your minus signs, and if done correctly then eventually you'll be able to factor an $h$ out of the numerator, cancel it with the $h$ in the denominator, and get your final answer of $2(a-1)$.