I have found the equation for line between $A$ and $B$: $$y=x-1$$
Equation for tangent is: $$y=x-\frac{5}{4}$$
Coordinates of point $M(\frac{1}{2},\frac{-3}{4})$
Because the area $AMB$ is approximation of a triangle, we need normal from $M$ to line $AB$ (Langrange's theorem is recommended).
I don't know how to find coordinates of that point (normal line from $M$ to $AB$)
The point you're looking for is of the form $(a, a^2-1)$ for some $a \in \mathbb{R}$. Now the projection of that $M$ will be some $(p, p-1): (p-a, p-a^2) \cdot (1, 1) = 0 \implies p = \frac{a^2+a}{2}$.
Now calculate its norm, find the triangle's area in terms of $a$ and derivate that expression to find a maximum (you will also get two minimums for $a=0,1$)