With vertices $(0, 0)$, $(b, a)$, $(x, y)$, prove the area of this triangle is $\frac{|by - ax|}{2}$.
We know area of a triangle = $\frac{rh}{2}$. ($r$ is the base.) Well, we have $r =$ the distance from the origin to $(x, y)$ and the height is the line perpendicular to $y = mx$, which is $y = -\frac{x}{m} + b'$, intersecting the points $(x_0, y_0)$ on the line $y = mx$ and $(b, a)$.
So then, the altitude from my calculations was $ h =\sqrt{1+\frac{1}{m^2}}|x_0 - b|$ The base is just the distance to the origin from $(x, y)$, which is $d = \sqrt{x^2+y^2}$. But I don't see how this equals what I'm trying to prove.
If we denote the point $(x,y)$ by $(x_1,y_1)$ to avoid confusion, we can take the base of the triangle to be $\;\;\;B=\sqrt{x_1^2+y_1^2}$.
The line through $(0,0)$ and $(x_1,y_1)$ has equation $y=\frac{y_1}{x_1}x$ (assuming $x_1\ne0$), so the perpendicular line through $(b,a)$ has equation $y-a=-\frac{x_1}{y_1}(x-b)$ or $y=-\frac{x_1}{y_1}(x-b)+a$ (assuming $y_1\ne0$).
The two lines intersect where $\frac{y_1}{x_1}x=-\frac{x_1}{y_1}(x-b)+a$, so solving for $x$ gives
$\left(\frac{y_1}{x_1}+\frac{x_1}{y_1}\right)x=\frac{x_1 b}{y_1}+a$, so $x=\frac{x_1y_1}{x_1^2+y_1^2}\left(\frac{x_1 b}{y_1}+a\right)\implies x=\frac {x_1(bx_1+ay_1)}{x_1^2+y_1^2}\text{ and }y=\frac{y_1(bx_1+ay_1)}{x_1^2+y_1^2}.$
Then the height of the triangle is given by the distance from $(b,a)$ to this point, so
$\displaystyle h=\sqrt{\left(b-\frac {x_1(bx_1+ay_1)}{x_1^2+y_1^2}\right)^2+\left(a-\frac{y_1(bx_1+ay_1)}{x_1^2+y_1^2}\right)^2}$
$\displaystyle=\sqrt{\frac{y_1^{2}(b y_1-a x_1)^{2}+x_1^{2}(ax_1-by_1)^2}{(x_1^{2}+y_1^{2})^2}}=\sqrt{\frac{(by_1-ax_1)^2}{x_1^2+y_1^2}}=\frac{\lvert by_1-ax_1\rvert}{\sqrt{x_1^2+y_1^2}}$.
Then $\displaystyle A=\frac{1}{2}Bh=\frac{|by_1-ax_1|}{2}$.