Having the three vertices of a quadrilateral of given area, how can I obtain the coordinates of the fourth vertex?
I have used the following formula which gives the quadrilateral area:
$$A = \frac12 |(x_4-x_2)(y_1-y_3) + (x_3-x_1)(y_4-y_2)|$$
Obviously, if the four points connecting $P_1$ to $P_2$ to $P_3$ to $P_4$ to $P_1$ have lines that cross each other, the answer is meaningless ─ they have to form a simple region with an inside and an outside and no crossing lines.
In my problem, the fourth point must be on a line of slope $m$ which passes through the third vertex. Therefore, we have:
$$y_4 = y_3 + m (x_4-x_3)$$
Substituting into the area formula yields:
$$x_4 = \frac{2A+x_2(y_1-y_3)-(x_3-x_1)[(y_3-y_2)-mx_3]}{(y_1-y_3)+m(x_3-x_1)}$$
Now, assume:
$P_1(4, 2.5), P_2(3, 2), P_3(2.5, 3), A = 1, m = 0.5$
The above formulas give $P_4(-0.1, 1.7)$ while I expected to have $P_4(3.1, 3.3)$. How can I calculate the correct vertex coordinates?
I would say this. Your points $P_1$, $P_2$ and $P_3$ form a triangle with area $A_0$. Now place the point $P_4$. Now, assuming there are no self-intersections, $$\operatorname{Area}(P_1 P_2 P_3 P_4) = A_0 \pm \operatorname{Area}(P_1 P_4 P_3),$$ with the last area counted negatively if $P_4$ and $P_2$ are on the same side of $P_1 P_3$, and positively if they're on opposite sides.
So this is really a problem about the area of a certain triangle $P_1 P_4 P_3$, taking into account the sign convention I mentioned. Now using the formula $(1/2) \times$ base $\times$ height, you'll see that there is a line of points $P_4$ that work. You then need to take the intersection of that line with the other line you mentioned. The point $P_4$ may or may not satisfy the condition of the quadrilateral not having self-intersections.