How to mathematically define "on the outer side of the triangle"?

Question

Given the coordinates of a triangle's vertexes, I'm trying to find its Fermat point programmatically. In one step of the algorithm that I'm trying to implement, I have to draw equilateral triangles on the sides of the original triangle. Here's a picture.

When constructing my equilateral triangles, I have the coordinates of two vertexes and I have to find the coordinates of the third one. The problem is, it's not unique - there are two possible solutions.

Now, if I were doing this manually, I could easily see which one was right and eliminate the other one, but I don't know how to express the fact that the new triangle has to be on the outer side of the original triangle mathematically.

Is there a more precise way to express "on the outer side"? I need something that I'll be able to express programmatically.

Answer 1

You guys gave me some great suggestions, but I came up with something which is, in my opinion, simpler:

There are two possible solutions for the third point. We can calculate the distance between them and the third vertex of the original triangle. Whichever point has the greater distance from the third vertex is the point that we're looking for.

Answer 2

Values of the linear function that represents the line passing through those 2 vertices will have different signs on different sides.

ADDED:

Say the vertices of the side on which you want to draw an equilateral triangle are $(a_1, b_1)$ and $(a_2, b_2)$. Let $(a_3, b_3)$ be the remaining vertex of the initial triangle. All you have to do is to eliminate that new point $(x, y)$ for which $$\left[ (b_2 − b_1)(x − a_1) − (a_2 − a_1)(y − b_1) \right] \left[ (b_2 − b_1)(a_3 − a_1) − (a_2 − a_1)(b_3 − b_1) \right] > 0.$$

Answer 3

Rather than thinking about "outside", it might be better to think about "on the opposite side from the other vertex". Apply a rotation and dilation so that the first vertex is at $(0,0)$ and the second is at $(1,0)$, and then check whether the $y$-coordinate of the third vertex is positive or negative.

Answer 4

One of the basic computations that can be done easily in plane coordinate geometry is "which side of a line is this point?"

I like to think of this in terms of the cross product: if $v \times w$ is positive, that means you go counter-clockwise to get from $v$ to $w$. If $v \times w$ is negative, then you go clockwise.

This gives an easy way to distinguish the two sides of a line. If $P$ is a point on the line, and $v$ is any vector parallel to the line, and you want to know which side of the line some other point $Q$ is on, you compute

$$v \times (Q - P)$$

and the sign tells you to which side of $v$ the vector from $P$ to $Q$ points, and thus which side of the line $Q$ lies on.

---

Another interpretation of the same calculation is by dot products. If we instead let $u$ be a vector perpendicular to the line, then $u \cdot w$ is positive if $u$ and $w$ point in the same direction (i.e. the magnitude of the angle between them is less than $\pi/2$ radians or $90^\circ$), and $u \cdot w$ is negative if they point in opposite directions. This time, our test for the side of a line is

$$ u \cdot (Q - P)$$

---

The problem you're trying to solve is exactly a "which side of the line?" problem. But if you were really interested in identifying which points were inside the triangle, the simplest method is to set it up as three "which side of a line" problems.