Combinatorics. Inscribed Triangle in a decagon. No shared sides.

Question

How many different triangles can be inscribed inside a regular decagon such that the triangle shares its vertices with the vertices of the decagon, but the triangle shares none of its sides?

Here is an example of what is allowed and not allowed:

I considered '10 choose 3', but the idea that the sides of the triangle cannot share it's sides with the sides of the decagon, this can't be correct. Or can it?

Answer 1

There are $\binom{10}{3}$ choices for triangle with no constraints. Now, each triangle in those $\binom{10}{3}$ shares $0,1$, or $2$ sides with the decagon.

Can you count triangles sharing exactly one edge with the decagon?

What about triangles sharing two edges with the decagon?

Edit:

There are $\binom{10}{3} = 120$ triangles. We subtract out the triangles sharing one or two edges with the decagon (no triangle can share all three edges).

Triangles with one edge shared - Choose which edge is the shared one ($10$ choices), then we have $2$ of the vertices of the given triangle, so we need one more. You can't pick any vertex on or adjacent to the edge (as then the triangle would share two edges with the decagon), so there are $10-4 = 6$ choices per edge. That means there are $60$ triangles that share a single edge with the decagon.

Triangles with two edges shared - Choose the vertex incident on the two shared edges ($10$ choices). Now we have the three vertices of the triangle, so everything else is forced. That means there are $10$ triangles that share two edges with the decagon.

Putting it all together, there are $120 - 60 - 10 = 50$ triangles in the decagon that don't share any edges with the decagon.

Answer 2

Revised: You want the number of ways of choosing $3$ of the $10$ vertices in such a way that no two of the chosen vertices are adjacent. Turn the problem around. Call the chosen vertices $A,B$, and $C$ in clockwise order, and look at how the other seven vertices of the decagon can be distributed between them if there must be at least one in each of the gaps (between $A$ and $B$, between $B$ and $C$, and between $C$ and $A$). There are four cases.

1. If one gap contains $5$ vertices, each of the other gaps must contain one vertex. The triangle is isosceles; if $A$ is the vertex on its axis of symmetry, $A$ can be any of the $10$ vertices of the decagon, so there are $10$ such triangles.

2. If one gap contains $4$ vertices, one of the other gaps must contain two vertices, and the other must contain just one. Reading clockwise, starting with the largest gap, the gaps can appear in the order $4,2,1$ or in the order $4,1,2$, and each of these orders can appear in any of $10$ positions around the decagon, so there are $20$ such triangles altogether.

3. Two gaps contain $3$ vertices each, and one contains just one; this is like the first case.

4. One gap contains $3$ vertices, and the others contain two vertices each; this is also like the first case.

The total is therefore $10+20+10+10=50$ triangles.

Thanks to Michael Biro for catching the overcounting in the original solution.