I'd be very interested in any thoughts on the following argument regarding the necessity of defining Platonic/regular polyhedra as convex. To be specific: are there any obvious flaws in the argument, or is there a much quicker / simpler way to arrive at the conclusion?
The motivation for the question is that many sources include the condition of convexity explicitly into the definition of regular polyhedra, e.g. Coxeter on page 5 of Regular Polytopes (although he modifies his definition later, the requirement of convexity isn't dropped), or David Richeson on page 33 of Euler's Gem. One reason for this seems to be to rule out self-intersecting polyhedra, e.g. Kepler's star polyhedra. The condition of convexity is stronger than non self-intersection however, and so if the motivation is ruling out self-intersection then it's not clear why convexity is stated rather than just non self-intersection. In Richeson, the condition of convexity is given in order to exclude the 'punched in' icosahedron considered on page 48, though it seems that this can be excluded simply because it's not regular given that the vertices are not all congruent.
I will take an initial a definition of a regular polyhedron to be a polyhedron where each face is the same non-intersecting regular polygon and each vertex is congruent to all others (there are many possible definitions of regular polyhedra, this one specifically doesn't mention convexity). To avoid some of the usual exceptions to Euler's polyhedra formula, we might add that the polyhedron is simply-connected, i.e. no holes, and also that in any polyhedra the polygons surrounding each vertex form a circuit, ruling out solids which are composed of two otherwise separate polyhedra joined at a vertex or edge, as Coxeter does on page 4.
We note first that three or more polygons must meet at any vertex. If the number of $p$-sided polygons that meet at each vertex is $q$ then $V=\frac{pF}{q}$ and $E=\frac{pF}{2}$ for $V, E$ and $F$ the number of vertices, edges and faces as usual. Substituting this into $V-E+F=2$ gives $F=\frac{4q}{2p-pq+2q}$. The numerator of this is positive and so the denominator must be too, leading to $(p-2)(q-2)<4$ from which it follows that $p$ cannot be 6 or more, i.e. the faces cannot be hexagons or higher-sided polygons. We also have that if $p=5$ then $q=3$, or if $p=4$ then $q=3$ and finally if $p=3$ then $q=3, 4$ or $5$. This all ties in with Euclid's argument at the end of the Elements, but we note that we have not assumed convexity as Euclid implicitly does.
(It's also quite illustrative to use Descartes's observation that the total angular defect of a polyhedron is $720^\circ$. This rules out the faces being regular polygons with more than five sides as the interior angles of such polygons are $120^\circ$ or more and so three or more of them have a total of $360^\circ$ or more, so each vertex has an angular defect of zero or less and therefore, given that all vertices are congruent, the total angular defect can never be $720^\circ$. Similarly, we can also exclude four or more pentagons or squares at a vertex, or more than six equilateral triangles. While Descartes's observation was only stated just for convex polyhedra, it is valid for any simply-connected non self-intersecting polyhedra.)
Noting that when three polygons form a vertex the vertex is rigid, it follows that when creating a polyhedron where exactly three equilateral triangles, squares or regular pentagons meet at each vertex, there is no `freedom' in how the solid is created and we get the tetrahedron, cube or dodecahedron respectively.
We are left with the possibilities of having four or five equilateral triangles at a vertex.
If four equilateral triangles meet at a vertex then the vertex must be convex. From the formula above, $F$=8 when $p=3$ and $q=4$. It follows from Cauchy's rigidity theorem that there can be only one such polyhedra, the octahedron. This can be illustrated by considering a non-regular vertex such as:
If we continue this with a second congruent vertex we get:
This has six faces and we cannot complete it with just two more except in the special case when the triangles around the vertex all have the same dihedral angle and we get an octahedron. We note again that convexity was a consequence of four equilateral triangles meeting.
We only one final case to look at, namely five equilateral triangle around a vertex. Unless I'm mistaken, this one case causes more work than all of the rest to show that convexity follows automatically, rather than having to be built into any definition. The reason is causes more work is that a non-convex vertex can be formed from five equilateral triangles e.g.:
It probably feels intuitively `obvious' that one can't make a regular polyhedron out of a set of such non-convex vertices, for example some of the edges dip down and others up. The aim of the rest of this is to justify this intuition for all cases, noting that we cannot just quote Descartes's observation as the angular defect is positive and so several such vertices might have a sum of $720^\circ$, nor can we quote Cauchy's rigidity theorem as the vertices are not convex.
Suppose then that we take a vertex where five equilateral triangles meet, such as those just shown, and label the dihedral angles along the edges meeting at the vertex by $A$ to $E$, describing the vertex as being of the form $ABCDE$. A vertex with dihedral angles $ABCDE$ is therefore congruent to one with angles $BCDEA$ (neither being congruent to a vertex described by $ACBDE$). If it turns out that two of the dihedral angles are the same, e.g. $A=E$, then we indicate this by saying that the dihedral angles around the vertex are of the form $ABCDA$.
Theorem: If a polyhedron is composed of congruent vertices each consisting of of five equilateral triangles then either three adjacent dihedral angles are the same or two pairs of adjacent dihedral angles are the same, i.e. the vertices are of the form $AAADE$ or $AABBC$.
Proof:
Figure (a) above shows one vertex with the dihedral angles $A, B, C, D$ and $E$ anti-clockwise around it, with one edge ‘opened’ in order to allow the vertex to be flattened onto a page, hence the two $A$s which are really the same edge. We can consider the dihedral angles around the vertex at the other end of the edge labeled $B$, labeled $X$, assuming for full generality that these can go either (i) anti-clockwise or (ii) clockwise in the order $ABCDE$ (if we insist that they can only go in the same direction as for the original vertex, this only makes the argument shorter).
(i) If the angles around $X$ go anti-clockwise, then given that we already have $B$ in place, we must get (b). We now consider another vertex, such as that at the centre of the bottom of (b) labeled $Y$, considering again the cases when the dihedral angles got either ($\alpha$) anti-clockwise or ($\beta$) clockwise.
($\alpha$) If the angles around $Y$ are considered anti-clockwise, we have $A$ following $C$. This can only happen for $ABCDE$ if either $D=A$, i.e. we actually have $ABCAE$, or $E=C$, i.e. we have $ABCDC$. These are both of the same form and so we only need to consider one of them, e.g. $ABCAE$. If we redraw the diagram based on this and extend it around the vertex labeled $Y$ we get (c). Considering the vertex labeled $Z$ on the left hand side (this actually appears in three places in the diagram due to the ‘opening' of edges), we see that we have two adjacent dihedral angles both $E$. But $A$ appears on either side of $E$ in the pattern currently being considered, $ABCAE$, and so $E=A$, and so the angles must be of the form $ABCAA$, i.e. the first of the two possible arrangements in the Theorem.
($\beta$) If the angles around $Y$ are considered clockwise then we have $C$ following $A$. This means that we must either have $B=C$, i.e. $ACCDE$, or $B=A$, i.e. $AACDE$. These are again of the same form and so choosing the first and again redrawing the diagram and completing the angles aroud $Y$ gives (d). Again we have a vertex at $Z$ with adjacent $E$s, leading to either $D = E$, i.e. $ACCEE$, or $E = A$, i.e. $ACCDA$.
(ii) Lastly, if we let the $ABCDE$ go clockwise around $X$ we get the diagram (e). The vertex in the middle of the top of the diagram labeled $Y'$ has an $A$ next to an $A$ and so we have either $B = A$, i.e. $AACDE$, or $E = A$, i.e. $ABCDA$. Considering also the vertex at the middle of the bottom we have $C$ next to $C$ and so for the $AACDE$ case we must have either $C = A$, i.e. $AAADE$ or $D = C$, i.e. $AACCE$, with similar results if we choose the $ABCDA$ case.
In all cases we have ended up with a set of dihedral angles of the form either $AAADE$ or $AABBC$. QED
We note in passing that it is not possible to continue this argument to show all dihedral angles are equal as it is possible to complete the diagram of a net of an icosahedron and fill it with angles of the form $ABAAA$).
We show instead that the angles must actually all be equal by considering a spherical pentagon (i.e. one drawn on the surface of a sphere) formed by traveling a short distance along each edge from any vertex, e.g. half of the shortest edge coming from that vertex. Given that each of the faces of the original solid are equilateral triangles, the spherical pentagon will be equilateral too, i.e.:
The main geometric fact that we now need is that the angles at the base of an isosceles triangle are equal, something which is valid for spherical geometry just as it is for planar geometry. We will also use the following lemma:
Lemma If two adjacent angles in a spherical (or planar) equilateral pentagon are equal then the two angles adjacent to those angles are equal to each other.
Proof: The four cases to consider are as shown next, with similar reasoning holding in each.
As the pentagon is equilateral, $PQ = QR = RS = ST = TP$, and we will suppose that $\angle QRS = \angle RST$. Let $U$ be the intersection of $QS$ and $TR$. Then $\triangle QRS \cong \triangle RST$ and both are are isosceles so $QS = RT$ and $\angle RQS = \angle RSQ = \angle SRT = \angle RTS$. $\triangle URS$ is therefore isosceles with $UR = US$ and so $QU = UT$. $\triangle UQT$ is therefore isosceles with $\angle TQU = \angle QTU$. $\triangle PQT$ is isosceles with $\angle PQT = \angle PTQ$.
In cases (a) and (b) we have:
$$\angle PQR = \angle RQS + \angle TQU + \angle PQT = \angle RTS + \angle QTU + \angle PTQ = \angle PTS$$
and for cases (c) and (d) we just need the slight modification that: $$\angle PQR = \angle RQS + \angle TQU – \angle PQT = \angle RTS + \angle QTU – \angle PTQ = \angle PTS$$ QED
Finally we show that:
Theorem: If each vertex of a solid is composed of five equilateral triangles, then the dihedral angles around each vertex are all equal to each other.
Proof: We consider the diagrams which correspond to $AAADE$ or $AABBC$ from the end of the first Theorem.
In (a) we have $\angle PQR = \angle QRS = \angle RST$. By the lemma, $\angle PQR = \angle PTS$ and $\angle QPT = \angle RST$ and so all angles are equal
In (b) we have $\angle QPT = \angle PRQ$ and $\angle QRS = \angle RST$. By the lemma, $\angle PTS = \angle QRS$ and so $\angle QPS = \angle QRS$ and so all angles are equal. QED
We have shown that the only polyhedron which has five equilateral triangles around each vertex with all vertices congruent is one in which each dihedral angle is the same, i.e. the icosahedron, which is convex. We do not need to assume convexity therefore in defining the regular solids, this follows from the congruence of the vertices.
Again, I'd be very keen to hear any thoughts on this, particularly any glaring faults in the argument, or a much simpler way to the conclusion. Of course, I'm aware that we can of course always just define the regular solids to be convex from the outset, as it seems Plato, Euclid etc. all did. I hope that the above shows that all that is actually needed is a condition of non self-intersection along with simple-connectedness.