Maximal size of triangulation in 17-gon

Question

Given convex 17-gon. What is the maximal count of triangles we can divide it if we draw all it's diagonals? (for 4-gon,answer is 4, for 5-gon answer is 11)

Answer 1

Number of faces for general polygon

From your pentagon example I assume you mean faces, not neccessarily triangles.

Every point on an $n$-gon has $n-3$ adjacent diagonals. Each such diagonal intersects other diagonals, so if you want to compute the number of segments between such crossings, you'd compute

$$\sum_{i=1}^{n-3}i\,(n-2-i)+1=\frac{n^3-6n^2+17n-24}6$$

The idea is that you have one corner from where you are shooting diagonals, and another corner at which you are shooting the current diagonal. Then there are $i$ other corners to the left of that target corner, and $n-2-i$ corners to the right. Each combination of left and right will result in a diagonal, so the product gives the number of crossings. The number of segments is one more than that number of crossings.

Now you do this for every vertex, and you'll have counted every segment twice, once for each direction. Divide by two to correct for that, then add the outer boundary of the polygon, and you obtain the edge count

$$E=\frac n2\frac{n^3-6n^2+17n-24}6+n=\frac{n^4-6n^3+17n^2-12n}{12}$$

Then do the same for vertices. First count the inner crossings for a single point as

$$\sum_{i=1}^{n-3}i\,(n-2-i)=\frac{(n-1)\,(n-2)\,(n-3)}{6}$$

This will count every crossing four times: twice for each of the diagonals involved. Correcting for that and adding the corners, you get

$$V=\frac n4\frac{(n-1)\,(n-2)\,(n-3)}{6}+n=\frac{n^4-6n^3+11n^2+18n}{24}$$

Now you can use Euler's polyhedron formula to obtain the number of faces.

$$F=2-V+E=\frac{n^4-6n^3+23n^2-42n+48}{24}$$

One of these faces will be the outside of the polygon, so you get the final result

$$F-1=\frac{n^4-6n^3+23n^2-42n+24}{24}$$

Here are some numbers:

$$ \begin{array}{rr|rr|rr} n & F-1 & n & F-1 & n & F-1 \\\hline 3 & 1 & 13 & 781 & 23 & 9{,}086 \\ 4 & 4 & 14 & 1{,}079 & 24 & 10{,}879 \\ 5 & 11 & 15 & 1{,}456 & 25 & 12{,}926 \\ 6 & 25 & 16 & 1{,}925 & 26 & 15{,}250 \\ 7 & 50 & \mathbf{17} & \mathbf{2{,}500} & 27 & 17{,}875 \\ 8 & 91 & 18 & 3{,}196 & 28 & 20{,}826 \\ 9 & 154 & 19 & 4{,}029 & 29 & 24{,}129 \\ 10 & 246 & 20 & 5{,}016 & 30 & 27{,}811 \\ 11 & 375 & 21 & 6{,}175 & 31 & 31{,}900 \\ 12 & 550 & 22 & 7{,}525 & 32 & 36{,}425 \end{array} $$

This sequence can also be found on OEIS as A006522. The description states:

4-dimensional analogue of centered polygonal numbers. Also number of regions created by sides and diagonals of n-gon in general position.

The latter is what you've been asking about, except for the distinction between triangles and non-triangular faces. So the answer to your question about the $17$-gon would be 2500. At least if the corners are in general position, so that never more than two diagonals intersect in any point.

Number of triangles for regular polygon

But perhaps your example of 11 triangles for the pentagon was simply wrong. If you really want to count only triangles, that count will depend on the shape of the polygon. Convexity alone does not fix the number. You are asking about the maximal number of triangles. Experiments suggest that a regular polygon might yield a maximal number of triangles, but I have no proof of that.

In any case, according to A062361, the number of triangles for the regular $17$-gon would be $1054$. There is no simple formula associated with that sequence, so I guess one has to “simply” count them. Doing so correctly without interference from limited precision rounding errors can be a bit tricky. For some non-regular polygons the number of triangles might be larger, but personally I doubt that.