Solution:
For n=3 the answer is 6. Suppose n⩾4. It is obvious that in each triangulation there are at least two triangles that share two sides with the polygon. We will prove that it is always best for Mary to choose a triangulation for which there is no more than two triangles of this kind.
We call a triangle in a triangulation bad if all of its sides are diagonals of the polygon. First we prove that Mary can choose an optimal triangulation that contains no bad triangles. Assume on the contrary that every optimal triangulation contains a bad triangle. For an optimal triangulation T let d(T) be the length of the smallest side of all bad triangles in T. Among all optimal triangulations with minimal number of bad triangles let T0 be such that d(T0) is minimal.
Consider a bad triangle ABC in T0 such that ∣AB∣=d(T0). Let ABD be the other triangle of T0 that contains AB as one of its sides. Since D lies on the arc AB of the circumcircle of ABC that does not contain C and ∠ACB is acute, we have ∣AD∣<∣AB∣ and ∣BD∣<∣AB∣.
Let T1 be the triangulation obtained from T0 by replacing AB with CD. If the sides AD and BD have labels a and b respectively, then
S(T1)−S(T0)=a+b−ab−1=−(a−1)(b−1)⩽0
Because T0 is optimal triangulation, we conclude that T1 is also optimal. Since T0 has the minimal number of bad triangles at least one of the segments AD and BD should be a diagonal, but then d(T1) is less than d(T0) what is a contradiction.
Now that we know that Mary can choose an optimal triangulation that contains no bad triangles, we easily conclude that in such a triangulation there are exactly two triangles that share two sides with the polygon. If we denote by x1 (respectively x2) the number of triangles that have exactly one (respectively two) of their sides being the sides of the polygon, then x1+x2=n−2 and x1+2x2=n, so x2=2.
Mary's strategy is to choose these two triangles so that the side of the polygon labeled with 1 is contained in one of these triangles and the side labeled with 2 is contained in the other.
By this strategy Mary makes sure that
S n(n+1) 2 -(1+2+n+n-1)+1 n+2 (n-1), n(n+1) 2 -(1+2+n+n-1)+1 (n-1)+2 n = n 2+3n-6 2
On the other hand, John can force Mary to achieve exactly this bound by labeling the sides of the polygon in the following order
1,n−1,4,n−3,5,…,n−2,3,n,2
Thus, the answer to our problem is S=2n2+3n−6, for each n⩾3.