The maximum number of acute-angled triangles is 24n(n−1)(n+1) if n is odd and is 24n(n−2)(n+2) if n is even.
We first count the number N1 of non-acute triangles with A1 as one vertex such that ∠A1<90∘. Let ℓ be the diameter of the circle which passes through A1. Then for points B,C on the circle, △A1BC is not acute with ∠BA1C<90∘ if and only if B and C lie on the same side of ℓ (possibly on ℓ). Suppose there are a points on one side of ℓ and b points on the other side of ℓ other than A1. Note that the two sides share at most one common point which is the other intersection of ℓ and the circle. Therefore, we must have a+b=n−1 or a+b=n. We find that
N1=(2a)+(2b).
We have to minimize N1. Note that b≥n−1−a so that (2b)≥(2n−1−a).
WLOG assume a≥n−1−a. If n is odd, then (a,n−1−a)>(2n−1,2n−1).
Applying the majorization inequality, we obtain
N1≥(2a)+(2n−1−a)≥(2n−1)+(2n−1)=4(n−1)(n−3).
The same bound holds if we replace A1 by any other vertex Aj. We sum up all the lower bounds. Note that each non-acute triangle is counted twice in
this sum (one corresponding to each acute angle). Therefore, the number of non-acute triangles is at least
n⋅4(n−1)(n−3)⋅21=8n(n−1)(n−3).
Thus, the number of acute triangles is at most
(3n)−8n(n−1)(n−3)=24n(n−1)(n+1).
This can be attained when A1A2⋯An is a regular polygon (since in that case all inequalities used become equalities).
Similarly, if n is even, then (a,n−1−a)>(2n,2n−2). Applying the majorization inequality, we obtain
N1≥(2a)+(2n−1−a)≥(22n)+(22n−2)=4(n−2)2.
Thus, the number of acute triangles is at most
(3n)−n⋅4(n−2)2⋅21=24n(n−2)(n+2).
The regular polygon case does not yield equality. Indeed, we first consider a regular polygon B1B2⋯Bn. Then we rotate half of the points, say B1,B2,…,B2n, anticlockwise about the centre of the circle with a sufficiently small angle θ to points C1,C2,…,C2n. Then C1C2⋯C2nB2n+1B2n+2⋯Bn is one possible case which gives the upper bound 24n(n−2)(n+2).