, 2023

Solution:

Number the vertices $1$ through $7$ in order. Then, the only way to have three vertices of a regular heptagon that do not form an isosceles triangle is if they are vertices $1,2,4$, rotated or reflected. Thus, it is impossible to have four vertices in the heptagon of one color because it is impossible for all subsets of three vertices to form a valid scalene triangle. We then split into two cases:

Case 1: Two colors with three vertices each, one color with one vertex. There is only one way to do this up to permutations of color and rotations and reflections; if vertices $1,2,4$ are the same color, of the remaining $4$ vertices, only $3,5,6$ form a scalene triangle. Thus, we have $7$ possible locations for the vertex with unique color, $3$ ways to pick a color for that vertex, and $2$ ways to assign the remaining two colors to the two triangles, for a total of $42$ ways.

Case 2: Two colors with two vertices each, one color with three vertices. There are $3$ choices of color for the set of three vertices, $14$ possible orientations of the set of three vertices, and $\binom{4}{2}$ choices of which pair of the remaining four vertices is of a particular remaining color; as there are only two of each color, any such assignment is valid. This is a total of $3 \cdot 14 \cdot 6 = 252$ ways.

Thus, the final total is $42 + 252 = 294$.