, 2012

Answer: $30\sqrt{3}$.

It is easy to show that if the area of the quadrilateral with three given sides is a maximum, then the quadrilateral is convex. Let $AB$, $BC$, $CD$ be given and the quadrilateral $ABCD$ have the maximal area. We have $S(ABCD) = S(ABD) + S(BCD)$ and $S(ABD) = 0.5 \cdot AB \cdot BD \sin \angle ABD$. If $\angle ABD \ne 90^\circ$, then there exists a quadrilateral with given sides having the greatest possible area. Hence $AB \perp BD$. In the same manner, we obtain $CD \perp AC$. So the right angle $ABD$ and $ACD$ subtend the segment $AD$, so the quadrilateral $ABCD$ is inscribed in the circle with the diameter $AC$.

Let $\angle CAD = \beta$, $\angle BDA = \alpha$, $AB = a$, $BC = b$, $CD = c$, $AC = x$, $BD = y$, $AD = z$. Then

$$a = z \sin \alpha, \quad c = z \sin \beta, \quad x = z \cos \beta, \quad y = z \cos \alpha,$$
$$\begin{aligned} b &= z \sin \angle CDA = z \sin(90^\circ - \beta - \alpha) = z \cos(\beta + \alpha) = \\ &= z(\cos \beta \cos \alpha - \sin \beta \sin \alpha). \end{aligned}$$

It is easy to see that $bz + ac = xy$. (Note that this equality follows from the Ptolemaeus theorem.) On the other hand, using the Pythagoras theorem for triangles $ABD$ and $ACD$, we obtain $ac + zb = \sqrt{(z^2 - c^2)(z^2 - a^2)}$. So $z^4 - (a^2 + b^2 + c^2)z^2 - 2abc z = 0$, and since $z \neq 0$, we have
$$z^{3} - (a^{2} + b^{2} + c^{2})z - 2abc = 0.$$
Note that the value of $z$ is independent of the lengths of the sides $AB$, $BC$ and $CD$, so without loss of generality, we set $a = 2$, $b = 7$, $c = 11$. We have
$$z^{3} - 174z - 308 = 0. \qquad (1)$$
We find one of the roots of this equation: $z = 14$. Two other roots of (1) are negative numbers.

Using the sines law for the triangle $BCD$, we obtain $7 = b = z \sin \angle BDC = 14 \sin \angle BDC$, so $\sin \angle BDC = 1/2$, i.e. $\angle BDC = 30^\circ$. Therefore the angle between the diagonals $AC$ and $BD$ is equal to $\angle COD = 90^\circ - \angle BDC = 90^\circ - 30^\circ = 60^\circ$. Now we find
$$AC = x = \sqrt{z^2 - c^2} = \sqrt{196 - 121} = \sqrt{75} = 5\sqrt{3},$$
$$BD = y = \sqrt{z^2 - a^2} = \sqrt{196 - 4} = \sqrt{192} = 8\sqrt{3}.$$
Therefore the required area is equal to
$$S(ABCD) = 0.5 \cdot AC \cdot BD \cdot \sin \angle COD = 0.5 \cdot 5\sqrt{3} \cdot 8\sqrt{3} \cdot \sqrt{3}/2 = 30\sqrt{3}.$$