Serbian Mathematical Olympiad

Solution:

Let $k$ be the circle circumscribed about triangle $ABD$, and let $l$ be the line through $D$ parallel to $AB$. The radius of the circle $k$ is $1$. The rays $BC$ and $AE$ intersect $k$ at points $H$ and $I$, and the line $l$ at $F$ and $G$, respectively. Triangles $FCD$ and $GDE$ are similar because $\angle CFD=\angle DGE=60^\circ$ and $\angle FCD=120^\circ-\angle CDF=\angle GDE$. Let us denote by $k=\frac{FC}{GD}=\frac{FD}{GE}$ the ratio of similarity, by $h$ the distance from the point $D$ to $HI$, and $x=FD$, $y=GD$. It is easy to find that $x+y=2+\frac{2}{\sqrt{3}} h$ and $xy=\frac{4}{3} h^2+\frac{2}{\sqrt{3}} h$ (the product $xy$ is the power of the point $F$ with respect to $k$ and is equal to $OF^2-1$, where $O$ is the center of the circle $k$). Thus we obtain
$$\begin{aligned} & P_{ABFG}=\frac{1}{2}(1+x+y)\left(\frac{\sqrt{3}}{2}+h\right)=\frac{1}{\sqrt{3}} h^2+2h+\frac{3\sqrt{3}}{4} \\ & P_{FCD}+P_{GDE}=\frac{1}{2}(x \cdot FC+y \cdot GE) \sin 60^\circ=\frac{\sqrt{3}}{4} xy\left(k+\frac{1}{k}\right) \geq \frac{\sqrt{3}}{2} xy=\frac{2}{\sqrt{3}} h^2+h \end{aligned}$$
so that
$$P_{ABCDE}=P_{ABFG}-(P_{FCD}+P_{GDE}) \leq -\frac{1}{\sqrt{3}} h^2+h+\frac{3\sqrt{3}}{4}=f(h)$$
The quadratic function $f(h)$ attains its maximum for $h=\frac{\sqrt{3}}{2}$, which proves that $P_{ABCDE} \leq \sqrt{3}$. Equality would hold only if $h=\frac{\sqrt{3}}{2}$ and $k=1$; then (if without loss of generality we assume $DA \geq DB$) the point $D$ would be symmetric to the point $B$ with respect to $HI$, so triangle $ADG$ would be equilateral and $FC=GD=GA=FB$, which is impossible because then $B$ and $C$ would coincide. Therefore the above inequality is strict.

Second solution.

Assume that triangle $ABD$ is not obtuse. Then the points $C'$ and $E'$ symmetric to the points $C$ and $E$ with respect to the lines $BD$ and $AD$ respectively lie inside triangle $ABD$, on the same line through $D$, and it holds that
$S_{ABCDE}=S_{ABD}+S_{ADE}+S_{BDC}=S_{ABD}+S_{ADE'}+S_{BDC'} \leq 2S_{ABD}-S_{ABF}$, where $F$ is the intersection point of the lines $AE'$ and $BC'$. Equality holds if and only if $F \equiv C' \equiv E'$. Let us denote $\angle BAD=\alpha$. Then $\angle ABD=150^\circ-\alpha$, $\angle BAF=2\alpha-120^\circ$, $\angle ABF=180^\circ-2\alpha$, so it holds that
$$\begin{aligned} & S_{ABD}=\sin \alpha \sin (150^\circ-\alpha)=\frac{\sqrt{3}}{4}+\frac{1}{2} \cos (150^\circ-2\alpha)=\frac{\sqrt{3}}{4}+\frac{1}{2}u \\ & S_{ABF}=\frac{1}{\sqrt{3}} \sin (2\alpha-120^\circ) \sin 2\alpha=-\frac{\sqrt{3}}{12}+\frac{\sqrt{3}}{6} \cos (300^\circ-4\alpha)=-\frac{\sqrt{3}}{4}+\frac{1}{\sqrt{3}}u^2 \end{aligned}$$
where $\cos (150^\circ-2\alpha)=u$ and from this $\cos (300^\circ-4\alpha)=2u^2-1$. Now we have
$$S_{ABCDE} \leq 2S_{ABD}-S_{ABF}=\frac{3\sqrt{3}}{4}+u-\frac{u^2}{\sqrt{3}} \leq \sqrt{3}$$
with equality that would hold for $u=\frac{\sqrt{3}}{2}$, i.e. $\alpha \in \{60^\circ, 90^\circ\}$, and $F \equiv C' \equiv E'$, but is never attained because for these values of $\alpha$ the point $F$ is located at the vertex of the right angle, so the pentagon is degenerate.

In the case of an obtuse triangle $ABD$, with the same notation, the point $F$ is located outside triangle $ABD$, but the above expression for the area of $\triangle ABF$ takes negative values, so again we obtain $S_{ABCDE} \leq \frac{3\sqrt{3}}{4}+u-\frac{u^2}{\sqrt{3}}<\sqrt{3}$.