Solution:

Answer: $\frac{31}{5} \sqrt{3}$

From the law of cosines we compute $\measuredangle A=\cos^{-1}\left(\frac{5^{2}+8^{2}-7^{2}}{2(5)(8)}\right)=60^{\circ}$. Using brackets to denote the area of a region, we find that
$$[ABC]=\frac{1}{2} AB \cdot AC \cdot \sin 60^{\circ}=10 \sqrt{3}$$
The radius of the incircle can be computed by the formula
$$r=\frac{2[ABC]}{AB+BC+CA}=\frac{20 \sqrt{3}}{20}=\sqrt{3}$$
Now the height from $A$ to $BC$ is $\frac{2[ABC]}{BC}=\frac{20 \sqrt{3}}{7}$. Then the height from $A$ to $DE$ is $\frac{20 \sqrt{3}}{7}-2r=\frac{6 \sqrt{3}}{7}$. Then $[ADE]=\left(\frac{6 \sqrt{3} / 7}{20 \sqrt{3} / 7}\right)^{2}[ABC]=\frac{9}{100}[ABC]$. Here, we use the fact that $\triangle ABC$ and $\triangle ADE$ are similar.

Similarly, we compute that the height from $B$ to $CA$ is $\frac{2[ABC]}{CA}=\frac{20 \sqrt{3}}{8}=\frac{5 \sqrt{3}}{2}$. Then the height from $B$ to $HJ$ is $\frac{5 \sqrt{3}}{2}-2r=\frac{\sqrt{3}}{2}$. Then $[BHJ]=\left(\frac{\sqrt{3} / 2}{5 \sqrt{3} / 2}\right)^{2}[ABC]=\frac{1}{25}[ABC]$.

Finally, we compute that the height from $C$ to $AB$ is $\frac{2[ABC]}{5}=\frac{20 \sqrt{3}}{5}=4 \sqrt{3}$. Then the height from $C$ to $FG$ is $4 \sqrt{3}-2r=2 \sqrt{3}$. Then $[CFG]=\left(\frac{2 \sqrt{3}}{4 \sqrt{3}}\right)^{2}[ABC]=\frac{1}{4}[ABC]$.

Finally we can compute the area of hexagon $DEFGHJ$. We have
$$\begin{gathered} [DEFGHJ]=[ABC]-[ADE]-[BHJ]-[CFG]=[ABC]\left(1-\frac{9}{100}-\frac{1}{25}-\frac{1}{4}\right)=[ABC]\left(\frac{31}{50}\right)= \\ 10 \sqrt{3}\left(\frac{31}{50}\right)=\frac{31}{5} \sqrt{3} . \end{gathered}$$