Notice that $|x| = 3$ is a root of the equation $|x|^3 - 2x^2 - 4|x| + 3 = 0$. Then the original inequality can be rewritten as $(|x| - 3)(|x|^2 + |x| - 1) < 0$, that is
$$(|x| - 3)\left(|x| - \frac{-1 + \sqrt{5}}{2}\right)\left(|x| - \frac{-1 - \sqrt{5}}{2}\right) < 0.$$
Since $|x| - \frac{-1 - \sqrt{5}}{2} > 0$, then $\frac{-1 + \sqrt{5}}{2} < |x| < 3$.
So the solution set is $(-3, -\frac{\sqrt{5} - 1}{2}) \cup (\frac{\sqrt{5} - 1}{2}, 3)$.