With $x = ab$, $y = bc$, $z = ca$, we have to find $\max(x - y^2)(y - z^2)(z - x^2)$. From the AM-GM inequality, the product is at most

$$\left( \frac{(x - x^2 + y - y^2 + z - z^2)}{3} \right)^3$$
But $x - x^2 \le \frac{1}{4}$ and its analogues lead to $(x - y^2)(y - z^2)(z - x^2) \le \frac{1}{4^3}$. This value, $\frac{1}{4^3}$, is indeed obtained for $x = y = z = \frac{1}{2}$, that is for $a = b = c = \frac{1}{\sqrt{2}}$, so the desired maximum value is $\frac{1}{4^3}$.

By using the Taiwanese Transformation, that is $a = x \cdot bc^2$, $b = y \cdot ca^2$, $c = z \cdot ab^2$ (with $xyz \ge 1$) we obtain $(xyz)(abc)^2 = 1$. So
$$E = a^4 b^4 c^4 (x-1)(y-1)(z-1) = \frac{(x-1)(y-1)(z-1)}{x^2 y^2 z^2} \le \frac{1}{64},$$
since $\frac{t-1}{t^2} \le \frac{1}{4}$ when $t \ge 1$, thus for $t \in \{x, y, z\}$.