Clearly $f(x) = 0$ is a solution to the original equation. So suppose there exists $t$ such that $f(t) \neq 0$. Substituting $(0,0)$ into the original equation gives $f(2f(0)) = 0$
Substituting $(2f(0),0)$ gives $f(f(0)) = 0$
Substituting $(f(0),f(0))$ gives $f(0) = 0$
Substituting $(x,0)$ gives
$$f(f(x)) = f(x)^2 \qquad (1)$$
Substituting $(x, x)$ gives
$$f(2f(x)) = -f(x)^2 \qquad (2)$$
For convenience, let $g: \mathbb{Z} \to \mathbb{Z}$ satisfy, for any integer $n$,
$$g(n) = \begin{cases} 0, & \text{when } n \equiv 0 \pmod{5}; \\ 1, & \text{when } n \equiv 1, 4 \pmod{5}; \\ -1, & \text{when } n \equiv 2, 3 \pmod{5}. \end{cases}$$
Next we first prove a lemma.
Lemma. If $f(s) \neq 0$, then $f(mf(s)) = g(m)f(s)^2 \quad \forall m \in \mathbb{N}$
Proof: We use mathematical induction. For $m = 1, 2$, this follows from (1) and (2) respectively.
Suppose it holds for all $m < k$; then for $m = k$ (with $k \ge 3$)
(a) $k = 5q$. Substituting $((5q-1)f(s), f(s))$ into the original equation and combining with the induction hypothesis, we obtain
$$f(2f(s)^2) + f(s)^4 = -f(5qf(s))f(s)^2 \qquad (3)$$
Also, by (1) we know $f(s)^2 = f(f(s))$, so by (1) and (2) we know
$$f(2f(s)^2) = f(2f(f(s))) = -f(f(s))^2 = -f(s)^4$$
Substituting into (3) shows $f(5qf(s))f(s)^2 = 0$; and since $f(s) \neq 0$, we have $f(5qf(s)) = 0 = g(5q)f(s)$

(b) $k = 5q + 1$. Substituting $((5q-1)f(s), 2f(s))$ into the original equation and combining with the induction hypothesis gives
$f((5q+1)f(s)) = f(s)^2 = g(5q+1)f(s)^2$
(c) $k = 5q + 2$. Substituting $((5q-1)f(s), 3f(s))$ into the original equation and combining with the induction hypothesis gives
$f((5q+2)f(s)) = -f(s)^2 = g(5q+2)f(s)^2$
(d) $k = 5q + 3$. Substituting $((5q+2)f(s), f(s))$ into the original equation and combining with the induction hypothesis gives
$f((5q+3)f(s)) = -f(s)^2 = g(5q+3)f(s)^2$
(e) $k = 5q + 4$. Substituting $((5q+3)f(s), f(s))$ into the original equation and combining with the induction hypothesis gives
$f((5q+4)f(s)) = f(s)^2 = g(5q+4)f(s)^2$
In all cases, $f(kf(s)) = g(k)f(s)^2$, so by mathematical induction the lemma is proved.
Returning to the original problem, by (1) and (2) there exists $p$ such that $f(p) = |f(t)|f(t)$ ($p$ being either $f(t)$ or $2f(t)$).
By (1) and the lemma we know $f(t)^4 = f(p)^2 = f(f(p)) = f(|f(t)|f(t)) = g(|f(t)|)f(t)^2$
So $f(t)^2 = g(|f(t)|)$ (since $f(t) \neq 0$)
But $|g(|f(t)|)| \le 1$ and $f(t) \ne 0$, so $f(t)^2 = 1$.
By (1) and (2) we get $f(f(t)) = 1, f(2f(t)) = -1$. Next we prove that for all $n \in \mathbb{Z}$,
$f(n) = g(n)$.
If $n = 0$, then $f(n) = 0 = g(n)$
If $n > 0$, substituting $s$ with $f(t)$ and $m$ with $n$ in the lemma gives
$$f(n) = f(nf(f(t))) = g(n)f(f(t))^2 = g(n)$$
If $n < 0$, substituting $s$ with $2f(t)$ and $m$ with $-n$ in the lemma gives
$$f(n) = f(-nf(2f(t))) = g(-n)f(2f(t))^2 = g(-n) = g(n)$$
In summary, $f(n) = g(n)$.
Verification by substitution: it is easy to see that $g(n)^2 \equiv n^2 \pmod 5$
$$\begin{aligned} & \text{So the left-hand side of the original equation} \equiv (x^2 + y^2)^2 + (xy)^2 \\ & \equiv (x^2 + y^2)^2 - (2xy)^2 \equiv (x^2 - y^2)^2 \equiv \text{the right-hand side (mod } 5 \end{aligned}$$
Also, since $|g(n)| \le 1$, we have $|\text{left-hand side} - \text{right-hand side}| \le 3$. Hence the left-hand side equals the right-hand side, and the verification is complete.
Therefore there are two solutions for $f$:
$$f(x) = 0 \quad \forall x \in \mathbb{Z}$$
$$f(x) = \begin{cases} 0, & \text{when } x \equiv 0 \pmod 5; \\ 1, & \text{when } x \equiv 1, 4 \pmod 5; \\ -1, & \text{when } x \equiv 2, 3 \pmod 5. \end{cases} \quad \forall x \in \mathbb{Z}.$$