Answer. For $\alpha = -1$, the identity is the only solution. For other values of $\alpha$, there is no solution.

The functional equation immediately implies that $f$ cannot be a constant function, as $\alpha xy$ would then have to be constant. In the following, we let $(F)$ denote the given functional equation.

Setting $y = 1$, $(F)$ gives us
$$f(f(x + 1)) = f(x + 1) + f(x)f(1) + \alpha x. \quad (1)$$
For $x = 1$ we therefore have
$$f(f(2)) = f(2) + f(1)^2 + \alpha. \quad (2)$$
and replacing $x$ by $x + 1$ then yields
$$f(f(x + 2)) = f(x + 2) + f(x + 1)f(1) + \alpha(x + 1). \quad (3)$$
For $y = 2$, $(F)$ yields
$$f(f(x + 2)) = f(x + 2) + f(x)f(2) + 2\alpha x. \quad (4)$$
For $x = 0$, we therefore obtain
$$f(f(2)) = f(2) + f(0)f(2).$$
Together with (2) this gives us
$$f(0)f(2) = f(1)^2 + \alpha. \quad (5)$$
If we now take $(F)$ and let $y = 0$ and replace $x$ by $x + 1$, we obtain
$$f(f(x + 1)) = f(x + 1) + f(x + 1)f(0). \quad (6)$$
From (1) and (6) we have
$$f(x + 1)f(0) = f(x)f(1) + \alpha x \quad (7)$$
and from (3) and (4)
$$f(x + 1)f(1) = f(x)f(2) + \alpha x - \alpha. \quad (8)$$
If we multiply (7) by $f(2)$ and (8) by $f(1)$, we obtain
$$f(x + 1)f(0)f(2) = f(x)f(1)f(2) + \alpha f(2)x$$
or
$$f(x + 1)f(1)^2 = f(x)f(1)f(2) + \alpha f(1)x - \alpha f(1).$$
After subtracting and taking (5) into consideration, we therefore have
$$\alpha f(x + 1) = \alpha(f(2) - f(1))x + \alpha f(1),$$
and thus (since $\alpha \neq 0$)
$$f(x+1) = (f(2) - f(1))x + f(1).$$
We see that $f$ is a linear function, and $f(x) = ax + b$ with $a \neq 0$. Substitution then gives us
$$a^2x + a^2y + ab + b = ax + ay + b + a^2xy + abx + aby + b^2 + \alpha xy.$$
For $y = 0$ we obtain
$$a^2x + ab = (a + ab)x + b^2, \quad x \in \mathbb{R},$$
an therefore by comparing coefficients $a^2 = a + ab$, or $a = 1 + b$, and $ab = b^2$. We therefore have $(1+b)b = b^2$, and thus $b = 0$, and $a = 1$. For the only possible function $f(x) = x$, we obtain from (F) that $(1 + \alpha)xy = 0$, $x, y \in \mathbb{R}$, or $\alpha = -1$ must hold.

(Walther Janous) ☐