Problem 1837

Taking $x = y = 0$ in (1), we obtain $f(0)^2 = 2f(0)$ and hence $f(0) = 0$ or $2$. If $f(0) = 2$, then $y = 0$ in (1) gives that $f(x) = 2 + x$ for all $x$. Putting this in (1), we get $2xy = 0$ for all $x, y$. This contradiction shows that there are no solutions with $f(0) = 2$. Thus $f(0) = 0$.
Putting $x = y = 1$ in (1), we have $f(2) + f(1)^2 = 5f(1)$. Similarly, $x = y = 2$ gives $f(2)^2 = 6f(2)$. If we take $f(2) = 0$, then $f(1) = 0$ or $5$. If $f(2) = 6$, then $f(1) = 2$ or $3$. Thus we have to consider 4 cases: $(f(2), f(1)) = (0, 0), (0, 5), (6, 2)$ and $(6, 3)$.

Case 1: $(f(2), f(1)) = (0, 0)$.
Taking $y = 1$ in (1), we obtain
$$f(x+1) = 3f(x). \quad (2)$$
Replacing $x$ by $x+1$, $y$ by $2$ in (1) and using $f(2) = 0$, we also get
$$f(x+3) = 3f(x+2) + f(2x+2),$$
for all $x$. This simplifies, in view of (2), to $f(2x) = 0$ for all $x$. Thus $f(x) \equiv 0$ is a solution.

Again $y = 1$ in (1) gives
$$f(x+1)+2f(x)=5(x+1), \quad (3)$$
for all $x$. Taking $y = 2$ in (1) and using $f(2) = 0$, we also obtain $f(x+2) = 3f(x)+f(2x)$, for all $x$. Writing $x+2 = (x+1)+1$ and using (3) repeatedly, we get
$$f(2x) = f(x) - 5x. \quad (4)$$
Putting $y = x$ in (1), we also get $f(2x) + f(x)^2 = 2(x+1)f(x) + f(x^2)$. Substituting for $f(2x)$ from (4), we get
$$f(x)^2 - f(x^2) = 2xf(x) + f(x) + 5x,$$
for all $x$. Here we change $x$ to $2x$ to get
$$f(2x)^2 - f(4x^2) = 4xf(2x) + f(2x) + 10x.$$
Using (4) repeatedly, this simplifies to $12xf(2x) = 0$ for all $x$. This implies that $f(2x) = 0$ for all $x \neq 0$. Taking $x = 1/2$, we obtain $f(1) = 0$, a contradiction. Thus there is no solution in this case.

Case 2: $(f(2), f(1)) = (0, 5)$.
Again $y = 1$ in (1) gives
$$f(x+1) + 5f(x) = 5(x+1) + f(x),$$
which simplifies to $f(x+1) = 5(x+1)$. This implies $f(x) = 5x$ for all $x$. Substituting into (1), we get $5(x+y) + 25xy = (1+y)5x + (1+x)5y + 5xy$, which simplifies to $5(x+y) + 25xy = 5x + 5xy + 5y + 5xy + 5xy$, or $5(x+y) + 25xy = 5(x+y) + 15xy$, which is $25xy = 15xy$, so $10xy = 0$ for all $x, y$, which is not possible. Thus, there is no solution in this case.

Case 3: $(f(2), f(1)) = (6, 2)$.
Again $y = 1$ in (1) gives
$$f(x+1) = f(x) + 2(x+1).$$
Replacing $y$ by $2$ in (1), we also obtain
$$f(x+2)+6f(x) = 3f(x) + 6(x+1) + f(2x).$$
Using the expression for $f(x+1)$, this reduces to
$$4f(x) = 2x + f(2x), \quad (5)$$
for all $x \in \mathbb{R}$. Taking $y = x$ in (1), we also get $f(2x)+f(x)^2 = 2(x+1)f(x)+f(x^2)$. This reduces to
$$f(x)^2 - f(x^2) = 2xf(x) - 2f(x) + 2x.$$
Replacing $x$ by $2x$ and repeatedly using (5), we obtain $f(x) = x^2+x$ for all $x$. It is easy to check that this is a solution.

Case 4: $(f(2), f(1)) = (6, 3)$ and $f(2) = 6$.
As in the earlier cases, we put $y = 1$ in (1) to get
$$f(x+1) = 3(x+1)$$
for all real $x$. This implies that $f(x) = 3x$, which is in fact a solution.
Thus there are three solutions: $f(x) \equiv 0$; $f(x) = x^2+x$; and $f(x) = 3x$.