Let P(x,y) denote the assertion
f(x+f(y+1))+f(xy)=f(x+1)(f(y)+1).
If f is a constant c, we have 2c=c(c+1). This implies c=0 or c=1. Hence, there are two constant solutions, f≡0 and f≡1.
P(0,y):f(f(y+1))+f(0)=f(1)(f(y)+1),(1)
P(x−1,0):f(x+f(1)−1)=f(x)(f(0)+1)−f(0).(2)
Assume f(1)=1. Then (2) becomes f(x)=f(x)(f(0)+1)−f(0), or equivalently, (f(x)−1)f(0)=0. Hence f(0)=0. Therefore, (1) becomes
f(f(y+1))=f(y)+1.(3)
Setting y=−1,−2,−3,… in (3), we conclude inductively that f(x)=x for all x≤0. In addition,
P(x,−1):f(x)+f(−x)=0,
so f(x)=x for all x∈Z, which is clearly a solution. Assume now that f(1)=1 and set c=f(1)−1=0. If f(0)+1=0, then (2) would imply that f is constant. Therefore, it must be h=f(0)+1=0. Equation (2) can now be rewritten as
f(x+c)=hf(x)−h+1.(4)
Next we consider
P(x,1):f(x+f(2))+f(x)=f(x+1)(f(1)+1)(5)
P(x+c,1):f(x+c+f(2))+f(x+c)=f(x+c+1)(f(1)+1)(6)
Using (4), we find that (6) is equivalent to
h(f(x+f(2))−h+1+hf(x)−h+1=(hf(x+1)−h+1)(f(1)+1).
Subtracting h times (5) we get
2(1−h)=(1−h)(f(1)+1),
hence, either h=1 or f(1)+1=2. The latter is impossible due to our initial assumption that f(1)=1. So h=1, i.e., f(0)=0. Now equation (4) becomes f(x+c)=f(x) and, as c=0, f is periodic. Let M=maxf and −m=minf. Note M,m≥0 because f(0)=0.
Choose x,y∈Z such that f(x+1)=f(y)=M. Then, the right-hand side of P(x,y) is equal to M(M+1), while the left-hand side is
f(x+f(y+1))+xy≤M+M=2M.
So M(M+1)≤2M, and thus M=0 or M=1.
Similarly, if we now choose x,y∈Z such that f(x+1)=f(y)=−m, we find
−m(−m+1)=m2−m=f(x+f(y+1))+xy≤2M≤2,
which implies m≤2. So f(x)∈{−2,−1,0,1} for all integers x. We analyze the possibilities for f(1):
* f(1)=−2: setting y=1 in (1) we get f(f(2))=2, absurd.
* f(1)=0: in this case, c=−1 and f is constant. Absurd.
* f(1)=1: previously excluded.
* f(1)=−1: in this case, c=−2, so f is 2-periodic, with f(x)=f(0)=0 for all even x, and f(x)=f(1)=−1 for all odd x. However, P(1,1) gives f(1+f(2))+f(1)=f(2)(f(1)+1), so −2=0, a contradiction.
In conclusion, the only solutions are f≡0,f≡1, and f(x)=x for all x.