First, when x=z≡x(modf(x)), we have f(z)≡x(modf(x)). Then substituting x+f(x) and f(x) for x,y in the original equation gives
f(x+f(x)+f(x)f(x+f(x)))=x+f(x)+f(x+f(x))f(f(x))
Taking both sides mod f(x), we obtain an important relation
f(x)∣xf(f(x))(1)
In particular, we have
f(x+yf(x))∣(x+yf(x))f(f(x+yf(x)))
that is,
x+f(x)f(y)∣(x+yf(x))(x+f(f(y))f(x))
which is equivalent to
x+f(x)f(y)∣f(x)2(f(y)−y)(f(f(y))−f(y))(2)
Suppose f(1)=1. We claim that there exists a large number a such that
(a,f(a))=1(3)
This is because, by a simple mathematical induction, we have
f(1+f(1)+⋯+f(1)n)=1+f(1)+⋯+f(1)n−1+f(1)n+1
and their greatest common divisor is
(1+f(1)+⋯+f(1)n,f(1)−1)
So as long as n satisfies (n+1,f(1)−1)=1, (3) will hold. Clearly n can be taken very large,
and the corresponding 1+f(1)+⋯+f(1)n is the a we want.
Now, taking x=a in equation (2), we have
a+f(a)f(y)∣(f(y)−y)(f(f(y))−f(y))
Hence for all y, we have
f(f(y))=f(y)(4)
Substituting f(x) for x in the original condition:
f((y+1)f(x))=(f(y)+1)f(x)(5)
Then substituting f(y)−1 for y (this is legal, since 1 is not in the range of f) gives
f(f(y)f(x))=(f(f(y)−1)+1)f(x)
Therefore there exists a constant c satisfying
f(x)=c(f(f(x)−1)+1)(6)
Taking x=1+f(1) in the above equation gives
1+f(1)2=c(f(f(1)2)+1)
Note that by using (4)
f(f(1)2)=(f(f(1)−1)+1)f(1)=cf(1)2
Substituting back gives c=1. Finally, suppose the smallest fixed point of f is k>1 (which exists by (3) and the well-ordering principle); then by (5) we get that k−1 is also a fixed point, a contradiction.
Combining the above, we must have f(1)=1, hence
f(x)=x,∀x∈N
Substituting back to verify, this clearly holds.