Denote the original relation by (*), and define P(a,b) as the property obtained by substituting x=a,y=b into the given condition on the function f. We analyze this through the following steps:
(A). Applying P(x,1) gives f(x+f(1))=f(1)+f(f(x)). From this we obtain f(f(1−f(1)))=0. Organizing this as follows:
f(1−f(1))=a,f(a)=0.(1)
Also, applying P(a,a) and using (1) gives
f(a2)=f(0).(2)
Then applying P(0,a2) and P(0,x), we can successively derive
f(0)=0,f(f(x))=f(x),∀x∈R.(3)
(B). Suppose there exists t=0,1 such that f(t)=0. Then from P(x,t) and (3) we obtain f(xt)=tf(x),∀x∈R. Then from P(tx,ty)−tP(x,y) we derive (t2−t)(f(xy)−yf(x))=0, and setting x=1 gives f(y)=f(1)y, which combined with condition (*) gives f(1)=0 or f(1)=1. Thus we obtain two possible functions: f(y)=0 or f(y)=y,∀y∈R. Clearly f(y)=0 satisfies condition (*), while f(y)=y does not satisfy the condition in this case. (Because f(t)=0,∀t=0.)
(C). Excluding the case in (B), we now assume f(t)=0,∀t∈R−{0,1}. Then (1) gives f(1)=0 or f(1)=1. If f(1)=0, then from P(1,y) we obtain f(1+f(y))=0. Therefore, f(y)=−1,∀y∈R−{0,1}, but this clearly contradicts (*). Hence f(1)=1, and we have the following results:
f(0)=0,f(1)=1;f(t)=0,∀t=0.(4)
From P(x,1) we obtain f(x+1)=f(x)+1. Then from P(x,y+1)−P(x,y) we obtain f(x+xy)=f(x)+f(xy). Therefore, when x=0, for any real number z we may set y=xz to obtain
f(x+z)=f(x)+f(z).(5)
Also noting that f(0)=0, this result clearly holds for all real numbers x,z. Then from P(x,1) we can further obtain f(f(x))=f(x). From this result together with applying the property (5) into (*), we deduce that f(xy)=yf(x). Finally, from this equation combined with the argument in (B), we must obtain f(y)=y,∀y∈R. Clearly this equation satisfies the given condition.
In summary, the functions satisfying the condition are f(x)=0,∀x∈R and f(x)=x,∀x∈R.