Answer: f(x)=x−x1, ∀x∈Q+.
We will prove several lemmas.
f(x)+f(y)=(f(x+y)+x+y1)(1−xy+f(xy))(∗)
Lemma 1. f(1)=0.
Proof: By putting x=y=1 to (*) we get 2f(1)=(f(2)+21)f(1).
Assume that f(1)=0. Then readily f(2)=23. By putting x=y=2 to (*) we get 3=2f(2)=(f(4)+41)(f(4)−3). Therefore, either f(4)=415 or f(4)=−1. By putting y=1 to (*) we get f(x)+f(1)=(f(x+1)+x+11)(1−x+f(x)). Taking x=2,3,4,5 in the last equation we get
23+f(1)=(f(3)+31)⋅21(1)
f(3)+f(1)=(f(4)+41)(f(3)−2)(2)
f(4)+f(1)=(f(5)+51)(f(4)−3)(3)
f(5)+f(1)=(f(6)+61)(f(5)−4)(4)
If f(4)=−1, (1) and (2) yield f(1)=−2719 and f(3)=2734, and (3) and (4) yield f(5)=27061, f(6)=−6114245. Finally by putting x=2,y=3 to (*) we get
f(2)+f(3)=(f(5)+51)(f(6)−5).
This relationship is not held for values of f(2),f(3),f(5),f(6) found above. Hence the only possibility is f(4)=415. In this case
we have
f(3)+f(1)=4(f(3)−2),23+f(1)=(f(3)+31)⋅21.
Solving last two equations we get f(1)=0. Done.
Lemma 2. f(2)=23.
Proof: Since f(1)=0, for all x∈Q+ we have
f(x)=(f(x+1)+x+11)(1−x+f(x))(5)
Taking x=2,3 in (5), we get
f(2)=(f(3)+31)(f(2)−1),f(3)=(f(4)+41)(f(3)−2).
By putting x=2,y=2 to (*) we get 2f(2)=(f(4)+41)(f(4)−3).
Last three equations yield a cubic equation in terms of t=f(4) as given below:
16t3−32t2−101t−15=0.
t=f(4) should be rational as the range of f is rational numbers.
The only rational root of this equation is t=415. Then it readily
follows that f(3)=38 and f(2)=23.
Lemma 3. f(n)=n−n1 for all positive integers n.
Proof: f(1)=0 for n=1. Proof for n≥2 readily follows from (5) by induction over n. Done.
Finally we prove that for all x=nm
f(m/n)=nm−mn(6)
Proof will be carried out by induction over m≥1. In the base case m=1 by putting y=x1 to (*) we get that for all x∈Q+
f(x)+f(x1)=0
and by Lemma 3 for all positive integers n we obtain the required formula
f(n1)=n1−n.
Now suppose that (6) is correct for m. By putting x=nm, y=n1 to (*) we get
f(m/n)+f(1/n)=(f(nm+1)+m+1n)(1−n2m+f(m/n2)).(7)
Putting f(m/n)=nm−mn and f(m/n2)=n2m−mn2 to (7) and
simplifying we get the required formula for m+1:
f(nm+1)=nm+1−m+1n.
We are done.