Solution:
Answer: f(x)=x1.
It is straightforward to check that f(x)=x1 works. We then focus on proving that there are no other solutions. Let P(x,y) denote the given functional equation. First note that for all x,y∈Q+, f(x)>f(x+y), so f is strictly decreasing and hence injective. Now, for all x,y∈Q+,
P(x,y)P(x,x2f(y))⟹f(x)=f(x+y)+f(x+x2f(y))⟹f(x)=f(x+x2f(y))+f(x+x2f(x2f(y)))
Equating these two equations gives
f(x+x2f(x2f(y)))x+x2f(x2f(y))f(x2f(y))=f(x+y)=x+y=x2y
for all x,y∈Q+. In particular, plugging in x=1 gives f(f(y))=y, and replacing y with f(y) gives f(x2y)=x2f(y) for all x,y∈Q+. There are two ways to finish.
Finish 1: The above implies that f(x2)=x2f(1) for all x∈Q+. Since {x2:x∈Q+} is dense in Q+ and f is decreasing we find that f(x)=c/x for some constant c. Plugging back in now gives c=1.
Finish 2: For all x∈Q+,
P(x,169x)⟹f(x)=f(1625x)+f(x+x2f(169x)).=2516f(x)+f(x+916x2f(x))
Hence, we have
f(925x)=259f(x)925x=f(x+916x2f(x))=x+916x2f(x)⟹f(x)=x1
as desired.