The answer is f(x)=C1x+xC2 with arbitrary constants C1 and C2.
Solution 1. Fix a real number a>1, and take a new variable t. For the values f(t),f(t2),f(at) and f(a2t2), the relation of the assumption provides a system of linear equations:
x=y=t:(t+t1)f(t)=f(t2)+f(1)(1)
x=at,y=at:(at+ta)f(at)=f(t2)+f(a2)(2)
x=a2t,y=t:(a2t+a2t1)f(t)=f(a2t2)+f(a21)(3)
x=y=at:(at+at1)f(at)=f(a2t2)+f(1)(4)
In order to eliminate f(t2), take the difference of Eq. (1) and Eq. (2); from Eq. (3) and Eq. (4) eliminate f(a2t2); then by taking a linear combination, eliminate f(at) as well:
(t+t1)f(t)−(at+ta)f(at)=f(1)−f(a2) and(a2t+a2t1)f(t)−(at+at1)f(at)=f(a21)−f(1), so((at+at1)(t+t1)−(at+ta)(a2t+a2t1))f(t)=(at+at1)(f(1)−f(a2))−(at+ta)(f(a21)−f(1)).
Notice that on the left-hand side, the coefficient of f(t) is nonzero and does not depend on t:
(at+at1)(t+t1)−(at+ta)(a2t+a2t1)=a+a1−(a3+a31)<0.
After dividing by this fixed number, we get:
f(t)=C1t+tC2(5)
where the numbers C1 and C2 are expressed in terms of a, f(1), f(a2) and f(1/a2), and they do not depend on t.
The functions of the form Eq. (5) satisfy the equation:
(x+x1)f(y)=(x+x1)(C1y+yC2)=(C1xy+xyC2)+(C1xy+C2yx)=f(xy)+f(xy).
Solution 2. We start with an observation. If we substitute x=a=1 and y=an in the assumption, we obtain
f(an+1)−(a+a1)f(an)+f(an−1)=0.
For the sequence zn=an, this is a homogenous linear recurrence of the second order, and its characteristic polynomial is
t2−(a+a1)t+1=(t−a)(t−a1)
with two distinct nonzero roots, namely a and 1/a. As is well-known, the general solution is
zn=C1an+C2(a1)n
where the index n can be as well positive as negative. Of course, the numbers C1 and C2 may depend of the choice of a, so in fact we have two functions, C1 and C2, such that
f(an)=C1(a)⋅an+anC2(a) for every a=1 and every integer n.(1)
The relation Eq. (1) can be easily extended to rational values of n, so we may conjecture that C1 and C2 are constants, and whence
f(t)=C1t+tC2.
As it was seen in the previous solution, such function indeed satisfy the assumption.
a solution of the assumption. In order to make our formulas simpler,
define
f0(c)=f(x)−f(1)⋅x.
This function is another one satisfying the assumption and the extra
constraint f0(1)=0. Repeating the same argument on linear recur-
rences, we can write
f0(a)=K(a)an+anL(a)
with some functions K and L. By substituting n=0, we can see that
K(a)+L(a)=f0(1)=0 for every a.
Hence,
f0(an)=K(a)(an−an1).
Now take two numbers a>b>1 arbitrarily and substitute
x=(a/b)n and y=(ab)n in the assumption:
(bnan+anbn)f0((ab)n)=f0(a2n)+f0(b2n), so
(bnan+anbn)K(ab)((ab)n−(ab)n1)=K(a)(a2n−a2n1)+K(b)(b2n−b2n1),
or equivalently
K(ab)(a2n−a2n1+b2n−b2n1)=K(a)(a2n−a2n1)+K(b)(b2n−b2n1).(2)
K(ab)=K(a).
Then Eq. (2) reduces to K(a)=K(b). Hence, K(a)=K(b) for all a>b>1.
Fix a>1. For every x>0 there is some b and an integer n such that 1<b<a and x=bn. Then
f0(x)=f0(bn)=K(b)(bn−bn1)=K(a)(x−x1).
Hence, we have
f(x)=f0(x)+f(1)x=C1x+xC2
with
C1=K(a)+f(1) and C2=−K(a).