Answer: f(x)=C1x+xC2 with arbitrary constants C1 and C2.
Solution 1. Fix a real number a>1, and take a new variable t. For the values f(t),f(t2), f(at) and f(a2t2), the relation (1) provides a system of linear equations:
x=y=t:x=at,y=at:x=a2t,y=t:x=y=at:(t+t1)f(t)(at+ta)f(at)=f(t2)+f(t2)+f(a2)(a2t+a2t1)f(t)=f(a2t2)+f(a21)(at+at1)f(at)=f(a2t2)+f(1)
In order to eliminate f(t2), take the difference of (2a) and (2b); from (2c) and (2d) eliminate f(a2t2); then by taking a linear combination, eliminate f(at) as well:
(t+t1)f(t)−(at+ta)f(at)=f(1)−f(a2) and (a2t+a2t1)f(t)−(at+at1)f(at)=f(1/a2)−f(1), so ((at+at1)(t+t1)−(at+ta)(a2t+a2t1))f(t)=(at+at1)(f(1)−f(a2))−(at+ta)(f(1/a2)−f(1))
Notice that on the left-hand side, the coefficient of f(t) is nonzero and does not depend on t :
(at+at1)(t+t1)−(at+ta)(a2t+a2t1)=a+a1−(a3+a31)<0.
After dividing by this fixed number, we get
f(t)=C1t+tC2
where the numbers C1 and C2 are expressed in terms of a,f(1),f(a2) and f(1/a2), and they do not depend on t.
The functions of the form (3) satisfy the equation:
(x+x1)f(y)=(x+x1)(C1y+yC2)=(C1xy+xyC2)+(C1xy+C2yx)=f(xy)+f(xy).
Solution 2. We start with an observation. If we substitute x=a=1 and y=an in (1), we obtain
f(an+1)−(a+a1)f(an)+f(an−1)=0.
For the sequence zn=an, this is a homogeneous linear recurrence of the second order, and its characteristic polynomial is t2−(a+a1)t+1=(t−a)(t−a1) with two distinct nonzero roots, namely a and 1/a. As is well-known, the general solution is zn=C1an+C2(1/a)n where the index n can be as well positive as negative. Of course, the numbers C1 and C2 may depend of the choice of a, so in fact we have two functions, C1 and C2, such that
f(an)=C1(a)⋅an+anC2(a) for every a=1 and every integer n.
The relation (4) can be easily extended to rational values of n, so we may conjecture that C1 and C2 are constants, and whence f(t)=C1t+tC2. As it was seen in the previous solution, such functions indeed satisfy (1).
The equation (1) is linear in f; so if some functions f1 and f2 satisfy (1) and c1,c2 are real numbers, then c1f1(x)+c2f2(x) is also a solution of (1). In order to make our formulas simpler, define
f0(x)=f(x)−f(1)⋅x.
This function is another one satisfying (1) and the extra constraint f0(1)=0. Repeating the same argument on linear recurrences, we can write f0(a)=K(a)an+anL(a) with some functions K and L. By substituting n=0, we can see that K(a)+L(a)=f0(1)=0 for every a. Hence,
f0(an)=K(a)(an−an1).
Now take two numbers a>b>1 arbitrarily and substitute x=(a/b)n and y=(ab)n in (1):
(bnan+anbn)f0((ab)n)(bnan+anbn)K(ab)((ab)n−(ab)n1)K(ab)(a2n−a2n1+b2n−b2n1)=f0(a2n)+f0(b2n), so =K(a)(a2n−a2n1)+K(b)(b2n−b2n1), or equivalently =K(a)(a2n−a2n1)+K(b)(b2n−b2n1).
By dividing (5) by a2n and then taking limit with n→+∞ we get K(ab)=K(a). Then (5) reduces to K(a)=K(b). Hence, K(a)=K(b) for all a>b>1.
Fix a>1. For every x>0 there is some b and an integer n such that 1<b<a and x=bn. Then
f0(x)=f0(bn)=K(b)(bn−bn1)=K(a)(x−x1).
Hence, we have f(x)=f0(x)+f(1)x=C1x+xC2 with C1=K(a)+f(1) and C2=−K(a).