BxMO/EGMO Team Selection Test

We are going to prove that exactly all pairs $(a, b)$ with $\gcd(a, b) = 1$ and with $a + b$ odd satisfy this.

First assume that $\gcd(a, b) = n \neq 1$. Consider the function
$$g(x) = \begin{cases} x + 1 & \text{if } \lfloor x \rfloor \not\equiv 0 \mod n \\ x + 1 - n & \text{if } \lfloor x \rfloor \equiv 0 \mod n \end{cases}$$
This function is unequal to $\mathrm{id}_R$ because $n \neq 1$. Then $\lfloor g(x) \rfloor \equiv \lfloor x \rfloor + 1 \mod n$. So the numbers $\lfloor x \rfloor$, $\lfloor g(x) \rfloor$, $\lfloor g(g(x)) \rfloor$, ..., $\lfloor g^{n-1}(x) \rfloor$ represent all residue classes modulo $n$. Hence, $g^n(x) = x + \underbrace{1 + \cdots + 1}_{n \text{ times}} - n = x$. With induction, it follows that $g^{cn}(x) = x$ for all natural $c$, so since $n \mid a, b$, $g^a(x) = x$ and $g^b(x) = x$ also holds. So
$$g^a(x)g^b(y) + g^b(x)g^a(y) = xy + xy = 2xy.$$
So the function $f = g \neq \mathrm{id}_R$ satisfies the functional equation in this case.

We now consider the case where $a + b$ is even. Then take the function $h(x) = -x$. Then it follows from simple induction that $h^c(x) = (-1)^c x$. So
$$h^a(x)h^b(y) + h^b(x)h^a(y) = (-1)^{a+b}xy + (-1)^{a+b}xy = 2xy.$$
So the function $f = h \neq \mathrm{id}_R$ satisfies the functional equation.

Now assume that $\gcd(a, b) = 1$ and that $a + b$ is odd. With $x = y$ we see
$$f^a(x)f^b(x) = x^2.$$
If we multiply the function equation by $f^a(x)f^a(y)$, we get
$$\begin{align*} 2(x f^a(y))(y f^a(x)) &= f^a(x) f^b(y) f^a(x) f^a(y) + f^b(x) f^a(y) f^a(x) f^a(y) \\ &= \left(\left(f^a(x)\right)^2 \underbrace{f^a(y) f^b(y)}_{=y^2} + \left(\left(f^a(y)\right)^2 \underbrace{f^a(x) f^b(x)}_{=x^2}\right)\right) \\ &= (y f^a(x))^2 + (x f^a(y))^2 \end{align*}$$
This is the equality case of the inequality of the arithmetic and geometric mean. Even better, we see that we can rewrite it as
$$(y f^a(x) - x f^a(y))^2 = 0.$$
So we conclude that $y f^a(x) = x f^a(y)$. With $y=1$, we see $f^a(x) = c_1 x$ for some $c_1 \in \mathbb{R}$. Analogously, multiplication by $f^b(x)f^b(y)$ gives $f^b(x) = c_2 x$ for a certain $c_2 \in \mathbb{R}$. If either constant were equal to 0, then the left side of the functional equation is always equal to 0, but the right side is not. So both constants are not equal to 0. Since $\text{gcd}(a, b) = 1$ there exist integers $p$ and $q$ such that $ap + bq = 1$. Assume that $p$ is positive and $q$ is negative and write $r = -q$. Then $ap = 1 + rb$ with $p$ and $r$ positive holds. We see
$$\begin{align*} c_1^p x &= (f^a)^p(x) = f^{ap}(x) \\ &= f^{1+rb}(x) \\ &= f((f^b)^r(x)) \\ &= f(c_2^r x) \end{align*}$$
Hence, $f(x) = \frac{c_1^p}{c_2^r} x = dx$ for some $d \in \mathbb{R}$. If we fill in this function, we see $2d^{a+b}xy = 2xy$, so $d^{a+b} = 1$. It follows $d = 1$ because $a+b$ is odd. This means that $f(x) = x$ is the only function that potentially satisfies. It is easy to see that this function actually satisfies, so all pairs $(a, b)$ of natural numbers for which $f(x) = x$ is the only function $f: \mathbb{R} \to \mathbb{R}$ that satisfies
$$f^a(x)f^b(y) + f^b(x)f^a(y) = 2xy$$
for all $x, y \in \mathbb{R}$ are exactly the pairs $(a, b)$ for which $a + b$ is odd and $\text{gcd}(a, b) = 1$. $\square$