Problem 1903

Solution: All functions satisfying (*) are $f(x) = x$ and $f(x) = -x$. It is easy to verify that these two functions satisfy (*).
In what follows, all sets are multisets. Take $x = y = 0$ in (*) gives $\{f(1), f(-1)\} = \{1, -1\}$. First consider the case when $f(1) = 1$. We want to show that $f(x) = x$ for any $x \in \mathbb{R}$.
*Step 1:* Prove that $f(n) = n$ for $n \in \mathbb{Z}$.
We first show that $f(0) = 0$. Take $x = 0$ in (*), we have $\{f(1), f(yf(0) - 1)\} = \{-1, yf(f(0)) + 1\}$. If $f(0) \neq 0$, take $y = \frac{2}{f(0)}$; then the LHS is $\{1, 1\}$, which cannot contain the $-1$ on the RHS. So $f(0) = 0$.
Next, we prove inductively $f(n) = n$ for $n \in \mathbb{N}$. The case $n = 1$ is known. Assume that $f(m) = m$ holds for $m \le n$. Take $x = 1$ and $y = n$ in (*), we deduce
$$\{f(n + 1), f(n - 1)\} = \{n - 1, n + 1\}$$
From this, we deduce that $f(n + 1) = n + 1$, completing the inductive proof. Using a similar argument but taking $x = 1$ and $y = -n$ proves that $f(-n) = -n$ for $n \in \mathbb{N}$. This completes Step 1.
*Step 2:* Prove that $f$ is a bijection.
$f$ is surjective because when setting $y = 1$ in (*), the RHS contains $xf(f(1)) - 1 = x - 1$, which can take an arbitrary real number. By the form of its LHS, all real numbers are in the image of $f$.
Next, we prove $f(y_0) = 0$ implies $y_0 = 0$. Suppose not, namely $y_0 \neq 0$. Then taking $y = y_0$ in (*) gives
$$\{f(1), f(y_0f(x) - 1)\} = \{-1, y_0f(f(x)) + 1\}.$$
We must have $1 = f(1) = y_0f(f(x)) + 1$. So $f(f(x)) = 0$ holds for all $x$. This contradicts with $f(1) = 1$.
We next prove the injectivity of $f$. If $y_1 \neq y_2$ satisfy $f(y_1) = f(y_2) \neq 0$. Taking respectively $y = y_1$ and $y = y_2$ in (*) gives
$$\{f(xf(y_1) + 1), f(y_1f(x) - 1)\} = \{xf(f(y_1)) - 1, y_1f(f(x)) + 1\},$$
$$\{f(xf(y_2) + 1), f(y_2f(x) - 1)\} = \{xf(f(y_2)) - 1, y_2f(f(x)) + 1\}.$$

Note that the first elements on both sides of the two equations are the same, respectively.
If some $x_0 \in \mathbb{R}$ satisfies $f(x_0f(y_1) + 1) \neq x_0f(f(y_1)) - 1$, then
$$y_1f(f(x_0)) + 1 = f(xf(y_1) + 1) = f(xf(y_2) + 1) = y_2f(f(x_0)) + 1$$
From this, we get $y_1f(f(x_0)) = y_2f(f(x_0))$, thus $f(f(x_0)) = 0$, which implies $x_0 = 0$.
So if $x \neq 0$, then $f(xf(y_1) + 1) = xf(f(y_1)) - 1$. From this, we deduce that $f(x) = ax + b$ for some $a, b \in \mathbb{R}$ and any $x \neq 1$ is a linear function. But $f(n) = n$ for $n \in \mathbb{Z}$ implies that $a = 1$ and $b = 0$. So $f(x) = x$ is injective.
Step 3: Prove that for any $n \in \mathbb{Z}$ and $y \in \mathbb{R}$, we have
$$f\left(f\left(\frac{n}{f(y)}\right)\right) = \frac{n}{y} \qquad (1)$$
It is enough to consider the case when $n \neq 0$ and $y \neq 0$. Taking $x = \frac{n}{f(y)}$ in $(*)$ gives
$$\left\{n + 1, f\left(yf\left(\frac{n}{f(y)}\right) - 1\right)\right\} = \left\{n\frac{f(f(y))}{f(y)} - 1, yf\left(f\left(\frac{n}{f(y)}\right)\right) + 1\right\}. \quad (2)$$
Similarly, substituting $y$ for $x$ and $\frac{n}{f(y)}$ for $y$ in $(*)$ gives
$$\left\{n - 1, f\left(yf\left(\frac{n}{f(y)}\right) + 1\right)\right\} = \left\{n\frac{f(f(y))}{f(y)} + 1, yf\left(f\left(\frac{n}{f(y)}\right)\right) - 1\right\}. \quad (3)$$
If there exists some $y_0 \in \mathbb{R} \setminus \{0\}$ such that $n \neq y_0f\left(f\left(\frac{n}{f(y_0)}\right)\right)$, then combining (2) and (3) gives
$$n + 1 = n \frac{f(f(y_0))}{f(y_0)} - 1, \quad n - 1 = n \frac{f(f(y_0))}{f(y_0)} + 1.$$
Taking the differences of the two equations gives $2 = -2$, which is a contradiction.
Step 4: For $\alpha \in \mathbb{Q}$ and $y \in \mathbb{R}$, prove that $f(\alpha y) = \alpha f(y)$.
We may assume that $\alpha \neq 0$ and $y \neq 0$. Substituting $f\left(\frac{m}{f(y)}\right)$ with $m \in \mathbb{Z} \setminus \{0\}$ for $y$ in (1) gives
$$f\left(f\left(\frac{n}{f\left(f\left(\frac{m}{f(y)}\right)\right)}\right)\right) = \frac{n}{f\left(\frac{m}{f(y)}\right)}.$$
Plugging (1) into the denominator of the LHS gives
$$f\left(f\left(\frac{n}{m}y\right)\right) = \frac{n}{f\left(\frac{m}{f(y)}\right)}.$$
In the above equality, substituting $\frac{t}{f(y)}$ with $t \in \mathbb{Z} \setminus \{0\}$ for $y$ gives
$$\frac{nt/m}{y} = f\left(f\left(\frac{n}{m}\frac{t}{f(y)}\right)\right) = \frac{n}{f\left(\frac{m}{f\left(\frac{t}{f(y)}\right)}\right)}.$$

This is equivalent to $f(\frac{m}{f(\frac{t}{f(y)})}) = \frac{m}{t}y$. Taking $f$ on both sides gives
$$\frac{m}{t} f(y) = f\left(f\left(\frac{m}{f(\frac{t}{f(y)})}\right)\right) = f\left(\frac{m}{t}y\right).$$
This completes Step 4.
Step 5: Prove that $f(y+a) = f(y) + a$ for all $y \in \mathbb{R}$ and $a \in \mathbb{Q}$.
Taking $x = \frac{1}{r} \in \mathbb{Q} \setminus \{0\}$ in (*) gives
$$\{f(\frac{1}{r}f(y) + 1), f(\frac{1}{r}y - 1)\} = \{\frac{1}{r}f(f(y)) - 1, \frac{1}{r}y + 1\}.$$
Using Step 4 and multiplying both sides by $r$ gives
$$\{f(f(y) + r), f(y - r)\} = \{f(f(y)) - r, y + r\}. \quad (4)$$
We want to prove that $f(y-r) = f(f(y)) - r$. Suppose that there exist some $y_0 \in \mathbb{R}$ and $r_0 \in \mathbb{R} \setminus \{0\}$ such that $f(y_0 - r_0) = y_0 + r_0$. Taking $y = y_0 - r_0$ and $r = -2r_0$ in (4) gives
$$\{f(f(y_0 - r_0) - 2r_0), f(y_0 - r_0 + 2r_0)\} = \{f(f(y_0 - r_0)) + 2r_0, y_0 - 3r_0\}.$$
This is the same as
$$\{y_0 + r_0, f(y_0 + r_0)\} = \{f(y_0 + r_0) + 2r_0, y_0 - 3r_0\}.$$
Since $y_0 + r_0 \neq y_0 - 3r_0$, we must have $y_0 + r_0 = f(y_0 + r_0) + 2r_0$, i.e. $f(y_0 + r_0) = y_0 - r_0$. But then the two sides of the equality becomes $\{y_0 + r_0, y_0 - r_0\} = \{y_0 + r_0, y_0 - 3r_0\}$. They are obviously not equal, giving a contradiction.
So $f(y - r) = f(f(y)) - r$ must hold for all $y \in \mathbb{R}$ and $r \in \mathbb{Q} \setminus \{0\}$. Taking two different values of $r$ and taking the difference gives $f(y + a) = f(y) + a$ for all $y \in \mathbb{R}$ and $a \in \mathbb{Q}$.
Step 6: Prove that $f(y) = y$ for $y \in \mathbb{R}$.
Combining Step 5 and (4) gives
$$\{f(f(y)) + r, f(y) - r\} = \{f(f(y)) - r, y + r\}.$$
Taking the sum of the two elements on both sides of the equality, we deduce that $f(f(y)) + f(y) = f(f(y)) + y$, i.e. $f(y) = y$.
Now we consider the case when $f(1) = -1$. We aim to prove that $f(x) = -x$ ($x \in \mathbb{R}$).
Step 1: Prove that $f(n) = -n$ ($n \in \mathbb{Z}$).

Taking $y = 1$ and respectively $x = 1, -1$ in (*) gives
$$\{f(0), f(-2)\} = \{0, 2\}, \quad \{f(2), f(0)\} = \{-2, 0\}.$$
From this, we deduce that $f(0) = 0$, $f(2) = -2$, $f(-2) = 2$.
Next, we inductively prove that $f(n) = -n$. When $|n| = 1, 2$, this is already proved. Suppose that we have proved this when $|n| \le n_0$ ($n_0 \ge 2$). Taking $y = 1$ and $x = n_0, -n_0$ in (*) gives
$$\begin{aligned} \{f(-n_0 + 1), f(f(n_0) - 1)\} &= \{n_0 - 1, f(f(n_0)) + 1\}, \\ \{f(n_0 + 1), f(f(-n_0) - 1)\} &= \{-n_0 - 1, f(f(-n_0)) + 1\} \end{aligned}$$
We deduce from this that $f(n_0 + 1) = -n_0 - 1$ and $f(-n_0 - 1) = n_0 + 1$. This completes the induction in Step 1.
Step 2: Prove that $f(x) = -x$ holds for all $x \in \mathbb{R}$.
First, as in the $f(1) = 1$ case, we can prove that $f$ is surjective.
For a nonzero integer $n$ and $z \in \mathbb{R}\{0\}$, taking $x = \frac{n}{f(z)}$, $y = z$, and taking $x = z$, $y = \frac{n}{f(z)}$ in (*), we obtain
$$\begin{aligned} \{-n-1, f\left(zf\left(\frac{n}{f(z)}\right) - 1\right)\} &= \{n\frac{f(f(z))}{f(z)} - 1, zf\left(f\left(\frac{n}{f(z)}\right)\right) + 1\}, & (5) \\ \{-n+1, f\left(zf\left(\frac{n}{f(z)}\right) + 1\right)\} &= \{n\frac{f(f(z))}{f(z)} + 1, zf\left(f\left(\frac{n}{f(z)}\right)\right) - 1\}. \end{aligned}$$
If for some $z = z_0 \in \mathbb{R}\{0\}$ we have $f(f(z_0)) \ne f(z_0)$, then by (5), we deduce that
$$-n-1 = zf\left(f\left(\frac{n}{f(z)}\right)\right) + 1, \quad -n+1 = zf\left(f\left(\frac{n}{f(z)}\right)\right) - 1.$$
Taking the difference of the two equalities gives $2 = -2$. This is a contradiction!
So $f(f(z_0)) = -f(z_0)$. But $f$ is surjective. So $f(x) = -x$ holds for all $x \in \mathbb{R}\{0\}$. Moreover, $f(0) = 0$ is known. This completes the proof of $f(x) = -x$ when $f(1) = -1$.
Combining the above two cases, the needed functions are $f(x) = x$ ($x \in \mathbb{R}$) and $f(x) = -x$ ($x \in \mathbb{R}$).