Problem 1795

1. Let $P(x, y)$ denote the given assertion:
$$P(x, y) : f(x + y) \leq y f(x) + f(f(x))$$
for all real numbers $x$ and $y$.

2. First, consider $P(x, 0)$:
$$P(x, 0) : f(x + 0) \leq 0 \cdot f(x) + f(f(x)) \implies f(x) \leq f(f(x))$$
This implies:
$$f(f(x)) \geq f(x)$$
Call this result (1).

3. Next, consider $P(x, f(x) - x)$:
$$P(x, f(x) - x) : f(x + (f(x) - x)) \leq (f(x) - x) f(x) + f(f(x)) \implies f(f(x)) \leq (f(x) - x) f(x) + f(f(x))$$
Simplifying, we get:
$$0 \leq (f(x) - x) f(x)$$
This implies:
$$f(x) (f(x) - x) \geq 0$$
Call this result (2).

4. Claim: $f(x) \leq 0$ for all $x \in \mathbb{R}$.

Proof:
- For an arbitrary real $z$, consider $P(x, f(z) - x)$:
$$P(x, f(z) - x) : f(x + (f(z) - x)) \leq (f(z) - x) f(x) + f(f(x)) \implies f(f(z)) \leq (f(z) - x) f(x) + f(f(x))$$
Interchanging variables $x$ and $z$ and adding the inequalities, we get:
$$f(f(z)) + f(f(x)) \leq (f(z) - x) f(x) + f(f(x)) + (f(x) - z) f(z) + f(f(z))$$
Simplifying, we get:
$$2 f(f(z)) \leq f(z) f(x) - x f(x) + f(x) f(z) - z f(z)$$
$$2 f(f(z)) \leq 2 f(z) f(x) - x f(x) - z f(z)$$
$$2 f(z) f(x) \geq x f(x) + z f(z)$$
Call this result (3).

- Suppose there exists a real number $t_0 \geq 0$ for which $f(t_0) > 0$. Consider $P(t_0, y)$:
$$P(t_0, y) : f(y + t_0) \leq y f(t_0) + f(f(t_0))$$
Letting $y \to -\infty$, we get:
$$\lim_{y \to -\infty} f(y + t_0) = -\infty$$
This implies $f(x) \to -\infty$ as $x \to -\infty$.

- Putting $z = t_0$ and letting $x \to -\infty$ in (3), we get:
$$0 > 2 f(x) f(t_0) \geq x f(x) + t_0 f(t_0) \geq 0$$
This is a contradiction.

- Now assume $t_1, t_2 < 0$ are reals for which $f(t_i) > 0$ for $i \in \{1, 2\}$. Without loss of generality, let $t_1 < t_2$. Consider $P(t_2, t_1 - t_2)$:
$$P(t_2, t_1 - t_2) : f(t_1) \leq (t_1 - t_2) f(t_2) + f(f(t_2))$$
Since $t_1 - t_2 < 0$ and $f(t_2) > 0$, we get:
$$f(t_1) \leq 0 + 0 = 0$$
This is a contradiction.

- Finally, if $f(x) > 0$ holds for only one point $x = t_3$, then $f(t_3) > 0 > t_3$. From (1) and the previous paragraph, we get:
$$0 \geq f(f(t_3)) \geq f(t_3) > 0$$
This is a contradiction.

Hence, the claim must hold: $f(x) \leq 0$ for all $x \in \mathbb{R}$.

5. Using (2), we get $x \to f(t)$ implies:
$$f(f(t)) (f(f(t)) - f(t)) \geq 0$$
Since $f(f(t)) \geq f(t)$ by (1), we see that $f(f(t)) \neq 0 \implies f(f(t)) = f(t)$.

6. Define:
$$\mathcal{S} := \{ x \in \mathbb{R} : f(f(x)) \neq 0 \}$$
For any $x, z \in \mathcal{S}$, replacing $(x, z) \to (f(x), f(z))$ in (3), we get:
$$2 f(x) f(t) = 2 f(f(x)) f(f(t)) \geq f(x) f(f(x)) + f(z) f(f(z)) = f(x)^2 + f(z)^2$$
This implies:
$$(f(x) - f(z))^2 \leq 0$$
So:
$$f(x) = f(z)$$

7. We are left with showing rather straightforward details to complete our proof. Note that $f \not\equiv 0$ since otherwise, there is nothing to see.

Case 1: $\mathbb{R} \neq \mathcal{S}$.

- Note that there exists $z_0$ such that $f(f(z_0)) = 0$. Put $z = f(z_0)$ in (3) to get:
$$0 = 2 f(f(z_0)) f(x) \geq f(z_0) f(f(z_0)) + x f(x) = x f(x)$$
So for all $x < 0$, we must have $f(x) = 0$. To see $f(0) = 0$, simply observe that:
$$0 = f(f(0)) \leq f(0) \leq 0$$

Case 2: $\mathbb{R} = \mathcal{S}$.

- Note that $f(f(x))$ is the constant function now. Therefore, set $y = f(t) - x$ for any $t \in \mathbb{R}$ to get:
$$0 \leq f(x) (f(t) - x)$$
Thus, for all $x < f(t) \leq 0$, we have $f(x) = 0$. Now, take $y > 0$ and $x < f(t) - y$ to get:
$$0 = f(x + y) \leq y f(x) + f(f(x)) = f(0) \leq 0$$
Which gives $f(0) = 0$. This contradicts the hypothesis of this case, since $f(f(0)) = 0$.

Hence, the conclusion must hold. $\blacksquare$