Problems of Ukrainian Authors

(Edward Turkevich)
Answer: the equation is true when $a = b, x = y = 1 - a, k = 1$.

From $a + kx \le 1$: $x \le \frac{1-a}{k}$ and from $b + \frac{x}{k} \le 1$: $x \le k(1-b)$.
So, $x^2 \le (1-a)(1-b)$ and $x \le \frac{1}{2}(2-a-b)$ (based on the inequality of arithmetic and geometric means of $1-a$ and $1-b$).
Similarly, $y^2 \le (1-a)(1-b)$ and $y \le \frac{1}{2}(2-a-b)$.

$$\begin{aligned} (a+x)^2 + (b+y)^2 &= a^2 + 2ax + x^2 + b^2 + 2by + y^2 \\ &\le a^2 + 2a \cdot \frac{1}{2}(2-a-b) + (1-a)(1-b) + b^2 + 2b \cdot \frac{1}{2}(2-a-b) + (1-a)(1-b) \\ &= a^2 + b^2 + (a+b)(2-a-b) + 2(1-a)(1-b) = 2, \end{aligned}$$
What was to be proven.

The equality is true when $1-a=1-b$, so $a=b, x=y=1-a, k=1$.