Problem 1736

First Solution. (By Richard Stong) From the condition, at least one of $a, b$, and $c$ does not exceed $1$, say $a \le 1$. Then
$$ab + bc + ca - abc = a(b + c) + bc(1 - a) \ge 0.$$
To obtain equality, we have $a(b + c) = bc(1 - a) = 0$. If $a = 1$, then $b + c = 0$ or $b = c = 0$, which contradicts the given condition $a^2 + b^2 + c^2 + abc = 4$. Hence $1 - a \ne 0$ and only one of $b$ and $c$ is $0$. Without loss of generality, say $b = 0$. Therefore $b + c > 0$ and $a = 0$. Plugging $a = b = 0$ back into the given condition gives $c = 2$. By permutation, the lower bound holds if and only if $(a, b, c)$ is one of the triples $(2, 0, 0)$, $(0, 2, 0)$, and $(0, 0, 2)$.

Now we prove the upper bound. Let us note that some two of the three numbers $a, b$, and $c$ are both greater than or equal to $1$ or less than or equal to $1$. Without loss of generality, we assume that the numbers with this property are $b$ and $c$. Then we have
$$(1 - b)(1 - c) \ge 0. \qquad (1)$$
The given equality $a^2 + b^2 + c^2 + abc = 4$ and the inequality $b^2 + c^2 \ge 2bc$ imply
$$a^2 + 2bc + abc \le 4, \quad \text{or} \quad bc(2 + a) \le 4 - a^2.$$
Dividing both sides of the last inequality by $2 + a$ yields
$$bc \le 2 - a. \qquad (2)$$
Combining (1) and (2) gives
$$\begin{aligned} ab + bc + ac - abc &\le ab + 2 - a + ac(1 - b) \\ &= 2 - a(1 + bc - b - c) \\ &= 2 - a(1 - b)(1 - c) \le 2, \end{aligned}$$
as desired.

The last equality holds if and only if $b = c$ and $a(1 - b)(1 - c) = 0$. Hence, equality for the upper bound holds if and only if $(a, b, c)$ is one of the triples $(1, 1, 1)$, $(0, \sqrt{2}, \sqrt{2})$, $(\sqrt{2}, 0, \sqrt{2})$, and $(\sqrt{2}, \sqrt{2}, 0)$.

Second Solution. (by Oaz Nir) We prove only the upper bound here. Either two of $a, b, c$ are less than or equal to $1$, or two are greater than or equal to $1$. Assume $b$ and $c$ have this property. Then
$$b + c - bc = 1 - (1 - b)(1 - c) \le 1. \qquad (3)$$
Viewing the given equality as a quadratic equation in $a$ and solving for $a$ yields
$$a = \frac{-bc \pm \sqrt{(b^2 - 4)(c^2 - 4)}}{2}.$$
Note that
$$(b^2 - 4)(c^2 - 4) = b^2c^2 - 4(b^2 + c^2) + 16$$
$$\leq b^2c^2 - 8bc + 16 = (4 - bc)^2.$$
For the given equality to hold, we must have $b, c \leq 2$ so that $4 - bc \geq 0$. Hence,
$$a \leq \frac{-bc + |4 - bc|}{2} = \frac{-bc + 4 - bc}{2} = 2 - bc,$$
or
$$2 - bc \geq a. \qquad (4)$$
Combining (3) and (4) gives
$$2 - bc \geq a(b + c - bc) = ab + ac - abc,$$
or
$$ab + ac + bc - abc \leq 2,$$
as desired.

Third Solution. We prove only the upper bound here. Define functions $f, g$ as
$$f(x, y, z) = x^2 + y^2 + z^2 + xyz = (x + y)^2 + z^2 - (2 - z)xy,$$
$$g(x, y, z) = xy + yz + zx - xyz = z(x + y) + (1 - z)xy$$
for all nonnegative numbers $x, y, z$. Observe that if $z \leq 1$, then both $f$ and $g$ are unbounded, increasing functions of $x$ and $y$.

Assume that $f(a, b, c) = 4$ and, without loss of generality, that $a \geq b \geq c \geq 0$. Then $c \leq 1$.
Let $a' = (a+b)/2$. Because $a+b = a'+a'$ and $ab \leq (\frac{a-b}{2})^2 + ab = a'^2$, we have
$$f(a', a', c) \leq f(a, b, c) = 4 \quad \text{and} \quad g(a', a', c) \geq g(a, b, c).$$
Now increase $a'$ to $e \geq 0$ such that $f(e, e, c) = 4$. Note that $g(e, e, c) \geq g(a', a', c)$. It suffices to prove that $g(e, e, c) \leq 2$.

Since $f(e, e, c) = 2e^2 + c^2 + e^2c = 4$, $e^2 = (4 - c^2)/(2 + c) = 2 - c$.
We obtain that
$$\begin{aligned} g(e, e, c) - 2 &= 2ec + (1-c)e^2 - 2 \\ &\le 2e(2-e^2) + (e^2-1)e^2 - 2 \\ &= e^4 - 2e^3 - e^2 + 4e - 2 \\ &= (e-1)^2(e^2-1) \le 0, \end{aligned}$$
since $1 \le e \le \sqrt{2}$.

Fourth Solution. We prove only the upper bound here. It is clear that $a, b, c \le 2$. If $a = 0$, then the given equality reduces to $b^2 + c^2 = 4$. Then $ab + bc + ca - abc = bc \le \frac{(b-c)^2}{2} + bc = \frac{b^2+c^2}{2} = 2$. Similarly, the inequality is true if $b$ or $c$ equals $0$.

Suppose that $a, b, c > 0$. Solving for $a$ yields
$$\begin{aligned} a &= \frac{-bc + \sqrt{b^2c^2 - 4(b^2 + c^2 - 4)}}{2} \\ &= \frac{-bc + \sqrt{(4 - b^2)(4 - c^2)}}{2}. \end{aligned}$$
We make the trigonometric substitution $b = 2 \sin u$ and $c = 2 \sin v$, where $0^\circ < u, v < 90^\circ$. Then
$$a = 2(-\sin u \sin v + \cos u \cos v) = 2 \cos (u + v).$$
Set $u = B/2$, $v = C/2$, and $A = 180^\circ - B - C$. Then $a = 2 \cos(u+v) = 2 \sin(A/2)$, $b = 2 \sin(B/2)$, and $c = 2 \sin(C/2)$, where $A, B, C$ are the angles of a triangle. We have
$$\begin{aligned} ab &= 4 \sin \frac{A}{2} \sin \frac{B}{2} = 2 \sqrt{\sin A \tan \frac{A}{2} \sin B \tan \frac{B}{2}} \\ &= 2 \sqrt{\sin A \tan \frac{B}{2} \sin B \tan \frac{A}{2}} \end{aligned}$$
By the AM-GM Inequality, this is at most
$$\begin{aligned} \sin A \tan \frac{B}{2} + \sin B \tan \frac{A}{2} \\ &= \sin A \cot \frac{A+C}{2} + \sin B \cot \frac{B+C}{2}. \end{aligned}$$
Likewise,
$$bc \le \sin B \cot \frac{B+A}{2} + \sin C \cot \frac{C+A}{2},$$
$$ca \le \sin C \cot \frac{C+B}{2} + \sin A \cot \frac{A+B}{2}.$$
Therefore, applying the Sum-to-product, Product-to-sum, and Double-angle formulas, we have
$$\begin{align*} ab + bc + ca &\le (\sin A + \sin B) \cot \frac{A+B}{2} + (\sin B + \sin C) \cot \frac{B+C}{2} \\ &\quad + (\sin C + \sin A) \cot \frac{C+A}{2} \\ &= 2 \cos \frac{A-B}{2} \cos \frac{A+B}{2} + 2 \cos \frac{B-C}{2} \cos \frac{B+C}{2} \\ &\quad + 2 \cos \frac{C-A}{2} \cos \frac{C+A}{2} \\ &= 2(\cos A + \cos B + \cos C) \\ &= 6 - 4 \left( \sin^2 \frac{A}{2} + \sin^2 \frac{B}{2} + \sin^2 \frac{C}{2} \right) \\ &= 6 - (a^2 + b^2 + c^2). \end{align*}$$
Using the given equality, this last quantity equals $2 + abc$. It follows that
$$ab + bc + ca \le 2 + abc,$$
as desired.