IRL ABooklet

To begin with,
$$1 = (a + b + c)^2 = a^2 + b^2 + c^2 + 2(ab + bc + ca) = 1 + 2(ab + bc + ca)$$
so that $ab+bc+ca = 0$. Now we can prove the upper bound of zero. Suppose (for a contradiction) that a real triple $a, b, c$ satisfies the given constraints and $abc > 0$. Then either

(i) all three of $a, b, c$ are positive, or
(ii) some two are negative and the other one is positive.
If (i) holds, then each of the products $ab, bc, ca$ is positive, in which case $ab + bc + ca > 0$, a contradiction. Say (ii) happens with $a < 0, b < 0, c > 0$. Then
$$\begin{aligned} 0 &= ab + bc + ca \\ &= ab + c(a + b) = ab + (a + b)(1 - (a + b)) \\ &= ab + a + b - (a + b)^2 = a + b - a^2 - ab - b^2 \end{aligned}$$
a sum of five negative numbers equal to zero, also a contradiction. Thus (ii) can't occur either. The conclusion is that $abc \le 0$ for all triples satisfying the given constraints.

Next, we will prove that $abc \ge -\frac{4}{27}$. In any case, $abc \le 0$ and $a + b + c > 0$, hence at most one of $a, b, c$ is negative. Suppose $a > 0, b > 0, c < 0$. Since, $0 = ab+bc+ca$, therefore $-ac = b(a+c) = b(1-b)$. Thus $ab(-c) = b^2(1-b)$. But $a^2 + b^2 + c^2 = 1$, so that $b^2 < 1$, in which case $0 < b < 1$. Hence, by AM-GM,
$$\left(\frac{b^2(1-b)}{4}\right)^{1/3} = \left(\frac{b}{2} \cdot \frac{b}{2} \cdot (1-b)\right)^{1/3} \le \frac{1}{3}\left(\frac{b}{2} + \frac{b}{2} + 1 - b\right) = \frac{1}{3}$$
whence $ab(-c) = b^2(1-b) \le \frac{4}{27}$ as desired.

First observe that $2ab = (a+b)^2 - (a^2+b^2) = (1-c)^2 - (1-c^2) = 2c^2 - 2c$. So $abc = c^3 - c^2 = c^2(c-1)$.

Next observe that $a$ and $b$ satisfy the equation $x^2 - (a+b)x + ab = 0$, that is $x^2 - (1-c)x + c^2 - c = 0$. Since $a$ and $b$ (and hence $c$) are real, the quadratic has a non-negative discriminant so that $(1-c)^2 - 4(c^2-c) \ge 0$. That reduces to $1+2c-3c^2 \ge 0$ and hence $(c-1)(c+\frac{1}{3}) \le 0$. So $c$ must satisfy $-\frac{1}{3} \le c \le 1$. It follows immediately that $abc = c^2(c-1) \le 0$, which proves the right side of the inequality.

Note that if $0 < c < 1$, then AM-GM implies:
$$2(c^2 - c^3) = (c)(c)(2 - 2c) \le \left( \frac{c+c+2-2c}{3} \right)^3 = \frac{8}{27}$$
So $c^3 - c^2 \ge -\frac{4}{27}$ which proves the left side of the inequality for these values of $c$. In the remaining case, $-\frac{1}{3} \le c \le 0$, we note that $c^3 - c^2$ is an increasing function of $c$ on that range (an increasing function minus a decreasing function) and so $c^3 - c^2$ is at most the value of that expression when $c = -\frac{1}{3}$, completing the proof.

Observing that
$$2ab = (a+b)^2 - (a^2+b^2) = (1-c)^2 - (1-c^2) = 2c^2 - 2c,$$
we obtain $abc = c^2(c-1) \le 0$ because we have $c \le 1$ from $a^2 + b^2 + c^2 = 1$. Now we see that
$$\begin{align*} abc \ge -\frac{4}{27} &\iff c^3 - c^2 + \frac{4}{27} \ge 0 \\ &\iff 27c^3 - 27c^2 + 4 \ge 0 \\ &\iff x^3 - 3x^2 + 4 \ge 0 \end{align*}$$
where we set $x = 3c$. We easily discover the root $x = -1$ of $x^3 - 3x^2 + 4$ and then the factorisation $x^3 - 3x^2 + 4 = (x+1)(x-2)^2$.
To show $x = 3c \ge -1$, we assume $c < -1/3$. This implies $c^2 > 1/9$ and $a+b = 1-c > 4/3$, hence we cannot have $a \le 0$ and $b \le 0$, i.e. at least one of $a,b$ is positive. Because $abc \le 0$ and $c < 0$ we have $ab \ge 0$. This implies now that $a \ge 0$ and $b \ge 0$ and we can use AM-QM to get $a^2 + b^2 \ge (a+b)^2/2 > 8/9$ which leads to the contradiction $a^2 + b^2 + c^2 > 1$. It follows that $x \ge -1$ and so $x^3 - 3x^2 + 4 = (x+1)(x-2)^2 \ge 0$, as desired.

The plane $a+b+c=1$ intersects the unit sphere in a circle containing the points $(1,0,0)$, $(0,1,0)$, $(0,0,1)$, and these three points divide the circle into arcs. At the three points, the inequalities are trivial. Except at these three points, we cannot have all three coordinates on these arcs non-negative, because $ab+bc+ca=0$ as seen in Solution 1. In fact, exactly one coordinate is negative, since if $a,b < 0$, say, then the first equation implies $c > 1$, whence $c^2 > 1$, contradicting the second equation. This shows, in particular, $abc \le 0$.

Let us assume, without loss of generality, that $c < 0$, so $a, b > 0$. We claim the point on the circle where $c$ is minimal equates $a$ and $b$, that is $(a, b, c) = (2/3, 2/3, -1/3)$. For the proof of minimality suppose $c < -1/3$, then $a+b > 4/3$ and $a^2+b^2 \ge \frac{1}{2}(a+b)^2 > 8/9$, while $c^2 > 1/9$, contradicting the second equation. If $c = -1/3$ we get $a+b = 4/3$ and $a^2+b^2 = 8/9$. Then we are in the equality case of the AM-QM inequality $\frac{8}{9} = a^2+b^2 \ge \frac{1}{2}(a+b)^2 = \frac{8}{9}$, hence $a = b = 2/3$.
Next we show for fixed $c$ satisfying $-1/3 \le c < 0$ that $ab$ is maximised (and so $abc$ is minimised) over the set $\{(a,b) : 0 < a, b; a^2 + b^2 \le 1 - c^2\}$ when $2a^2 = 2b^2 = 1 - c^2$. Call the corresponding point $P_c$.
To maximise $xy$ inside the closed disc centred at the origin, intersected with the first quadrant, $\{(x, y) : 0 < x, y; x^2 + y^2 \le r^2\}$, we proceed as follows. It is easy to see that on any line of slope $-1$, i.e. for fixed $x+y$, the maximum of $xy$ occurs when $x = y$. We then head out along the diagonal to the maximum at $x = y = \frac{r}{\sqrt{2}}$ and see that the maximum of $xy$ is equal to $r^2/2$. In our situation, $r^2 = 1 - c^2$ and so the minimum of $abc$ (for fixed $-1/3 \le c < 0$) occurs at the point $P_c$, and is equal to $\frac{c(1-c^2)}{2}$.

To find the minimum of $abc$ for these points $P_c$ (now allowing $c$ to vary), we substitute $c = x - \frac{1}{3}$ to obtain
$$abc = \frac{c(1 - c^2)}{2} = -\frac{4}{27} - \frac{x}{6}(3x^2 - 3x - 2).$$
For $0 \le x < 1/3$ we have $3x < 1$ and $3x^2 \le x \le 3x < 3x + 2$ and therefore
$$-\frac{x}{6}(3x^2 - 3x - 2) \ge 0 \quad \text{with equality iff } x = 0.$$
Thus we have shown that the minimum of $abc$ is obtained for $x = 0$, i.e. $c = -1/3$, and this corresponds to the point $P_{-1/3} = (2/3, 2/3, -1/3)$, where $abc = -\frac{4}{27}$. This is the minimum over all $(a, b, c)$ that satisfy $a^2+b^2+c^2 \le 1$, $a, b > 0$, and $-1/3 \le c < 0$. Earlier we showed that $(a, b, c)$ satisfies these conditions if it is a point on the arc mentioned above on which $c < 0$. By symmetry it follows that $abc \ge -\frac{4}{27}$ for all points considered in the problem.