, 2010

Let us work from first principles. Whenever (at least) one variable is zero, say $c$, it follows $a^2 + b^2 = k + 3$, hence $a + b + c \le \sqrt{2(k+3)} \le 3$ for $k \le 3/2$, with equality holding for $a = b = 3/2$. We thus have a starting tentative limiting bound of $K = 3/2$.

Let also notice that $a = b = c = 1$ checks for any value of $k$, while providing a maximal admissible value for $a + b + c = 3$.

Assume $\sigma = a + b + c > 3$. Notice that then (for some $k > 0$) $k(1 - abc) = a^2 + b^2 + c^2 - 3 \ge \frac{1}{3}(a+b+c)^2 - 3 = \frac{1}{3}\sigma^2 - 3 > 0$, by Cauchy-Schwarz, and so $abc < 1$. Now is the time for the key step. Since $a + b + c > 3$, assuming some ordering of our variables, say $0 \le a \le b \le c$, it will follow $c > 1$, and so we can compute

$$\begin{align*} k &= \frac{a^2 + b^2 + c^2 - 3}{1 - abc} \ &= \frac{(a+b)^2 + c^2 - 3 - 2ab}{1 - abc} \ &= \frac{(\sigma-c)^2 + c^2 - 3 - 2ab}{1 - abc} \ &= \frac{c(2c^2 - 2\sigma c + \sigma^2 - 3) - 2 + 2(1 - abc)}{c(1 - abc)} \ &= \frac{2}{c} + \frac{2c^3 - 2\sigma c^2 + (\sigma^2 - 3)c - 2}{c(1 - abc)} \end{align*}$$
Denote $f(c) = 2c^3 - 2\sigma c^2 + (\sigma^2 - 3)c - 2$. As $(x-y)(f(x)-f(y)) = (x-y)^2 \left( \frac{3}{2} \left( x+y-\frac{2}{3}\sigma \right)^2 + \frac{1}{2}(x-y)^2 + \frac{1}{3}(\sigma^2-9) \right) \ge 0$, $f$ is increasing; and as $f(1) = (\sigma+1)(\sigma-3) > 0$, it follows $f(c) > 0$. Therefore the minimal value for $k$ is reached when $ab = 0$, for $a = 0$, whence $k \ge b^2 + c^2 - 3 \ge \frac{1}{2}(b+c)^2 - 3 = \frac{1}{2}\sigma^2 - 3 > \frac{3}{2}$.

Define
$$L(a, b, c) = a + b + c - \lambda(a^2 + b^2 + c^2 + kabc - k - 3).$$

The analysis of the values on the border of the domain of $L$ has been done above, leading to $K \le 3/2$ (we could now simplify our work to just $k = 3/2$, but it is enlightening to see it in full generality). The system of partial derivatives is
$$\left\{ \begin{array}{l} \displaystyle \frac{\partial L}{\partial a} = 1 - \lambda(2a + kbc) \\[1em] \displaystyle \frac{\partial L}{\partial b} = 1 - \lambda(2b + kca) \\[1em] \displaystyle \frac{\partial L}{\partial c} = 1 - \lambda(2c + kab) \end{array} \right.$$

Equaling the partial derivatives to zero forbids $\lambda = 0$. Then, from pairwise equalities, we get
$$\lambda(a - b)(2 - kc) = \lambda(b - c)(2 - ka) = \lambda(c - a)(2 - kb) = 0.$$
One possibility is $a = b = c = x$, thus (from the constraint) $3x^2 + kx^3 = k + 3$, or
$(x-1)(kx^2+(k+3)x+(k+3)) = 0$, with only non-negative real solution $x = 1$, since
the coefficients of the quadratic factor are non-negative. Then $a + b + c = 3x = 3$,
the admissible maximum.
The other possibility is for two variables to be equal, say $a = b$, but not equal
to the third, hence needing $a = b = 2/k$, thus (from the constraint) $8/k^2 + c^2 +
4c/k = k + 3$, or$k^2c^2 + 4kc - (k^3 + 3k^2 - 8) = 0$, with non-negative real solution
$c = \frac{(k+2)\sqrt{k-1}-2}{k}$ for $k^3+3k^2-8 \ge 0$, i.e. $k \ge \kappa \approx 1.3553$. Then $a+b+c =$
$\frac{(k+2)\sqrt{k-1}+2}{k} < 3$ for $k < 2$, since it is equivalent to $(k-2)^3 < 0$. As $k$ needs
be at most $K \le 3/2$, these points are critical, but not global maxima (in fact they turn
to be global minima).

Alternative Solution:
The fact the value 3 for $a+b+c$ is reached for $k = 3/2$ both at $a = b = c = 1$, and at $a = b = 3/2$ and $c = 0$ et al., suggests this is a Schur-type inequality. Indeed, assume $0 \le k \le 3/2$ and $\sum a > 3$.
Then
$$\begin{aligned} k+3 &= \sum a^2 + kabc \\ &\ge \sum a^2 + \frac{3kabc}{\sum a} \\ &= \left(1-\frac{k}{3}\right) \sum a^2 + \frac{k}{3} \left(\sum a^2 + \frac{9abc}{\sum a}\right) \\ &\ge \left(1-\frac{k}{3}\right) \sum a^2 + \frac{2k}{3} \sum ab \\ &= \left(1-\frac{2k}{3}\right) \sum a^2 + \frac{k}{3} \left(\sum a^2 + 2\sum ab\right) \\ &= \left(1-\frac{2k}{3}\right) \sum a^2 + \frac{k}{3} \left(\sum a\right)^2 \\ &\ge \left(\frac{1}{3}\left(1-\frac{2k}{3}\right) + \frac{k}{3}\right) \left(\sum a\right)^2 \\ &= \frac{k+3}{9} \left(\sum a\right)^2 \\ &> k+3 \end{aligned}$$
since inequality $\sum a^2 + \frac{9abc}{\sum a} \ge 2\sum ab$ is in turn equivalent to $(\sum a^2)(\sum a) + 9abc \ge 2(\sum ab)(\sum a)$, then $\sum a^3 + 3abc \ge \sum a^2b + \sum ab^2$, at last $\sum a(a-b)(a-c) \ge 0$, a basic form of Schur; while $\sum a^2 \ge \frac{1}{3}(\sum a)^2$ by Cauchy-Schwarz.

Putting it all together, the largest admissible value for $K$ turns to be $3/2$, with the maximum value $a+b+c = 3$ being reached only at points $(3/2, 3/2, 0)$, $(3/2, 0, 3/2)$, $(0, 3/2, 3/2)$ and $(1, 1, 1)$. For $0 \le k < 3/2$ the unique maximum is reached at $(1, 1, 1)$. Notice the importance of examining the values on the border; without that, the other critical interior points found but $(1, 1, 1)$ (which works for any $k$) achieve a larger value than 3 for $a + b + c$ only starting with $k > 2$, so would induce the erroneous bound $K = 2$.