First Solution. (Based on work by Matthew Tang and Anders Kaseorg)
By multiplying a, b, and c by a suitable factor, we reduce the problem to the case when a+b+c=3. The desired inequality reads
2a2+(3−a)2(a+3)2+2b2+(3−b)2(b+3)2+2c2+(3−c)2(c+3)2≤8.
Set
f(x)=2x2+(3−x)2(x+3)2
It suffices to prove that f(a)+f(b)+f(c)≤8. Note that
f(x)=3(x2−2x+3)x2+6x+9=31⋅x2−2x+3x2+6x+9=31(1+x2−2x+38x+6)=31(1+(x−1)2+28x+6)≤31(1+28x+6)=31(4x+4).
Hence,
f(a)+f(b)+f(c)≤31(4a+4+4b+4+4c+4)=8,
as desired, with equality if and only if a=b=c.
Second Solution. (By Liang Qin) Setting x=a+b, y=b+c, z=c+a gives 2a+b+c=x+z, hence 2a=x+z−y and their analogous forms. The desired inequality becomes
(x+z−y)2+2y22(x+z)2+(z+y−x)2+2x22(z+y)2+(y+x−z)2+2z22(y+x)2≤8.
Because 2(s2+t2)≥(s+t)2 for all real numbers s and t, we have 2(x+z−y)2+2y2≥(x+z−y+y)2=(x+z)2. Hence
(x+z−y)2+2y22(x+z)2=2(x+z−y)2+4y24(x+z)2≤(x+z)2+2y24(x+z)2=1+2⋅(x+z)2y24≤1+2⋅2(x2+z2)y24=x2+y2+z24(x2+z2).
It is not difficult to see that the desired result follows from summing up the above inequality and its analogous forms.
Third Solution. (By Richard Stong) Note that
(2x+y)2+2(x−y)2=4x2+4xy+y2+2x2−4xy+2y2=3(2x2+y2).
Setting x=a and y=b+c yields
(2a+b+c)2+2(a−b−c)2=3(2a2+(b+c)2).
Thus, we have
2a2+(b+c)2(2a+b+c)2=2a2+(b+c)23(2a2+(b+c)2)−2(a−b−c)2=3−2a2+(b+c)22(a−b−c)2.
and its analogous forms. Thus, the desired inequality is equivalent to
2a2+(b+c)2(a−b−c)2+2b2+(c+a)2(b−a−c)2+2c2+(a+b)2(c−a−b)2≥21.
Because (b+c)2≤2(b2+c2), we have 2a2+(b+c)2≤2(a2+b2+c2) and its analogous forms. It suffices to show that
2(a2+b2+c2)(a−b−c)2+2(a2+b2+c2)(b−a−c)2+2(a2+b2+c2)(c−a−b)2≥21,
or,
(a−b−c)2+(b−a−c)2+(c−a−b)2≥a2+b2+c2.
Multiplying this out, the left-hand side of the last inequality becomes 3(a2+b2+c2)−2(ab+bc+ca). Therefore the last inequality is equivalent to 2[a2+b2+c2−(ab+bc+ca)]≥0, which is evident because
2[a2+b2+c2−(ab+bc+ca)]=(a−b)2+(b−c)2+(c−a)2.
Equalities hold if and only if (b+c)2=2(b2+c2) and (c+a)2=2(c2+a2), that is, a=b=c.
Fourth Solution. We first convert the inequality into
2a2+(b+c)22a(a+2b+2c)+2b2+(c+a)22b(b+2c+2a)+2c2+(a+b)22c(c+2a+2b)≤5.
Splitting the 5 among the three terms yields the equivalent form
cyc∑3[2a2+(b+c)2]4a2−12a(b+c)+5(b+c)2≥0,(1)
where ∑cyc is the cyclic sum of variables (a,b,c). The numerator of the term shown factors as (2a−x)(2a−5x), where x=b+c. We will show
3(2a2+x2)(2a−x)(2a−5x)≥−3(a+x)4(2a−x).(2)
Indeed, (2) is equivalent to
(2a−x)[(2a−5x)(a+x)+4(2a2+x2)]≥0,
which reduces to
(2a−x)(10a2−3ax−x2)=(2a−x)2(5a+x)≥0,
which is evident. We proved that
3[2a2+(b+c)2]4a2−12a(b+c)+5(b+c)2≥−3(a+b+c)4(2a−b−c),
hence (1) follows. Equality holds if and only if 2a=b+c, 2b=c+a, 2c=a+b, i.e., when a=b=c.
Fifth Solution. Given a function f of n variables, we define the symmetric sum
sym∑f(x1,…,xn)=σ∑f(xσ(1),…,xσ(n))
where σ runs over all permutations of 1,…,n (for a total of n! terms). For example, if n=3, and we write x,y,z for x1,x2,x3,
sym∑x3sym∑x2ysym∑xyz=2x3+2y3+2z3=x2y+y2z+z2x+x2z+y2x+z2y=6xyz.
We combine the terms in the desired inequality over a common denominator and use symmetric sum notation to simplify the algebra. The numerator of the difference between the two sides is
2sym∑(4a6+4a5b+a4b2+5a4bc+5a3b3−26a3b2c+7a2b2c2),(3)
and it suffices to show the the expression in (3) is always greater or equal to 0. By the Weighted AM-GM Inequality, we have 4a6+b6+c6≥6a4bc, 3a5b+3a5c+b5a+c5a≥8a4bc, and their analogous forms. Adding those inequalities yields
sym∑6a6≥sym∑6a4bcandsym∑8a5b≥sym∑8a4bc.
Consequently, we obtain
sym∑4a6+4a5b+5a4bc≥sym∑13a4bc.(4)
Again by the AM-GM Inequality, we have a4b2+b4c2+c4a2≥4a2b2c2, a3b3+b3c3+c3a3≥3a2b2c2, and their analogous forms. Thus,
sym∑a4b2+5a3b3≥sym∑6a2b2c2,
or
sym∑a4b2+5a3b3+7a2b2c2≥sym∑13a2b2c2.(5)
Recalling Schur's Inequality, we have
a3+b3+c3+3abc−(a2b+b2c+c2a+ab2+bc2+ca2)=a(a−b)(a−c)+b(b−a)(b−c)+c(c−a)(c−b)≥0,
or
sym∑a3−2a2b+abc≥0.
Thus
sym∑13a4bc−26a3b2c+13a2b2c2≥13abcsym∑a3−2a2b+abc≥0.(6)
Adding (4), (5), and (6) yields (3).