Solution:
We have from the AM-GM inequality, that
xy≤(2x+y)2=1
Thus we obtain 0<xy≤1. We write
x3y3(x3+y3)=(xy)3(x+y)(x2−xy+y2)=2(xy)3((x+y)2−3xy)=2(xy)3(4−3xy)
Thus we need to prove that
(xy)3(4−3xy)≤1
Putting z=xy, this inequality reduces to
z3(4−3z)≤1
for 0<z≤1. We can prove this in different ways. We can put the inequality in the form
3z4−4z3+1≥0
Here the expression in the LHS factors to (z−1)2(3z2+2z+1) and (3z2+2z+1) is positive since its discriminant D=−8<0. Or applying the AM-GM inequality to the positive reals 4−3z,z,z,z, we obtain
z3(4−3z)≤(44−3z+3z)4≤1