Let z=x+y. Then 0≤z≤1, whence z(1−z)≥0. Now
5x(1−x)+5y(1−y)−8xy=5(x+y)−5(x2+y2)−8xy=5z−5(x2+2xy+y2)+2xy=5z−5z2+2xy=5z(1−z)+2xy≥0,
Solution 2:
From x+y≤1 we obtain y≤1−x and x≤1−y. As x≥0 and y≥0, these imply xy≤x(1−x) and xy≤y(1−y). Multiplying both by 5 and adding them we obtain
10xy≤5x(1−x)+5y(1−y).
As xy≥0, we have 8xy≤10xy which implies the result.
For equality we need 8xy=10xy and 10xy=5x(1−x)+5y(1−y). Hence xy=0, and xy=x(1−x) as well as xy=y(1−y). Therefore, we either have x=0,y=1 or y=0,x=1 or x=y=0.