We show that the maximum value is attained when x1=x2=⋯=xn=n1, and Fmax=2n−1(n1+a+b+nab).
We induct on n to show a more general statement: for non-negative real numbers x1,x2,…,xn satisfying x1+x2+⋯+xn=s (where s is a fixed non-negative real number), the maximum value of F=∑1≤i<j≤nmin{f(xi),f(xj)} is attained when x1=x2=⋯=xn=ns.
Since F is symmetric, we may assume x1≤x2≤⋯≤xn. Note that f(x) is strictly increasing on non-negative real numbers, we have
F=(n−1)f(x1)+(n−2)f(x2)+⋯+f(xn−1).
When n=2, F=f(x1)≤f(2s), equality holds when x1=x2. Assume that the statement holds for n, consider the case of n+1. Applying inductive hypothesis on x2+x3+⋯+xn+1=s−x1, we have
F≤nf(x1)+21n(n−1)f(ns−x1)=g(x1),
where g(x) is a quadratic function of x, the leading coefficient is 1+2n2n−1, and the coefficient of x is a+b−2nn−1(a+b+2ns), therefore, the axis of symmetry is
2+n2n−12nn−1(a+b+2ns)−a−b≤2(n+1)s.
(The above inequality is equivalent to [(n−1)s−2n(n+1)(a+b)](n+1)≤2s(2n2+n−1); obviously, left-hand side <(n2−1)s< right-hand side.) Therefore, g(n+1s) is the maximum of g(x) on [0,n+1s]. Thus, F attains its maximum when x2=x3=⋯=xn+1=ns−x1=n+1s=x1, completing the solution. □