a) Since f is continuous and bijective, we deduce that f(0)=0 and f is increasing, with limx→∞f(x)=∞. Then f−1:[0,∞)→[0,∞) is also increasing, with f−1(0)=0 and limx→∞f−1(x)=∞. The limit from the hypothesis ensures the existence of a number u>0 such that xf−1(f(x)/x)>21, for any x>u. Therefore, xf(x)>f(2x), for any x>u. Since limx→∞f(2x)=∞, we obtain limx→∞xf(x)=∞. Let a>0 be an arbitrary fixed number.
The case a∈(0,1). There is t>0 such that f−1(x)>a1, for any x>t. Hence
f−1(x)>f−1(ax)=f−1(af(f−1(x)))>f−1(f−1(x)f(f−1(x))), for any x>t.
We obtain
f−1(x)f−1(f(f−1(x))/f−1(x))<f−1(x)f−1(ax)<1, for any x>t.
From the assumption and the condition limx→∞f−1(x)=∞, we get
x→∞limf−1(x)f−1(f(f−1(x))/f−1(x))y=f−1(x)=y→∞limyf−1(f(y)/y)=1.
The case a=1 is clear.
The sandwich theorem ensures limx→∞f−1(x)f−1(ax)=1.
The case a∈(1,∞). Then b=1/a∈(0,1). From the previous case, we have
x→∞limf−1(x)f−1(ax)=x→∞lim(f−1(ax)f−1(x))−1=x→∞lim(f−1(ax)f−1(b(ax)))−1=1−1=1.
In conclusion, limx→∞f−1(x)f−1(ax)=1, for any a>0.
b) Example. The bijective function f:[0,∞)→[0,∞), f(x)=ex−1, with the inverse f−1:[0,∞)→[0,∞), f−1(x)=ln(x+1), satisfies the conditions from the statement.