We start with some lemmas.
Lemma. For positive integers m>3 and k>2 we have mm+k>(m+k+1)m.
Proof. We must show that mk>(mm+k+1)m=(1+mk+1)m. But we have (1+mk+1)m<ek+1 and so
(1+mk+1)m<ek+1<(2.8)k+1≤(2.8)34k<4k≤mk.
Lemma. For all positive integers m>4 and n>5, we have em>(m+1)3 and 2n>(n+1)2.
Proof. The proof of both parts are consequences of some elementary inductions.
□
Now for the main problem suppose that k is the largest possible integer and x1<x2<⋯<xk are k numbers satisfying the condition. First note that if [xi]=[xj] for some i,j we have
xi[xi]=xj[xj]⟹xi=xj⟹i=j
This means that [xi]'s are distinct and so [xk]−[xi]≥k−i. We claim that k≤4. Assume to the contrary that k>4.
* If [x2]>3 since [xk]−[x2]>2 using the first lemma we get ([xk]+1)[x2]<[x2][xk]. But on the other hand we have
[x2][xk]≤x2[xk]=xk[x2]≤([xk]+1)[x2]
This leads to a contradiction and implies that [x1]<[x2]≤3.
* If [x2]=3, then
3[xk]≤x2[xk]=xk[x2]=xk3≤([xk]+1)3
By the second lemma, we deduce that [xk]≤4 and therefore k≤4 which contradicts with our assumption.
* If [x2]=2, then similar to the previous part we have
2[xk]≤x2[xk]=xk[x2]=xk2≤([xk]+1)2
and so by the second lemma [xk]≤5 therefore k≤5. Now if k=5, then [xi]=i for all 1≤i≤5 which contradicts, because
6>x5=x15=x12×x13=x2×x3≥2×3=6
Therefore k≤4. For k=4, the numbers x1=331, x2=332, x3=3, x4=334 satisfy the condition, because [xi]=i and so xi[xj]=33ij=xj[xi]. So the proof is complete and the desired maximum is 4. ■