Without loss of generality, let $a_{1} \leq a_{2} \leq \cdots \leq a_{2023}$. Then, note that $a_{k+1}-a_{k}=\sum_{i=1}^{2023}\left|a_{k+1}-a_{i}\right|-\left|a_{k}-a_{i}\right|=k(a_{k+1}-a_{k})-(2023-k)(a_{k+1}-a_{k})=(2k-2023)(a_{k+1}-a_{k})$. Thus, $a_{k+1}=a_{k}$ unless $k=1012$, so $a_{1}=a_{2}=\cdots=a_{1012}$ and $a_{1013}=\cdots=a_{2023}$, and we can check that that they must be in a ratio of $1011:1012$. Thus, $a_{1}, \ldots, a_{2023}$ must consist of 1012 copies of $1011c$, and 1011 copies of $1012c$ for some $c$, so for everything to be a positive integer, we need $c \geq 1$. This gives us the final answer of $1011 \cdot 1012+1012 \cdot 1011=2046264$.