First, note that $\lfloor \frac{n}{k} \rfloor$ represents the number of multiples of $k$ among $1, 2, \dots, n$. Hence,
$$\sum_{k=1}^{n} \lfloor \frac{n}{k} \rfloor \tau(k)^2 = \sum_{k=1}^{n} \sum_{1 \le m \le n,\ k|m} \tau(k)^2 = \sum_{m=1}^{n} \sum_{k|m} \tau(k)^2.$$
To prove the problem statement, it suffices to justify, for $m = 1, \dots, n$,
$$5^{\omega(m)} \le \sum_{k|m} \tau(k)^2 \le 5^{\Omega(m)}.$$
When $m=1$, it is obvious. When $m > 1$, let $m = p_1^{\alpha_1} p_2^{\alpha_2} \cdots p_r^{\alpha_r}$ be the prime factorization ($p_1, p_2, \dots, p_r$ are distinct prime numbers and $\alpha_1, \alpha_2, \dots, \alpha_r$ are positive integers). For $k = p_1^{\beta_1} p_2^{\beta_2} \cdots p_r^{\beta_r}$, $\tau(k) = (\beta_1 + 1)(\beta_2 + 1) \cdots (\beta_r + 1)$. Thus,
$$\begin{aligned} \sum_{k|m} \tau(k)^2 &= \sum_{\substack{0 \le \beta_1 \le \alpha_1 \\ 0 \le \beta_r\cdots \le \alpha_r}} (\beta_1 + 1)^2 (\beta_2 + 1)^2 \cdots (\beta_r + 1)^2 \\ &= \prod_{i=1}^{r} (1^2 + 2^2 + \cdots + (\alpha_i + 1)^2). \end{aligned}$$
Now it suffices to show for each $1 \le i \le r$,
$$5 \le 1^2 + 2^2 + \cdots + (\alpha_i + 1)^2 \le 5^{\alpha_i}.$$
For $j \in \mathbb{N}_+$, define $T(j) = 1^2+2^2+\cdots+(j+1)^2 = \frac{1}{6}(j+1)(j+2)(2j+3)$.
Then
$$\frac{T(j+1)}{T(j)} = \frac{j+2}{j+1} \cdot \frac{j+3}{j+2} \cdot \frac{2j+5}{2j+3} \in [1, \frac{2}{1} \cdot \frac{3}{2} \cdot \frac{5}{3}] = [1, 5].$$
Since $T(1) = 5$, it is straightforward to check $T(j) \in [5, 5^j]$ ($\forall j \in \mathbb{N}_+$) by induction. This finishes the proof. $\Box$