Harvard-MIT Mathematics Tournament

Solution:

Answer: $\left(\frac{\sqrt{30}}{3}, \frac{\sqrt{30}}{4}, \frac{2 \sqrt{30}}{5}\right),\left(\frac{\sqrt{30}}{3}, \frac{\sqrt{30}}{2}, \frac{\sqrt{30}}{5}\right)$

Write $p=a b c$, $q=\lfloor a\rfloor\lfloor b\rfloor\lfloor c\rfloor$. Note that $q$ is an integer.

Multiplying the three equations gives:
$$p=\sqrt{\frac{60}{q}}$$
Substitution into the first equation,
$$p=3 \frac{a}{\lfloor a\rfloor}<3 \frac{\lfloor a\rfloor+1}{\lfloor a\rfloor} \leq 6$$
Looking at the last equation:
$$p=5 \frac{c}{\lfloor c\rfloor} \geq 5 \frac{\lfloor c\rfloor}{\lfloor c\rfloor} \geq 5$$
Here we've used $\lfloor x\rfloor \leq x<\lfloor x\rfloor+1$, and also the apparent fact that $\lfloor a\rfloor \geq 1$. Now:
$$\begin{aligned} & 5 \leq \sqrt{\frac{60}{q}} \leq 6 \\ & \frac{12}{5} \geq q \geq \frac{5}{3} \end{aligned}$$
Since $q$ is an integer, we must have $q=2$. Since $q$ is a product of 3 positive integers, we must have those be 1, 1, and 2 in some order, so there are three cases:

Case 1: $\lfloor a\rfloor=2$. By the equations, we'd need $a=\frac{2}{3} \sqrt{30}=\sqrt{120 / 9}>3$, a contradiction, so there are no solutions in this case.

Case 2: $\lfloor b\rfloor=2$. We have the solution
$$\left(\frac{\sqrt{30}}{3}, \frac{\sqrt{30}}{2}, \frac{\sqrt{30}}{5}\right)$$

Case 3: $\lfloor c\rfloor=2$. We have the solution
$$\left(\frac{\sqrt{30}}{3}, \frac{\sqrt{30}}{4}, \frac{2 \sqrt{30}}{5}\right)$$