Given that $a_3$ and $a_7$ are the two roots of the equation $3x^2-11x+9=0$,
we have $a_3 + a_7 = \frac{11}{3}$ and $a_3 \cdot a_7 = 3$.
Therefore, both $a_3$ and $a_7$ are positive.
By the properties of geometric sequences, we have $a_5^2 = a_3 \cdot a_7 = 3$,
and since the terms of the geometric sequence have the same sign for every other term, we get $a_5 = \sqrt{3}$.
Hence, the answer is $\boxed{\sqrt{3}}$.
This problem tests the properties of geometric sequences and the general formula, involving Vieta's formulas, and is considered a basic question.