Problem 509

6. $\frac{8 \sqrt{3}}{27} R^{3}$.

Let the tetrahedron be $P-ABC$, and the circumradius of $\triangle ABC$ be $r$. Then
$$S_{\triangle ABC}=2 r^{2} \sin A \cdot \sin B \cdot \sin C \leqslant \frac{3 \sqrt{3}}{4} r^{2} .$$

The equality holds if and only if $\angle A=\angle B=\angle C=60^{\circ}$.
If the distance from the center of the sphere $O$ to the plane $ABC$ is $h$, then
$$\begin{array}{l} V \leqslant \frac{1}{3} S_{\triangle ABC}(R+h) \leqslant \frac{\sqrt{3}}{4} r^{2}(R+h) \\ =\frac{\sqrt{3}}{4}\left(R^{2}-h^{2}\right)(R+h) \\ =\frac{\sqrt{3}}{8}(R+h)(R+h)(2 R-2 h) \\ \leqslant \frac{\sqrt{3}}{8}\left(\frac{R+h+R+h+2 R-2 h}{3}\right)^{3} \\ =\frac{8 \sqrt{3}}{27} R^{3} . \end{array}$$

The equality holds if and only if the tetrahedron $P-ABC$ is a regular tetrahedron.