Solution:

It is easy to see that the points $O$ and $C$ must be in different semi-planes with respect to the line $AB$.

Let $OPB$ be an equilateral triangle ($P$ and $C$ on the same side of $OB$). Since $\angle PBC = 60^{\circ} - \angle ABP$ and $\angle OBA = 60^{\circ} - \angle ABP$, then $\angle PBC = \angle OBA$. Hence the triangles $AOB$ and $CPB$ are equal and $PC = OA$. From the triangle $OPC$ we have
$$OC \leq OP + PC = OB + OA = 8$$
Hence, the maximum value of the distance $OC$ is $8$ (when the point $P$ lies on $OC$).

Figure 8

Figure 9

Let $\phi$ be a $60^{\circ}$ rotation with center at $B$. Then $\phi(A) = C$, $\phi(O) = P$ and $PC = OA$, $OP = O$, etc.

Let $OA = x$, $AB = BC = CA = a$. From the second theorem of Ptolemy we get
$$OA \cdot BC + OB \cdot AC \geq AB \cdot OC \Leftrightarrow 3a + 5a = a x \Leftrightarrow x \leq 8$$
The value $x = 8$ is attained when the quadrilateral $OACB$ is circumscribable, i.e. when $\angle AOB = 120^{\circ}$.
The point $B$ can be constructed as follows:
It is the point of intersection of the circle $(O,5)$ with the ray coming from the rotation of the ray $OA$ with center $O$ by an angle $\theta = -120^{\circ}$ (figure 10).

Figure 10