Solution:

We have $(x y)^{2} + (x + y)^{2} = 40$ and $x y + (x + y) = 8$.

Squaring the second equation and subtracting the first gives $x y (x + y) = 12$.

So $x y$, $x + y$ are the roots of the quadratic $a^{2} - 8a + 12 = 0$.

It follows that $\{x y, x + y\} = \{2, 6\}$.

If $x + y = 2$ and $x y = 6$, then $x, y$ are the roots of the quadratic $b^{2} - 2b + 6 = 0$, which are non-real, so in fact $x + y = 6$ and $x y = 2$, and $x, y$ are the roots of the quadratic $b^{2} - 6b + 2 = 0$.

Because $x > y$, we take the larger root, which is $\frac{6 + \sqrt{28}}{2} = 3 + \sqrt{7}$.