Turkish Mathematical Olympiad

Answer: $(x, y) = (-1/3, -1/3)$, $(5/3, 5/3)$.
Firstly, we obtain that
$$x^2 + y^2 + x + y - xy(x + y) + 2 = -(1-x)(1-y)(x+y+2)$$
and hence we get
$$(1-x)(1-y)(x+y+2) = \frac{64}{27}.$$
Let $k^3 = 1-x$, $\ell^3 = 1-y$, $m^3 = x+y+2$. Then we have
$$k^3 + \ell^3 + m^3 = 4 \quad \text{and} \quad k\ell m = \frac{4}{3}.$$
This shows that $k^3 + \ell^3 + m^3 = 3k\ell m$ and it can be expressed as
$$(k + \ell + m)((k - \ell)^2 + (\ell - m)^2 + (m - k)^2) = 0.$$
The last equality implies that either $k + \ell + m = 0$ or $k = \ell = m$.
For $k = \ell = m$, we obtain the solution $x = y = -1/3$.
Let $k + \ell + m = 0$. In this case using $k\ell m = 4/3$, we get $k\ell(k + \ell) = -4/3$. Consider the condition $|xy| = |(1-k^3)(1-\ell^3)| \le 25/9$. Define $u = k + \ell$, $v = k\ell$. In this case, we get
$$uv = -\frac{4}{3} \quad \text{and} \quad |(1-k^3)(1-\ell^3)| = |u^3 - v^3 + 3| \le \frac{25}{9}.$$
These two conditions imply that
$$-\frac{52}{9} \le u^3 - v^3 = u^3 + \frac{64}{27u^3} \le -\frac{2}{9} \quad (*)$$
By AM-GM, we get $u^2 \ge 4v = -16/(3u)$ which is equivalent to $u > 0$ or $u^3 \le -16/3$. From (*), we conclude that $u < 0$ and hence $u^3 \le -16/3$. In this case since $u^3 + 16/3 \le 0$ and $4/(9u^3) + 1 \ge 11/12 > 0$ we get that
$$u^3 + \frac{64}{27u^3} + \frac{52}{9} = \left(u^3 + \frac{16}{3}\right) \left(\frac{4}{9u^3} + 1\right) \le 0$$
Therefore, (*) holds only if $u^3 = -16/3$ and $u^2 = 4v$ which implies that $k = \ell = -\sqrt[3]{2}/3$. This yields the solution $x = y = 5/3$. Both obtained solutions satisfy the problem conditions.

Let $x+y=a$, $xy=b$. Then by AM-GM, we have $a^2 \ge 4b$ and hence
$$x^2 + y^2 + x + y - xy(x + y) = a^2 + a - b(a + 2) = -\frac{10}{27},$$
$$b = \frac{a^2 + a + \frac{10}{27}}{a + 2} \le \frac{a^2}{4},$$
$$|b| = \left| \frac{a^2 + a + \frac{10}{27}}{a + 2} \right| \le \frac{25}{9}.$$
Therefore, we get
$$\frac{a^2}{4} - \frac{a^2 + a + \frac{10}{27}}{a + 2} = \frac{\left(a + \frac{2}{3}\right)^2 \left(a - \frac{10}{3}\right)}{4(a + 2)} \ge 0 \quad (1),$$
$$\frac{a^2 + a + \frac{10}{27}}{a + 2} - \frac{25}{9} = \frac{\left(a - \frac{10}{3}\right)\left(a + \frac{14}{9}\right)}{a + 2} \le 0 \quad (2),$$
$$\frac{a^2 + a + \frac{10}{27}}{a + 2} + \frac{25}{9} = \frac{\left(a + \frac{17}{9}\right)^2 + \frac{191}{81}}{a + 2} \ge 0 \quad (3).$$
Using (1), (2) and (3), we obtain the following solutions:
$$(1) \iff a < -2 \quad \text{or} \quad a \ge \frac{10}{3} \quad \text{or} \quad a = -\frac{2}{3},$$
$$(2) \iff a < -2 \quad \text{or} \quad -\frac{14}{9} \le a \le \frac{10}{3},$$
$$(3) \iff a > -2.$$
Hence, the only solutions are $a = -\frac{2}{3}$ and $a = \frac{10}{3}$. In both cases, $a^2 = 4b$ and it follows that $x = y = \frac{a}{2}$. The obtained solutions $(x, y) = (-1/3, -1/3)$, $(5/3, 5/3)$ satisfy the problem conditions.