, 2019

$5x^2 - 18x + 17$
By the given conditions, we see that $P(x) = a(x-1)(x-2)$ with $a$ being a non-zero constant. We then see that we have
$$a^2(x-1)^2(x-2)^2 = P(x)^2 = R(x)^2 - Q(x)^2 = (R(x) + Q(x))(R(x) - Q(x))$$
valid for all $x$. We know that both $R(x) + Q(x)$ and $R(x) - Q(x)$ have degrees less than or equal to $2$, but since the left hand-side of the equation above has degree $4$, we see that both $R(x) + Q(x)$ and $R(x) - Q(x)$ have degree $2$. Therefore, $(R(x) + Q(x), R(x) - Q(x))$ can be written by using integers $b$ and $c$ in one of the following form:
$$(b(x-1)(x-2), c(x-1)(x-2)), (b(x-1)^2, c(x-2)^2), (b(x-2)^2, c(x-1)^2)$$
In the first possibility above, we get
$$Q(x) = \frac{1}{2}((R(x) + Q(x)) - (R(x) - Q(x))) = \frac{b-c}{2}(x-1)(x-2),$$
from which it follows that we have $b-c = Q(3) = 0$, but this will imply that the coefficient of $x^2$ term in $Q(x)$ is $0$, contradicting the assumption. Thus, the first possibility is not valid.
For the second possibility above, if we replace $Q$ by $-Q$, $R(x) + Q(x)$ and $R(x) - Q(x)$ get interchanged, and this case reduces to the third possibility. So, it suffices to consider only the third possibility.
In the third case, we have
$$Q(x) = \frac{1}{2}((R(x) + Q(x)) - (R(x) - Q(x))) = \frac{b}{2}(x-2)^2 - \frac{c}{2}(x-1)^2.$$
From $Q(3) = 0$ we get $b = 4c$, and from $a^2 = bc$, we get $a = \pm 2c$. We may assume that $a = 2c$ by replacing $P$ with $-P$ if necessary. Then, we get
$$P(x) = 2c(x-1)(x-2), \quad Q(x) = 2c(x-2)^2 - \frac{c}{2}(x-1)^2, \quad R(x) = 2c(x-2)^2 + \frac{c}{2}(x-1)^2.$$
Since the coefficient of $x^2$ in $R$ must be positive, we must have $c > 0$. Furthermore, since all the coefficients of $P$, $Q$, $R$ must be integers, we see, in particular, $\frac{c}{2}$ can be written as $c = 2d$ for some positive integer $d$. Then, all the coefficients of $P$, $Q$, $R$ become multiples of $d$, and therefore, we must have $d = 1$. We can therefore, conclude that
$$P(x) = 4(x-1)(x-2), \quad Q(x) = 3x^2 - 14x + 15, \quad R(x) = 5x^2 - 18x + 17$$
hold, and we get $5x^2 - 18x + 17$ is the desired solution to the problem.