Problem 1184

By applying the formula $\binom{a}{b} = \frac{a!}{b!(a-b)!}$ we obtain the equation
$$1 + \frac{(n-k)^2}{(k+1)^2} = \frac{(n-k)^2(n-k-1)^2}{(k+1)^2(k+2)^2} \binom{n}{k+2}^2.$$
Hence $(k+2)^2 [(k+1)^2 + (n-k)^2] = (n-k)^2(n-k-1)^2 \binom{n}{k+2}^2$, which implies that $(k+1)^2 + (n-k)^2$ is a perfect square.
Let $(k+1)^2 + (n-k)^2 = t^2$, where $t \in \mathbb{N}$. We have
$$(k+2)t = (n-k)(n-k-1) \binom{n}{k+2} \ge 2 \binom{n}{k+2}.$$
Using that $k+2 \le n$ and $t = \sqrt{(k+1)^2 + (n-k)^2} < n+1$ we conclude that $(k+2)t \le n^2$.
Let $3 \le k+2 \le n-3$ (the left hand side of this inequality follows from the condition of the problem). If $n \ge 6$, we have
$$2 \binom{n}{k+2} \ge 2 \binom{n}{3} = \frac{n(n-1)(n-2)}{3} > n^2,$$
i.e. the equation has no solution in this case.
When $k+2 = n-2$ we obtain $t^2 = (n-3)^2 + 16$, hence $t = 5$, $n = 6$, $k = 2$. Direct computation shows that $n = 6$ and $k = 2$ is not a solution. If $k+2 = n-1$ we have $t^2 = (n-2)^2 + 9$, so $t = 5$, $n = 6$, $k = 3$ and as above we conclude that no solution exists. Therefore positive integers $n$ and $k$ satisfying the equation do not exist.