, 2020

We fix $\kappa > 1$. The idea for the solution of the problem is to find $\alpha, \beta$ with respect to $\kappa$ such that the denominator of the fraction is equal to $1$. In other words, we seek polynomials $\alpha = P(\kappa)$, $\beta = Q(\kappa)$ such that
$$P(\kappa)^2 + \kappa^2 Q(\kappa)^2 - \kappa^2 P(\kappa)Q(\kappa) = 1. \\ P(\kappa)^2 = \kappa^2 Q(\kappa)(P(\kappa) - Q(\kappa)) + 1 \quad (1)$$
If $m = \deg P(\kappa), n = \deg Q(\kappa)$ with $m > n \ge 0$, then from (1) it follows that
$$2m = 2 + m + n \Leftrightarrow m = n + 2.$$
We will deal with the easiest case, by seeking polynomials of degrees $2$ and $0$, respectively, which satisfy relation (1).
Finally, we find that $P(\kappa) = \kappa^2 - 1$, $Q(\kappa) = 1$ satisfy relation (1). We choose $\alpha = \kappa^2 - 1$, $\beta = 1$ and then the expression is equal to $\frac{\kappa^2 - 1 + 1}{1} = \kappa^2$, which is composite for all $\kappa > 1$. Therefore, all values of $\kappa > 1$ are not solutions.

For $\kappa = 1$, we will prove that: $0 < A(\alpha, \beta, \kappa) = \frac{\alpha + \beta}{\alpha^2 + \beta^2 - \alpha\beta} \le 2$, which means that the rational number $A(\alpha, \beta, 1)$ cannot be a composite positive integer for all $\alpha, \beta$. Indeed, we have
$$\alpha + \beta > 0 \quad \text{and} \quad \alpha^2 - \alpha\beta + \beta^2 = \left(\alpha - \frac{\beta}{2}\right)^2 + \frac{3\beta^2}{4} > 0, \text{ and}$$
$$\frac{\alpha + \beta}{\alpha^2 + \beta^2 - \alpha\beta} \le 2 \Leftrightarrow 2\alpha^2 + 2\beta^2 - 2\alpha\beta \ge \alpha + \beta \Leftrightarrow (\alpha - \beta)^2 + \alpha^2 - \alpha + \beta^2 - \beta \ge 0 \\ \Leftrightarrow (\alpha - \beta)^2 + \alpha(\alpha - 1) + \beta(\beta - 1) \ge 0, \text{ which holds.}$$