To prove that $n = 1$ is the only possible value for which the set $\mathbb{Z} \backslash \{ax^n + by^n \mid x,y \in \mathbb{Z}\}$ is finite, we will consider the cases where $n \neq 1$ and show that in these cases, there are infinitely many integers that cannot be expressed in the form $ax^n + by^n$.

1. **Assume $n \neq 1$**:
- We want to show that there are infinitely many integers not of the form $ax^n + by^n$.
- If $g = \gcd(a, b) > 1$, then we can take all integers that are $1 \mod g$. Thus, we assume $g = 1$.

2. **Case 1: $n \neq 2$**:
- Take any prime $p \ge 7$ such that $p \equiv 1 \mod n$. By Dirichlet's Theorem on arithmetic progressions, there are infinitely many such primes.
- The function $f(x) = x^n$ assumes only $\frac{p-1}{n} + 1$ different values modulo $p$:
- $f(0) = 0$ is one value.
- For $x \neq 0$, consider $f$ as a group endomorphism on $(\mathbb{Z}/p\mathbb{Z})^*$. The kernel consists of all $x$ such that $x^n = 1$. Since $(\mathbb{Z}/p\mathbb{Z})^*$ is cyclic and $n \mid (p-1)$, there are $n$ such values.
- By the First Isomorphism Theorem, the image of $f$ has size $\frac{p-1}{n}$. Including $0$, there are $\frac{p-1}{n} + 1$ values for $x^n \mod p$.
- Therefore, $ax^n + by^n$ assumes at most $2\left(\frac{p-1}{n} + 1\right)$ distinct values modulo $p$. If $n > 2$, then (since $p \ge 7$):
$$2\left(\frac{p-1}{n} + 1\right) \leq 2\left(\frac{p-1}{3} + 1\right) \leq p-1 < p$$
- Hence, there is an entire equivalence class modulo $p$ which is not of the form $ax^n + by^n$.

3. **Case 2: $n = 2$**:
- The previous argument does not work, and we need a different approach.
- If $a$ and $b$ have the same sign, the numbers $ax^2 + by^2$ have the same sign too, and we can choose all the numbers of the opposite sign.
- Assume without loss of generality that $a \ge 0 \ge b$.
- If $ab = 0$, the case is simple. For example, if $a = 0$ and $b \neq 0$, then the values $(x^2 + 1)b$ are never assumed. If $a = b = 0$, then only $0$ is assumed.
- Subcase 1: $a$ and $-b$ are both squares. All the values of the set are differences of squares, and they are never equal to $2 \mod 4$.
- Subcase 2: $a$ and $-b$ are not both squares.
- Let $p$ be a prime such that $(p, ab) = 1$. If $p = ax^2 + by^2$, then $p$ does not divide $x$ or $y$. If $p$ divides $x$, it also divides $y$, and hence $p^2$ divides the RHS while it does not divide the LHS.
- If $p = ax^2 + by^2$, then $-ab^{-1}$ is a square modulo $p$, which is equivalent to $-ab$ being a square modulo $p$.
- If we can find infinitely many primes $p$ such that $\left(\frac{-ab}{p}\right) = -1$, we are done. This means $ax^2 + by^2$ is never equal to these primes.
- This is equivalent to $\left(\frac{t}{p}\right) = -1$, where $t$ is the square-free part of $-ab$ (and $t \neq 1$ since $(a, b) = 1$ and they are not both squares). Since $t \neq 1$ and $\left(\frac{t}{p}\right)$ is identically $1$ for all primes $p$ (except for finitely many) only for $t = 1$, we are done.

4. Proof of the claim:
- Decomposing $t$ into distinct primes: $t = \prod_{i=1}^{k} p_i$, and using properties of the Legendre symbol (multiplicative!) and quadratic reciprocity, shows that $\left(\frac{t}{p}\right) = -1$ is equivalent to:
$$\prod_{p_i \neq 2} \left(\frac{p}{p_i}\right) = (-1)^{1 + \frac{p-1}{2} \frac{\sum_{p_i \neq 2} p_i - 1}{2}} \cdot (-1)^{\frac{p^2-1}{8}} \text{ if } 2 \mid t$$
- By the Chinese Remainder Theorem, the LHS is $1$ if and only if $p$ is equivalent to certain values modulo $\frac{t}{(t, 2)}$ (since the Legendre symbol is periodic: $\left(\frac{x}{p_i}\right)$ has period $p_i$), and the RHS is $1$ if and only if $p$ is equivalent to certain values modulo $8$. Hence, there is no identity in the above equation, and we can choose primes with specific values modulo $8\frac{t}{(t, 2)}$ that will violate the equality.
- We use Dirichlet's Theorem, but it can be avoided (see the chapter about Quadratic Reciprocity in Ireland and Rosen).

Since we have shown that for $n \neq 1$, there are infinitely many integers not of the form $ax^n + by^n$, it follows that the only possible value for $n$ is $n = 1$.

$\blacksquare$