Clearly $a$ must be a prime (set $x=0$). More exactly $a$ is an odd prime because $a=2$ is not a solution. Furthermore $a+1$ must be a power of 2. Indeed suppose that $a+1$ has an odd prime divisor $p$. Note that $p \le \frac{a+1}{2}$ as $a+1$ is even. Set $x = \frac{1}{2}(p-1)$; it is clear that $0 < x < a$. We have

$$4x^2 + a = 4 \cdot \frac{1}{4} (p-1)^2 + a = p(p-2) + (a+1). \text{ Since } p \text{ divides } a+1, \text{ it also divides } 4x^2 + a.$$
addition, $p < a < 4x^2 + a$, hence $4x^2 + a$ is composite.

The primes $3 = 2^2 - 1$ and $7 = 2^3 - 1$ satisfy the conditions. The values of $4x^2 + 3$ for $x = 0, 1, 2$ are the primes $3, 7, 19$; the values of $4x^2 + 7$ for $x = 0, 1, 2, 3, 4, 5, 6$ are the primes $7, 11, 23, 43, 71, 107, 151$. We show that $a=3$ and $a=7$ are the only solutions by rejecting all $a = 2^m - 1$ with $m \ge 4$. For numbers of this form consider $a+9 = (a+1)+8$. This even number is not a power of 2 (powers of 2 greater than 8 cannot differ by 8). Let $q \le \frac{a+9}{2}$ be an odd prime divisor of $a+9$. Set $x = \frac{1}{2}(q-3)$:
$$\text{note that } x \text{ may be zero but is less than } a. \text{ Now } 4x^2 + a = 4 \cdot \frac{1}{4}(q-3)^2 + a = q(q-6) + (a+9), \text{ so } q$$
divides $4x^2 + a$. Also $q \le \frac{a+9}{2} < a$ as $a \ge 15$, hence $q < a \le 4x^2 + a$. Then $4x^2 + a$ is composite, which completes the proof. The answer is $a=3$ and $a=7$.