a.

We show that
$$\int_0^a f(x) dx \le a.$$
To this end, write
$$\begin{aligned} a - \int_0^a f(x) dx &= a \int_0^1 f(x) dx - \int_0^a f(x) dx \\ &= a \int_a^1 f(x) dx - (1-a) \int_0^a f(x) dx \\ &\ge a \int_a^1 f(a) dx - (1-a) \int_0^a f(a) dx \\ &= 0. \end{aligned}$$
Here equality holds if and only if $f(x) = 1$ for $0 < x < 1$; at $x = 0$ the function may take any non-negative value less than or equal to $1$, and at $x = 1$ it may take any value greater than or equal to $1$.

Therefore,
$$\max_{f \in C} \int_0^a f(x) dx = a.$$

b.

The required maximum is $a$ if $a \le 1/2$, and $1/(4(1-a))$ if $a > 1/2$. In both cases, it is achieved at an essentially unique function. In the former case, the maximum is achieved at $f \equiv 1$ on the open interval $(0, 1)$; at $x = 0$ the function may take any non-negative value less than or equal to $1$, and at $x = 1$ it may take any value greater than or equal to $1$. In the latter case, the maximum is achieved at
$$f(x) = \begin{cases} 0 & \text{if } 0 \le x < 2a - 1, \\ \frac{1}{2(1-a)} & \text{if } 2a - 1 < x < 1; \end{cases}$$
at $x = 2a - 1$ the function may take any non-negative value less than or equal to $1/(2(1-a))$, and at $x = 1$ it may take any value greater than or equal to $1/(2(1-a))$.

Let $f$ be a function in $C$. Begin by noticing that
$$\begin{aligned} \int_0^a (f(x))^2 dx &\le f(a) \int_0^a f(x) dx \\ &\le \left( \frac{1}{1-a} \int_a^1 f(x) dx \right) \left( \int_0^a f(x) dx \right) \\ &= \frac{1}{1-a} \left( 1 - \int_0^a f(x) dx \right) \left( \int_0^a f(x) dx \right). \end{aligned}$$
Next, as shown above,
$$\int_0^a f(x) dx \le a.$$
Further, notice that
$$\max \{ t(1-t) : t \le a \} = \begin{cases} a(1-a) & \text{if } a \le 1/2, \\ \frac{1}{4} & \text{if } a > 1/2; \end{cases}$$
in both cases, the maximum is achieved at a single point: at $t = a$ in the former case, and at $t = 1/2$ in the latter.

Consequently,
$$\int_0^a (f(x))^2 dx \le \begin{cases} a & \text{if } a \le 1/2, \\ 1/(4(1-a)) & \text{if } a > 1/2. \end{cases}$$
If $a \le 1/2$, then
$$\int_0^a (f(x))^2 dx = a$$
forces
$$\int_0^a f(x) dx = a,$$
so $f(x) = 1$ for $0 < x < 1$, by the third paragraph above; at $x = 0$ the function may take any non-negative value less than or equal to $1$, and at $x = 1$ it may take any value greater than or equal to $1$.

If $a > 1/2$, then
$$\int_0^a (f(x))^2 dx = \frac{1}{4(1-a)}$$
forces
$$\int_0^a f(x) dx = \frac{1}{2} \qquad (1)$$
and
$$\int_0^a (f(x))^2 dx = f(a) \int_0^a f(x) dx, \qquad (2)$$
so
$$\int_a^1 f(x) dx = \frac{1}{2} \quad \text{and} \quad f(a) = \frac{1}{2(1-a)}.$$
The last two conditions force in turn $f(x) = 1/(2(1-a))$, $a \le x < 1$; at $x = 1$ the function may take any value greater than or equal to $1/(2(1-a))$.

We now show that (1) and (2) force
$$f(x) = \begin{cases} 0 & \text{if } 0 < x < 2a - 1, \\ \frac{1}{2(1-a)} & \text{if } 2a - 1 < x < a. \end{cases}$$
Of course, $f(0) = 0$, and at $x = 2a - 1$ the function may take any non-negative value less than or equal to $1/(2(1-a))$. To prove this, consider the three sets below:
$$A = \{ x : 0 < x < a, f(x) = 0 \},$$
$$B = \{ x : 0 < x < a, 0 < f(x) < f(a) \},$$
$$C = \{ x : 0 < x < a, f(x) = f(a) \}.$$
Clearly, if $x, y, z$ are members of $A, B, C$, respectively, then $x < y < z$.
Notice that $B$ contains at most one point. Otherwise, consider two points $b < b'$ in $B$ to contradict (2):
$$0 < \int_b^{b'} f(x) (f(a) - f(x)) dx \le \int_0^a f(x) (f(a) - f(x)) dx = 0.$$
If $A$ is empty, then so is $B$ and $C = (0, a)$ and
$$\int_0^a f(x) dx = a f(a) = \frac{a}{2(1-a)} > \frac{1}{2},$$
which contradicts (1).
Similarly, if $C$ is empty, then again so is $B$ and $A = (0, a)$ and
$$\int_0^a f(x) dx = 0,$$
which contradicts again (1).
Consequently, neither $A$ nor $C$ are empty, $\sup A = b = \inf C$ for some $b$ in $(0, a)$, and $(0, b) \subseteq A$ and $(b, a) \subseteq C$.
Finally, $b$ is determined by (1):
$$\frac{1}{2} = \int_0^a f(x) dx = \int_b^a \frac{1}{2(1-a)} dx = \frac{a-b}{2(1-a)},$$
so $b = 2a - 1$. The conclusion follows.

Let again $f$ be a function in $C$. If $f(a) < 1$, then
$$\int_0^a (f(x))^2 dx \le \int_0^a (f(a))^2 dx = a(f(a))^2 < a.$$
If $f(a) \ge 1$, consider the function
$$\varphi: [0, 1] \to [0, \infty),$$
defined by
$$\varphi(t) = \int_0^t f(x) dx + (1-t)f(a).$$
Clearly, $\varphi$ is continuous and it is easily seen that $\varphi$ is decreasing on $[0, a]$ and increasing on $[a, 1]$. Since $\varphi(1) = 1$, it follows that $\varphi(a) \le 1$ (this can also be established directly), and since
$$\varphi(0) = f(a) \ge 1,$$
there exists $b$ in $[0, a]$ such that $\varphi(b) = 1$, by continuity; that is,
$$\int_0^b f(x) dx = 1 - (1 - b)f(a).$$
Next, define $g: [0, 1] \to [0, \infty)$ by
$$g(x) = \begin{cases} f(x) & \text{if } 0 \le x \le b, \\ f(a) & \text{if } b < x \le 1. \end{cases}$$
It is readily checked that $g$ belongs to $C$; moreover, if $0 \le x \le a$, then
$$g(x) \ge f(x) \ge 0,$$
so
$$\int_0^a (g(x))^2 dx \ge \int_0^a (f(x))^2 dx.$$
On the other hand,
$$\begin{aligned} \int_0^a (g(x))^2 dx &\le g(a) \int_0^a g(x) dx \\ &= f(a) \left( \int_0^b f(x) dx + (a-b)f(a) \right) \\ &= f(a) (1 - (1-b)f(a) + (a-b)f(a)) \\ &= f(a) (1 - (1-a)f(a)). \end{aligned}$$
Further, notice that
$$\max \{ t(1 - (1 - a)t) : t \ge 1 \} = \begin{cases} a & \text{if } a \le 1/2, \\ 1/(4(1-a)) & \text{if } a > 1/2; \end{cases}$$
in both cases, the maximum is achieved at a single point: at $t = 1$ in the former case, and at $t = 1/(2(1-a))$ in the latter.

Consequently,
$$\int_0^a (f(x))^2 dx \le \int_0^a (g(x))^2 dx \le \begin{cases} a & \text{if } a \le 1/2, \\ 1/(4(1-a)) & \text{if } a > 1/2. \end{cases}$$
Proceed now along the lines in the corresponding part of the previous solution.