a) If $x, y \in G$, then $(xy)^{p+1} = a^{-1} x y a = a^{-1} x x a^{-1} y a = x^{p+1} y^{p+1}$. We can write $x(yx)^p y = x^{p+1} y^{p+1}$, then $(yx)^p = x^p y^p$. For $x = a$ we get $a^p = e$ so by the preceding equality $(y a)^p = y^p$. Multiplying at left by $y a$ we obtain $y a y^p = (y a)^{p+1} = y^{p+1} a$, that is $a y^p = y^p a$, for all $y \in G$. From the hypothesis we have $y^{p(p+1)} = a^{-1} y^p a = y^p$, so $y^{p^2} = e$, for all $y \in G$. Because $p$ is a prime, every element of the group has order $1$, $p$ or $p^2$ and by the Cauchy theorem we deduce $\text{ord}(G) = p^k$, for a $k \in \mathbb{N}^*$.

b) For $x, y \in H$, we have $(xy)^p = y^p x^p = e$, that is $xy \in H$, proving that $H$ is a stable part of $G$, and, as it is finite, $H$ is a subgroup.

Consider $f : G \to G$, given by $f(x) = x^p$. Because $e = x^{p^2} = (x^p)^p$, the image of $f$ is contained in $H$. Moreover, $x, y \in G$ and $f(x) = f(y)$ imply $x^p (y^{-1})^p = e$, so $(y^{-1} x)^p = e$, that is $y^{-1} x \in H$. This gives $x \in H y$. We conclude that for every element in $\text{Im} f$, the number of its pre-images in $G$ is exactly $\text{ord}(H)$, so $|\text{Im} f| = \frac{\text{ord}(G)}{\text{ord}(H)}$.

Because $a \neq e$ we get $a \notin \text{Im} f$: for if not $a = b^p$ for some $b \in G$. This would imply $b^{p+1} = b^{-p} b b^p$, that is $e = b^p = a$ a contradiction. As $a \in H$, we conclude $\text{ord}(H) > |\text{Im} f| = \frac{\text{ord}(G)}{\text{ord}(H)}$, which gives the conclusion.