Solution:
For a positive integer n, let Z/nZ denote the set of residues modulo n and (Z/nZ)∗ denote the set of residues modulo n that are relatively prime to n. Then, rephrased, Sm is the set of residues modulo m of the form x+x−1, where x∈(Z/mZ)∗.
For part (a), suppose a=x+x−1∈Sm and b=y+y−1∈Sn. By the Chinese Remainder Theorem, there exists a residue z∈(Z/mnZ)∗ such that m∣(x−z) and n∣(y−z), and thus z+z−1≡x+x−1(modm) and z+z−1≡y+y−1(modn). Therefore, all f(m)f(n) residues modulo mn which result from applying the Chinese Remainder Theorem to an element each of Sm and Sn are in Smn. Conversely, given z+z−1∈(Z/mnZ)∗, taking z+z−1 modulo m and n gives elements of Sm and Sn, so indeed f(mn)=f(m)f(n).
We now proceed to part (b). For each x∈(Z/pkZ)∗, denote q(x) to be the largest non-negative integer i≤k such that pi divides x2−1 (this is clearly well-defined). For a given x, let g(x) be the number of y∈(Z/pkZ)∗ such that x+x−1≡y+y−1(modpk). Note that this condition is equivalent to (x−y)(xy−1)≡0(modpk).
First, consider the case in which q(x)≥k/2, in which case we have p⌈k/2⌉∣(x−1)(x+1), and because p is odd, x≡±1(modp⌈k/2⌉). Thus, either (x−1)2≡0(modpk) or (x+1)2≡0(modpk), and it follows that x+x−1≡±2(modpk) (clearly 2 and −2 are distinct). Conversely, x+x−1≡±2 implies (x±1)≡0(modp⌈k/2⌉), which in turn implies q(x)≥k/2. It is now clear that there are exactly two elements of Sm corresponding to residues of the form x+x−1 with q(x)≥k/2, and all other elements of Sm come from x with q(x)<k/2.
Fix x with q(x)<k/2; we will compute g(x). Suppose (x−y)(xy−1)≡0(modpk), and say x−y,xy−1 have j,j′ factors of p, respectively. If j≤j′, note that x2−1≡xy−1≡0(modpj), so j≤q(x). Similarly, if j′≤j, x≡y(modpj′), so x2−1≡xy−1≡0(modpj′), and so j′≤q(x). It follows that min(j,j′)≤q(x), and thus max(j,j′)≥k−q(x).
Suppose pk−q(x)∣(x−y). Then, xy−1≡x2−1≡0(modpq(x)) because q(x)<k/2, so any y with pk−q(x)∣(x−y) satisfies (x−y)(xy−1)≡0(modpk). Now, suppose pk−q(x)∣(xy−1), that is, y≡x−1(modpk−q(x)). Then, xy≡1(modpq(x)), and since x2≡1(modpq(x)), we have x≡y(modpq(x)), so again we have (x−y)(xy−1)≡0(modpk). It follows that the set of y satisfying (x−y)(xy−1)≡0(modpk) is exactly the set of y with y≡x(modpk−q(x)) or y≡x−1(modpk−q(x)). x,x−1 are distinct residues modulo pk−q(x), because x2−1 has fewer than k/2 factors of p, so it follows that g(x)=2pq(x). In particular, note that these values are distinct for different values of q(x)<k/2, so x+x−1≡y+y−1 implies q(x)=q(y) or q(x),q(y)≥k/2.
For each integer i with 0≤i<k/2, we need to compute the number of x∈(Z/pkZ) with q(x)=i. Clearly, this is the number of x with pi∣x2−1 minus the number of x with pi+1∣x2−1. When i=0, the number of x with pi∣x2−1 is clearly pk−1(p−1), and when i>0, this number is 2pk−i, as we have x≡±1(modpi).
We can now count the number of elements of Spk using casework on the value of q(x) where we take x to be such that x+x−1 is a particular element of Sm. Applying the results from the previous two paragraphs, our answer is
2+2pk−1(p−3)+i=1∑⌈k/2⌉−12pi2pk−i−2pk−i−1
where the summand 2 comes from ±2∈Spk, corresponding to all x with q(x)>k/2, the next summand comes from those x with q(x)=0, and each additional summand comes from those x with q(x)=i in the relevant range. We may evaluate the last sum as a geometric series, to obtain the final closed form answer of
2+2pk−1(p−3)+p+1pk−1−p21+(−1)k