Solution:

The answer is $\mathbf{(B)}$. Let us draw the line $r$ perpendicular to the side $AD$ and passing through $X$. Let $H$ be the intersection between $r$ and $AD$, and let $K$ be the intersection between $CB$ and $r$. Let $J$ be the projection of $X$ onto the side $CD$. Let us first show that triangles $DHX$ and $DAY$ are similar. Indeed, $\angle XDH = \angle YDA$ since it is shared by the two triangles, and $\angle DHX = \angle DAY = 90^\circ$, since the first is the angle formed by the two perpendicular lines $HX$ and $AD$, while the second is an angle of a square; the first similarity criterion for triangles leads to the conclusion we sought. From this we derive that
$$\frac{HX}{DH} = \frac{AY}{AD} = \frac{AY}{AB}.$$
We now observe that $CKDJ$ forms a square: indeed it has three angles equal to $90^\circ = \angle CJX = \angle XKC = \angle KCJ$, the first two because they are angles formed by two orthogonal lines and the last because it is an angle of a square. Therefore we have $KX = CK = CJ$. Let us call $\ell$ the side of the square. Since $CX = CB = \ell$, the radius of the circle, we have that $CX = \sqrt{2} \cdot CK$. $KCDH$ is a rectangle, because it has all right angles, hence we can obtain the relation $CK = DH$, and by difference of segments $HX = HK - XK = DC - JC$. Putting together what we have found, we obtain that $DH = CK = \frac{\ell}{\sqrt{2}}$ and that $HX = DC - CJ = \ell - \frac{\ell}{\sqrt{2}} = \left(\frac{2-\sqrt{2}}{2}\right) \ell$. We can now conclude, since
$$\frac{AY}{AB} = \left(\frac{2-\sqrt{2}}{2}\right) \frac{1}{\sqrt{2}} = \sqrt{2} - 1$$