Let $R$ denote the rectangle, and let $S$ denote the square with minimum length that covers $R$. Let $s$ denote the length of a side of $S$. We claim that $R$ is inscribed in $S$, that is, the vertices of $R$ lie on the sides of $S$. We also claim that $R$ can only be inscribed in two ways, as shown below. Let $S_1(S_2)$ denote the square shown on the left-hand side (right-hand side). For $S_2$, the sides of $R$ are parallel to the diagonals of $S_2$. It is easy to see that $s = a$ if $S = S_1$.

$$s = \begin{cases} a, & \text{if } a < (\sqrt{2}+1)b, \\ \frac{\sqrt{2}(a+b)}{2}, & \text{if } a \ge (\sqrt{2}+1)b. \end{cases}$$

Now we prove our claim that these are only two ways. Let $R = ABCD$ and $S = XYZW$. Without loss of generality, we place $XY$ horizontally. By the minimality of $S$, we can assume that at least one vertex, say $A$, of $R$ lies on one side of $S$, say $WX$ (see the left-hand side figure shown below). If neither $B$ nor $D$ lies on the sides of $S$, we can then slide $R$ down (vertically), so that one of them, say $B$ lies on side $XY$ (see the middle figure shown below). If neither $C$ or $D$ lies on the sides of $S$, then we can apply an enlargement, centered at $X$ with scale less than 1, to $S$ such that the image of $S$ still covers $R$. This violates the minimality of $S$. Hence at least one of $C$ and $D$ lies on the sides of $S$, that is, three consecutive vertices of $R$ lie on the sides of $S$ (see the right-hand side figure shown below). Without loss of generality, we assume that they are $A$, $B$ and $C$. If any of these three vertices coincide with any of the vertices of $S$, then we clearly have $S = S_1$. Hence we may assume that $A$, $B$ and $C$ are on sides $WX$, $XY$ and $YZ$, respectively. By symmetry, we may also assume that $AB = a > b = BC$.

If $D$ does not lie on line segment $ZW$, then we can slide $R$ up a bit so both $B$ and $D$ lie in the interior of $S$ (see the left-hand side figure shown below). Let $O$ be the center of $R$. We can then rotate $R$ around $O$ with a small angle so that all four vertices lie inside $S$ (see the middle figure shown below). It is easy to see that we can use a smaller square to cover $R$ (by applying an enlargement centered at $O$ with a scale less than 1), violating the minimality of $S$. Thus our assumption was wrong, and $D$ must lie on side $ZW$ (see the right-hand side figure shown below), which is the case when $S = S_2$.

We finish our proof that if $S = S_2$, then the sides of $R$ are parallel to the diagonals of $S_2$. By symmetry, it suffices to show that $AX = XB$. It is not difficult to see that $\angle XAB = \angle YBC = \angle ZCD = \angle WDA$. So triangles $ABX$, $BCY$, $CDZ$ and $DAW$ are similar. Set $AX = ax$ and $XB = ay$. Then $BY = bx$ and $CY = by$. Also, $DW = bx$ and $WA = by$. Hence $by + ax = WA + AX = WX = XY = XB + BY = ay + bx$, implying that $(a-b)x = (a-b)y$, or $x = y$, as desired.