a. We denote the reflections of $A$, $B$, $C$ and $D$ with $A'$, $B'$, $C'$ resp. $D'$ and we denote the intersection of the diagonals with $S$. Since the points $A$ and $C$ are reflected in the same line $BD$ and the point $S$ remains invariant under this reflection, the whole line $ASC$ becomes $A'SC'$ after reflection in $BD$. Analogously, the line $BSD$ becomes $B'SD'$ after reflection in $AC$.
If we denote the smaller angle between the two diagonals by $\varphi$, these two actions on the lines correspond to a rotation of the line $AC$ with center $S$ in direction $BD$ with rotation angle $2\varphi$ and a rotation of the line $BD$ with center $S$ in direction $AC$ with rotation angle $2\varphi$.
Therefore, the angle between the lines $A'SC'$ and $B'SD'$ is the angle $3\varphi$. This has to be a multiple of $180^{\circ}$, so that the original angle has to be $0^{\circ}$ or $60^{\circ}$. The first case is not possible since the points of the inscribed quadrilateral cannot lie on a line.
We obtain that the four new points lie on a line if and only if the diagonals of the given inscribed quadrilateral make an angle of $60^{\circ}$.

Figure 3: Problem 17

b. Since the reflections do not only preserve the collinearity of $ASC$ and $BSD$, but also the position of $S$ between the two points and the distances to the two points, we want to use the power of $S$ with respect to the circle $ABCD$.

Because of the reflections, we have
$$SA = SA', \quad SB = SB', \quad SC = SC', \quad SD = SD'$$

and since $ABCD$ is an inscribed quadrilateral, we have
$$SA \cdot SC = SB \cdot SD.$$
Therefore, we obtain
$$SA' \cdot SC' = SA \cdot SC = SB \cdot SD = SB' \cdot SD'.$$
Since the two lines $A'SC'$ and $B'SD'$ do not coincide in this case, we can apply the properties of the power of a point in reverse, and we get that $A'$, $B'$, $C'$ and $D'$ lie on a circle.