I have two coordinate frames, A and B. I want to create the rotation matrix RAB which takes you from A to B. A is a right-handed system, and B is a left-handed system. Furthermore, after moving from a right- to a left-handed system, there is a further rotation. These two coordinate frames are illustrated in the image below.
As can be seen, it appears that the axes in B are simply the negative of the axes in A. So, my first attempt was to simply make these negative in the rotation matrix:
$$ R_{AB} = \left [\begin{array}{ccc} -1 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & -1 \\ \end{array} \right ]$$
However, after working through some examples, this did not work out.
Please could somebody answer either (or both!) these questions?
(a) Why does my above solution not hold?
(b) What is the correct rotation matrix RAB?

The example $R_{AB}$ you give is the correct transformation matrix to go from frame $A$ as shown to frame $B$ as shown; you may have worked your examples wrong if they seemed no to work. By the way, $R_{AB}$ is nto a rotation matrix (which would need to have determinant $+1$); you could call it an "improper rotation." No proper rotation matrix goes from a right-handed to a left-handed coordinate system.