I'm trying to establish conversion between coordinate frames of reference of a phone camera and onboard gyroscope. Because some phones flip Y axis of video, I do not want to limit solution to RHS<->RHS case.
Is there a quaternion way to encode a matrix with negative determinant, e.g. negative identity matrix?
If $n$ is a quaternion with real part zero such that $n^2=1$, then $q\mapsto -nqn$ achieves a reflection in the plane normal to $n$. (This maps $\mathbb H\to \mathbb H$, but you are mainly interested in the fact that it maps the pure quaternions into the pure quaternions, since that's your model of $3$-space.)
You can use this to transform between right and left handed coordinate systems.