If we consider the sphere in $\mathbb{R}^3$, the set of all possible rotations of it forms a group isomorphic to $SO(3)$. That is, rotations of the sphere are exactly the orthogonal linear transformations from $\mathbb{R}^3$ to $\mathbb{R}^3$ with determinant $1$.
I understand why rotations of the sphere are linear transformations, and I also understand that since they all preserve length they must be orthogonal. What I don't understand is why are they necessarily of determinant $1$?
My intuition is that if we allow for negative determinant that would correspond to some reflection of the sphere, which would reverse its 'orientation' and that's impossible for rotation. I don't know how to make this intuition precise and follow up on it though.
Is there a way to precisely define the 'orientation' of a space and 'orientation-preserving transformations' and then conclude such transformations have non-negative determinants? If not, is there some other geometric notion that would explain why sphere rotations can't have determinant $-1$?
Here's a good reason why they all have determinant 1.
Consider a rotation about $v$ by some angle $s$, $Rot(v, s)$; you can connect this rotation to the "identity" rotation by $$ c(t) = Rot(v, s(1-t)) $$ where $t$ goes from $0$ to $1$.
This gives a continuous path in the space of rotations.
Thus
$$ det \circ c : [0, 1] \to \Bbb R: t \mapsto \det(c(t)) $$ is also continuous. And since the determinant must be $\pm 1$, and the set $\{+1, -1\}$ is disconnected, this function must be a constant function. (Any continuous function to a discrete space is constant). Hence, since the determinant of the identity is $+1$, so is the determinant of any rotation.
I know this isn't exactly what you asked, but the answer to your question about "Is there a precise way to define orientation" is "Sure, and it's done in many multivariable calculus and linear algebra books, and other places as well...and usually involves determinants." So repeating here what all those books say seems silly. [One such book, although I don't actually recommend it for this topic: Spivak's "Calculus on Manifolds".]
Hidden in the argument above is the claim that every non-identity rotation is actually rotation about some vector by some (nonzero) angle; that requires proof, but it's proved elsewhere as well, so I'm just using it as a lemma here. [See for instance the section on "Log map" in the Wikipedia entry on axis-angle representations.]