I've often seen it written that $SU(2)$ is a "two-valued representation" of $SO(3)$ (in theoretical physics books mainly). I have a major conceptual issue with this however.
I know there is a Lie group isomorphism $SU(2)/\mathbb{Z}_2=SO(3)$ so we can assign to every matrix $R$ in $SO(3)$ one of two matrices $U$ or $-U$ in $SU(2)$. But surely the definition of a representation forces us to choose $D(I) = I$ so this "two-valued" business breaks down?
Could someone explain where I'm getting stuck? Also would anyone be able to point me to a resource which treats all this material rigorously? I have no background in Representation Theory and don't particularly want to read a really long text, but at the same time I am unhappy with the heuristic arguments of most physics books. Is there any lucid text taking some middle way?
"$SU(2)$ is a two-valued representation of $SO(3)$" is a rather bizarre sentence - it isn't totally clear what is meant by "two-valued" or "representation". One possible meaning, as suggested in the comments, is simply that $SU(2)$ is a double cover of $SO(3)$. But I suspect that something slightly different is meant.
$SU(2)$ has a canonical unitary representation $\pi$ on $\mathbb{C}^2$: namely, view an element of $SU(2)$ as a unitary operator on $\mathbb{C}^2$. As is well-known, this does not descend to an ordinary representation of $SO(3)$. One could try to define $\pi' \colon SO(3) \to U(\mathbb{C}^2)$ by $\pi'(g) = \pi(g')$ where $g'$ is a lift of $g$ to $SU(2)$ along the double cover $SU(2) \to SO(3)$, but the problem is there is no way to choose a lift $g'$ of every $g$ in such a way that $\pi'$ becomes a homomorphism. Nevertheless, there are only two choices for a lift of any $g$ and they differ only by a minus sign; thus while it is impossible to make $\pi'$ an actual homomorphism, it is a homomorphism up to sign:
$$\pi'(g_1 g_2) = \pm \pi'(g_1)\pi'(g_2)$$
Thus physicists sometimes like to think of the canonical representation of $SU(2)$ (which incidentally is the spin representation) as a "two valued" representation of $SO(3)$. This can be convenient if you only care about things like the position and momentum of a particle - which only depend on the magnitude of the wavefunction - and not about things like phase and spin.
Mathematicians have their own way to make sense of this. Recall that the projective general linear group of a vector space $V$ over a field $F$ is defined to be $GL(V)/F^\times$ where $F^\times$ is the multiplicative group of $F - 0$. One defines a projective representation of a group $G$ on $V$ to be a homomorphism $G \to PGL(V)$. If $V$ has a Euclidean / Hermitian structure (in the real / complex case), one can also speak of projective orthogonal / unitary representations; these are homomorphisms into the unitary group of $V$ modulo the unit group of the ground field. With these definitions, the discussion above implies that the canonical representation of $SU(2)$ is in fact a projective orthogonal representation of $SO(3)$ (since the unit group of $\mathbb{R}$ is just $\{\pm 1\}$).
So in conclusion, I interpret the sentence "$SU(2)$ is a two-valued representation of $SO(3)$" to mean "the canonical representation of $SU(2)$ is a projective representation of $SO(3)$".