Let $R$ be a number ring that is Dedekind. Show that the norm function $N:I \to [R:I]$ on $R$-ideals extends to a homomorphism $N: \mathcal{I}(R) \to \mathbb{Q}^*$ where $\mathcal{I}(R)$ is the set of all invertible ideals.
Note that if $R$ is Dedekind domain then every ideal of $R$ is invertible and if $I,J \subset R$ then $N(I \cdot J)=[R:I][R:J]=N(I)\cdot N(J)$. But I don't know what can replace $[R:I]$ if $I \not \subset R$.
I think you mean the following: $\newcommand{Int}{\operatorname{Int}}$ $\newcommand{\Frac}{\operatorname{Frac}}$
$$ \Int R \rightarrow \mathbb{Z}^+, \ I \in \Int R \mapsto N(I) = \# R/I$$
The extension is as follows: for a fractional $R$-ideal $K$, write it as $I/J$ for nonzero integral $R$-ideals and put
$$ N(I/J) = N(I)/N(J) \in \mathbb{Q}^+.$$
Why does it have to be this? Well, $\Int R$ is a commutative monoid under multiplication of ideals and the already defined norm map is a monoid homomorphism: for all $I,J \in \Int R$ we have $N(IJ) = N(I)N(J)$. The subgroup of $\Frac R$ generated by $\Int R$ is $\Frac R$ itself, so the above map is the only possible extension. Moreover the map is well-defined: if $I_1/J_1 = I_2/J_2$ then $I_1 J_2 = I_2 J_1$ so $N(I_1) N(J_2) = N(I_2) N(J_1)$ and thus $N(I_1)/N(J_1) = N(I_2)/N(J_2)$.
Remark: In fact $\Int R$ is the free commutative monoid on the set of prime ideals and $\Frac R$ is the free commutative group on the set of prime ideals, which means that $\Frac R$ is the group completion of $\Int R$. The extension then follows from the universal property of the group completion of a commutative monoid: if $M$ is a commutative monoid with group completion $G(M)$ and $f: M \rightarrow H$ is a monoid homomorphism into a commutative group $H$, then $f$ extends uniquely to a group homomorphism from $G(M)$ to $H$. Again, the extension is just $f(m/n) = f(m)/f(n)$.
Also, by some coincidence(?) I am working on notes involving non-maximal orders in (e.g.) number fields. If $R$ is an order in $K$ but not necessarily the full ring of integers -- i.e., $R$ is a subring of the ring of integers with fraction field $K$ -- then all of the above holds provided you replace $\Int R$ with the invertible integral ideals and $\Frac R$ with the invertible fractional ideals. For this the only nontrivial thing is the multiplicativity of the norm map, for which see Proposition 22.4 of my commutative algebra notes.