In chapter 6, section 4 of Griffiths--Harris, Principles of Algebraic Geometry, on page 604 (at least in my version) one reads in point 4:
As the reader may check by the same method as used in the case of blow-ups of a point, blow-ups of submanifolds are unique, in the sense that if $\pi\colon N\to M$ is any map of complex manifolds that is an isomorphism away from a smooth subvariety $X$ of dimension $k$ in $M$, and such that the fiber of $\pi$ over any point $x\in X$ is isomorphic to projective space $\mathbb{P}^{n-k-1}$, then $\pi\colon N\to M$ is the blow-up of $M$ along $X$.
In other words, if a morphism passes the duck test for blow-ups (if it looks like a blow-up, and quacks like a blow-up, it is indeed a blow-up. I was wondering where I could find a reference where this is actually proven (or analogous statements are discussed).
Uniqueness statements often arise from universal properties. Objects defined via a universal property are always unique up to a unique isomorphism. This is what you call "to pass the duck test"! And it is in fact the case for blowups. (Below are references.)
Say $X\subset M$ is a subvariety and $\dim M=n$, $\dim X=k$.
Definition. The blowup of $M$ at $X$ is the Cartesian square $$\require{AMScd} \begin{CD} E @>>> B_XM \\ @VVV @VVV \\ X @>>> M \end{CD}$$ universal with respect to the property that $E$ is an effective Cartier divisor on $B_XM$.
As usual with universal properties, this can be rephrased by saying that such a fiber square is the final object in an appropriate category. In a way or another, the blowup is unique up to unique isomorphism, if it exists.
Theorem. The blow up exists and is given as $B_XM=\textrm{Proj }\bigoplus_{d\geq 0}\mathscr I^d$, where $\mathscr I\subset \mathscr O_M$ is the ideal of $X$ in $M$.
Corollary. If $X$ and $M$ are smooth (implicit in Griffiths-Harris' book), then $E=\mathbb P(N_{X/M})$, where $N_{X/M}$ is the normal bundle to $X$ in $M$.
Proof of Corollary. You can write $$\begin{align}E&=B_XM\times_MX\notag\\ &=\bigl(\textrm{Proj }\bigoplus_{d\geq 0}\mathscr I^d\bigr)\otimes_{\mathscr O_M}\mathscr O_X\notag\\ &=\bigl(\textrm{Proj }\bigoplus_{d\geq 0}\mathscr I^d\bigr)\otimes_{\mathscr O_M}\mathscr O_M/\mathscr I\notag\\ &=\textrm{Proj }\bigl(\bigoplus_{d\geq 0}\mathscr I^d\otimes_{\mathscr O_M}\mathscr O_M/\mathscr I\bigr)\notag\\ &=\textrm{Proj }\bigl(\bigoplus_{d\geq 0}\mathscr I^d/\mathscr I^{d+1}\bigr)\notag\\ &=\mathbb P(N_{X/M}). \end{align}$$ In particular, the condition you mentioned is met: the fibers of $E\to X$ are projective spaces of dimension $$\textrm{rank }N_{X/M}-1=\textrm{codim }(X,M)-1=n-k-1.$$
You can check, as a reference, Hartshorne's book Algebraic Geometry, Chapter II, Proposition 7.14. He uses the phrasing "invertible ideal sheaf": this is what I called "effective Cartier divisor". You can also check Vakil's notes, which has the description above.