Everything is over $\mathbb C$. Let $\bar{\mathfrak X}\to \mathbb P^N$ be the universal family of hypersurfaces in $\mathbb P^{n+1}$ of degree $d$ and $\mathfrak X \to U$ ($U\subset \mathbb P^N$) be the sub-family of smooth hypersurfaces. I read the following statement which I do not understand:
This induces a monodromy representation $$\rho: \pi_1(U,0)\to {\rm GL}(H^n(\mathfrak X_0,\mathbb Z))$$ where $0\in U$ and $\mathfrak X_0$ is the fiber over $0$. And as $(R^n\pi_* \mathbb Z)_t \cong H^n(\mathfrak X_t,\mathbb Z)$, $\rho$ is also the corresponding representation of $\pi_1$
I have no idea how this map is defined?
I only know the monodromy representation for a cover (and then get some $\pi_1\to S_d$). Are these two notions related?
This is a topological notion : let $f : X \to U$ be a proper surjective submersion. Then, by a theorem of Ehresmann, all the fibers of $f$ are diffeomorphic, moreover there is a covering $U_i$ of $U$ with diffeomorphism $\psi_i : f^{-1}(U_i) \to U_i \times F$ with $\text{pr}_1 \circ \psi_i = f_{|U_i}$, where $F$ is a fiber of $f$.
This means that in particular for any ring $A$, the sheaf $R^kf_*A$ is a local system, with fiber $H^k(F,A)$. So it corresponds to a representation $\rho : \pi_1(U,u_0) \to \rm{GL}$$(H^k(F,A))$.
Concretely, for any path $\gamma \subset U_j$, there is a canonical diffeomorphism $\varphi_{\gamma} : f^{-1}(\gamma(0)) \to f^{-1}(\gamma(1))$, using $\psi_j$. Now, for any loop $\gamma$, splits $\gamma$ into paths $\gamma_i \subset U_i $, and you can compose these diffeomorphisms to get a diffeomorphism $ \varphi_{\gamma_n} \circ \dots \circ \varphi_{\gamma_1} : f^{-1}(u_0) \to f^{-1}(u_0)$. It induces a map on homology : you can check that it is well defined up to homotopy.
Also you have a little typo : this is $(R^n\pi_*\Bbb Z)_{u_0} \cong H^n(\mathfrak{X}_0, \Bbb Z)$ (by proper base change) and not $R^n\pi_* \Bbb Z$ which is a sheaf and not a vector space.
The notion of local system is more general that the monodromy associated to a covering space. Local systems corresponds to vector bundle with a flat connexion, there are many references explaining this, for example the book "Galois groups and fundamental groups" by T. Szamuely.