A representation of a Lie group $G$ on a vector space $V$ is defined to be a Lie group homomorphism $\rho$ such that $$\rho: G \rightarrow GL(V),$$ whereas the representation of a Lie algebra $\mathfrak{g}$ on a vector space $V$ is defined to be a Lie algebra homomorphism $\phi$ such that $$\phi: \mathfrak{g} \rightarrow \text{End}(\phi).$$
Why is the image space for a Lie algebra representation less restricted than for a Lie group (requiring only a homomorphism instead of an isomorphism)?
On
Representations in very broad sense associate (through a homomorphism) for every element of abstract object an element of specimen objects (they are specimens of the same kind as the abstract objects).
For example when the abstract objects are groups, we talk of representation theory of groups: then speciment Objects in this case are $GL(V)$ for various vector spaces $V$ (over various fields).
Now for for an abstract Lie Algebras (over a field $k$) , the SPECIMENS are the very natural concrete examples, viz., $End(V)$ for various $V$ on the same field $k$. $Aut(V)$ DOES NOT satisfy the axioms of a Lie Algebra only End(V) does.
On
It might also be useful to note that if $(V,\rho)$ is a representation of a Lie group $G$, then the elements of $G$ are mapped to automorphisms of $V$, but to obtain the associated Lie algebra homomorphism, you differentiate $\rho$ at the identity element: $\phi=d\rho\colon T_eG\to T_e\mathrm{GL}(V)$ and so one answer to your question is that the reason you get an endomorphism is because the vector space of all endomorphisms is what you get for the tangent space at the identity of $\mathrm{GL}(V)$.
The fact that $T_e\mathrm{GL}(V)=\mathrm{End}(V)$ is easy, it follows immediately from the fact that $\mathrm{GL}(V)$ is open in $\mathrm{End}(V)$, and hence if you start at $I_V$ the identity linear map in $\mathrm{GL}(V)$, then you can move a little bit at least in any direction you like in $\mathrm{End}(V)$. (The fact that $\mathrm{GL}(V)$ is open is also easy -- for example, it follows from the fact that $\mathrm{GL}(V)$ is the complement of the hypersurface $\{A \in \mathrm{End}(V): \mathrm{det}(A)=0\}$.)
It may also be useful to remember that the Lie algebra structure on $T_eG$ comes from identifying $T_eG$ with the Lie algebra of left-invariant vector fields or derivations on $G$, which is a sub-Lie algebra of the Lie algebra of all left-invariant derivations. It follows from the definitions that $\phi = d\rho$ is a Lie algebra homomorphism from left-invariant derivations on $G$ to left-invariant derivations on $\mathrm{GL}(V)$. In that sense, it is not ``just'' a map to $\mathrm{End}(V)$ the space of linear self-maps of $V$, it is a map on left-invariant derivations, and derivations are the infinitesimal analogue of automorphisms.
Up to the question of the centre, which is the kernel of the adjoint representation, you can pretty much see this through the map $\phi$: indeed if $X \in \mathfrak g$ then if we set $a_X = \mathrm{ad}(\phi(X))$ then $a_X$ is a derivation of $\mathrm{End}(V)$, that is, its action on products obeys the Leibniz product rule: $$ \begin{split} a_X(Y_1)Y_2 + Y_1a_X(Y_2) &= (XY_1-Y_1X)Y_2 + Y_1(XY_2-Y_2X) \\ &= XY_1Y_2 - Y_1XY_2 + Y_1 XY_2 + Y_1Y_2X \\ &= X(Y_1Y_2) - (Y_1Y_2)X = a_X(Y_1Y_2) \end{split} $$ so that $\phi$ induces a map $a\colon \mathfrak g \to \mathrm{Der}(\mathrm{End}(V))$ where $$ \mathrm{Der}(\mathrm{End}(V))= \{\alpha \in \mathrm{End}(\mathrm{End}(V)): \alpha(Y_1Y_2) = \alpha(Y_1)Y_2 + Y_1 \alpha(Y_2), \forall Y_1,Y_2 \in \mathrm{End}(V)\}. $$
The short answer: Because $GL(V)$ is a group and $\mathrm{End}(V)$ is a Lie algebra.
In particular, $GL(V)$ has the following:
Whereas $\mathrm{End}(V)$ has the following:
So it is only natural to represent a Lie group in the group $GL(V)$, and a Lie algebra in the algebra $\mathrm{End}(V)$.