Let $X$ be a set. Two endomaps $f,f':X\to X$ are isomorphic if there is a bijection $g:X\to X$ such that $f'=g\circ f\circ g^{-1}$. A bijection $g:X\to X$ satisfying $f=g\circ f\circ g^{-1}$ is called an automorphism of $f$. The identity of $X$ is the trivial automorphism of $f$. An endomap is rigid if it admits no non-trivial automorphism.
Do all sets have a rigid endomap?
Clearly, the existence of a rigid endomap of a given set $X$ depends only on the cardinality $|X|$ of $X$.
We claim:
If $|X|\le2^{\aleph_0}$, then $X$ has a rigid endomap.
Proof:
Let $X$ be a set of cardinality at most $2^{\aleph_0}$, and let us show that $X$ has a rigid endomap $f$. We can assume that $X$ is nonempty.
If $X=\{1,\ldots,n\}$ with $n\ge2$ we set $f(i)=\max\{1,i-1\}$. If $X=\mathbb N$ we set $f(i)=\max\{0,i-1\}$.
Now assume $\aleph_0<|X|\le2^{\aleph_0}$. (We write $|X|$ for the cardinality of $X$.)
Let $I$ be the set of isomorphisms classes of rigid endomaps of $\mathbb N$. We claim
(1) $|I|=2^{\aleph_0}$.
Let us show that (1) implies that $X$ has a rigid endomap. We can assume $$ X=\bigsqcup_{j\in J}X_j $$ where $\bigsqcup$ means "discrete union", where $J$ is a cardinality $|X|$ set of non-isomorphic rigid endomaps of $\mathbb N$, and where $X_j=\mathbb N$ for all $j\in J$. For each $j$ let $f_j$ be an endomap of $X_j$ of type $j$. Then $$ f:=\bigsqcup_{j\in J}f_j $$ (obvious notation) is a rigid endomap of $X$.
It only remains to prove (1).
Let $X_0,X_1,\ldots$ be nonempty finite subsets of $\mathbb N$ such that:
$\bullet\ \mathbb N=X_0\sqcup X_1\sqcup\cdots,$
$\bullet\ X_0=\{0\}$.
For $n\ge1$ let $f_n:X_n\to X_{n-1}$ be a map whose fibers have distinct cardinalities, let $f_0$ be the only endomap of $X_0$, and define $f:\mathbb N\to\mathbb N$ by $f(x)=f_n(x)$ if $x\in X_n$.
Then it is easy to see that $f$ is rigid, and that we have continuum-many isomorphism classes of such endomaps of $\mathbb N$.
The question was answered by YCor on MathOverlow.
I wanted to post a community wiki answer which contained only the above sentence, but the software converted it to a comment. I'm trying again after having added the present sentence and the following excerpt of YCor's answer:
"... there exists (for $X\neq\emptyset$) a rooted tree structure on $X$ whose automorphism group is trivial. Indeed, granting this, and denoting $v_0$ the root, for a vertex $v$ define $f(v)$ as $v_0$ if $v_0=v$, and as the unique vertex in $[v_0,v]$ at distance 1 to $v$ otherwise. Then $f\in X^X$ and its centralizer in $\mathrm{Sym}(X)$ is the automorphism group of the corresponding rooted tree, which is reduced to $\{\mathrm{id}_X\}$."