Let $P$ be the poset $(X,<)$ where $X$ is a set and $<$ is a partial order on $X$. Let $\operatorname{Aut}(P)$ be the group of automorphisms of $X$ that preserve $<$.
Clearly the poset on one element has trivial automorphism group. Any linear order also has trivial automorphism group. We will exclude these possibilities with the following condition:
We will call an element $x\in X$ a total element if $\forall a\in X$, $a<x$ or $a>x$. Let $P'$ be the poset formed by removing the total element $x$ from $P$. If $P'$ is not empty, then $$ \operatorname{Aut}(P)\cong \operatorname{Aut}(P') $$
Therefore, we will exclude any poset with a total element.
Furthermore, we will require that the poset is connected.
What is the smallest connected poset that does not contain a total element that has trivial automorphism group?
Four is the smallest. First note that we four is a lower bound. Indeed, as we do not allow total element, we have at least two minima and two maxima and therefore at least four elements.
Define $P=\{0, 1, \tilde{0}, \tilde{1}\}$ with the partial order $0<1$, $0<\tilde{1}$ and $\tilde{0}<\tilde{1}$. This contains no total element as $0,\tilde{0}$ and $1, \tilde{1}$ are not comparable (and it is clearly connected). This has trivial automorphism group. Indeed, $0$ is comparable to both maximal elements, but $\tilde{0}$ compares only to one maximal element. Thus, $0, \tilde{0}$ get mapped to themselves (as they are minimal elements the only other possibility would be to swap them). The same argument shows that $1, \tilde{1}$ get mapped to themselves.