I am trying to understand irreducible elements, primitive elements, and UFD from Milne's algebraic geometry book. There is something which is confusing me.
In his book which is freely available. He mentions that irreducible elements of A[x] are irreducible elements of A. But that doesn't account for all irreducible elements right ? How does that follow from Gauss lemma? I also don't see why primitive elements are irreducible?

It seems like you have parsed the content of the corollary incorrectly. It says that an polynomial $f\in A[x]$ is irreducible over $A[x]$ if and only if one of these two things hold:
($\leftarrow$) If $f=a$ where $a$ is irreducible, and $a=bc$ then one of $b$ and $c$ is a unit in $A,$ and hence a unit in $A[x].$ Let $f$ be a nonconstant primitive polynomial and $f(x)=g(x)h(x).$ If $g$ and $h$ are both non-constant, then $f$ is reducible over $F[x]$. If $g$ is constant, then since $f$ is primitive, $g$ must be a unit in $A$, hence in $A[x].$
($\rightarrow$) If $f$ is a constant polynomial that is reducible in $A$, then this nontrivial factorization is a nontrivial factorization in $A[x].$ If $f$ is a nonconstant polynomial that is not primitive, then it has a factorization $f(x) = ag(x)$ where $a$ is not a unit in $A$, and this is a nontrivial factorization over $A[x]$. Lastly, if $f(x)$ is primitive but reducible in $F[x],$ then $f(x) = g(x)h(x)$ where $g$ and $h$ are in $F[x]$ and of smaller degree than $f,$ since all nonzero constants are units in $F[x].$ Then, we can clear denominators and factor out gcds from the right hand side and get $$df(x) = c g'(x)h'(x)$$ where $g'(x)$ and $h'(x)$ are primitive polynomials in $A[x]$ ($d$ is the product of the cleared denominators and $c$ is the product of the gcds of the coefficients factored out of the resulting polynomials in $A[x]$). The product $l(x)=g'(x)h'(x)$ is primitive by Gauss's lemma, so $df(x) = c l(x).$ And since $f$ and $l$ are both primitive, $c=ud$ where $u$ is a unit in $A,$ so, cancelling $d$ from both sides of the equations gives $f(x) = ul(x)=u g'(x)h'(x).$ And since the degree of $g'$ and $h'$ are less than the degree of $f,$ this is a nontrivial factorization in $A[x].$