Let $\mathscr F:=\widetilde{k[t]/(t)}$ be the coherent sheaf on $Spec(k[t])$. I want to calculate the rank of $\mathscr F$. In general, if $\mathscr F$ is a (quasi)coherent sheaf on an irreducible scheme $X$ we define the rank of $\mathscr F$ to be
$$\dim_{\kappa(\eta)}\mathscr F_{\eta}/\mathfrak m_{\eta}\mathscr F_{\eta}$$
where $\eta$ is the generic point of $X$.
In the case we are considering, $X:=Spec(k[t])$, the generic point is equal to $(0)$, hence if we let $S=k[t]\smallsetminus\{0\}$, we have that $\mathscr F_{\eta}=S^{-1}(k[t]/(t))=\{\frac{a}{f}\mid a\in k,f\in k[t]\smallsetminus 0\}$, with addition given by
$$\frac{a}{f}+\frac bg=\frac{ag(0)+bf(0)}{fg}$$
and $k(t)$-module structure given by
$$\left(\frac{f}{g}\right)\cdot\left(\frac{a}{h}\right)=\frac{a\,f(0)}{gh}.$$
Since the maximal ideal $\mathfrak m_{\eta}$ is zero here, we see that the rank is just equal to $\dim_{k(t)}\mathscr F_{\eta}$. I believe this is equal to $1$, since $\mathscr F_{\eta}$ seems to be generated over $k(t)$ by $\frac{1}{1}$.
Now where my trouble lies: this must be wrong, because it is an exercise in Vakil that rank is additive on exact sequences of coherent sheaves, so in particular we could take the sequence
$$0\to\widetilde{k[t]}\overset{\times\,t}{\to}\widetilde{k[t]}\to\widetilde{k[t]/(t)}\to0$$
and conclude that the rank of $\widetilde{k[t]/(t)}$ is zero. So I am wondering, what's wrong with my calculation?
You are correct that $\mathscr F_{\eta}$ is generated over $k(t)$ by $\frac{1}{1}$. However, this doesn't tell you the rank is $1$; it only tells you the rank is at most $1$, since the set $\{\frac{1}{1}\}$ may be linearly dependent in $\mathscr F_{\eta}$. This can only happen if $\frac{1}{1}=0$, but that is in fact exactly what happens. Indeed, remember that when we write elements of a localization as fractions, there is an equivalence relation on these fractions saying that $\frac{a}{f}=\frac{b}{g}$ whenever there is $s\in S$ such that $sga=sfb$. In this case, $\frac{1}{1}=\frac{0}{1}$ by taking $s=t$, since $t\cdot 1\cdot 1=t\cdot 1\cdot 0$ in $k[t]/(t)$. So actually $\frac{1}{1}=0$, and so $\mathscr F_{\eta}=0$