(From Rota's paper): Let $P$ be a locally finite poset. The incidence algebra of $P$ is the set of all real-valued functions $f$ of two variables $x$ and $y$ in $P$ with the property that $f(x, y) = 0$ if $x\not\le y$. The sum of two such functions $f$ and $g$, as well as multiplication by scalars, are defined as usual. The product $h = fg$ is defined as follows: $$h(x, y) = \sum_{x\le z \le y} f(x,z) g(z,y).$$ The familiar delta function $\delta(x, y)$ is the identity for this product. The zeta function $\zeta(x,y)$ is defined to be $1$ if $x \le y$ and $0$ otherwise.
We define the inverse $\mu(x, y)$ of the zeta function by induction over the number of elements in the segment $[x, y]$. First, set $\mu(x,x) = 1$ for all $x$ in $P$. Suppose now that $\mu(x, z)$ has been defined for all $z$ in the open segment $[x, y)$. Then set $$\mu(x,y) = -\sum_{x\le z<y}\mu(x,z).$$ Then $\mu$ is inverse to $\zeta$ and it is called the Möbius function of $P$. The importance of the Möbius function is the following fundamental inversion formula: $$g(x) = \sum_{y\le x} f(y) \implies f(x) = \sum_{y \le x}g(y)\mu(y,x)$$ where $f$ is any real-valued function on $P$ which vanishes under some $p$ in $P$.
Question. Can we define the Möbius function if $\le$ is not antisymmetric? This is vague, because what do I mean by "the Möbius function"? I mean something satisfying the inversion formula. The proof of the inversion formula uses the $\zeta$ function, so I guess the answer depends on whether $P$ (now just a preordered set) admits a $\zeta$ function...
Update. Qiaochu's answer explains that the $\zeta$ function always exists for a finite category, but that it may not always be invertible in the algebra. So, how is $\le$'s antisymmetry being used in proving the invertibility of $\zeta$?
You can define the Möbius function in much more generality than this: $P$ can be replaced by a category.
Let me work with a finite category $C$ for simplicity. Then it has a "category algebra" $\mathbb{Z}[C]$ given by the free vector space on its morphisms, where the product of two morphisms is their composition if that is well-defined and zero otherwise. This is a very general construction:
The zeta function in $\mathbb{Z}[C]$ is the sum of every morphism in $C$. It is not always invertible in $\mathbb{Z}[C]$ (unlike the case where $P$ is a poset), but when it is, its inverse is the Möbius function of $C$.
Tom Leinster has written several papers about this; see, for example, this blog post.