For positive integers x and y, we have that
$$ B(x,y) = \frac{\Gamma(x) \Gamma(y)}{\Gamma(x+y)} = \frac{1}{x} \left( \begin{array}{c} x+y-1 \\ x \end{array} \right)^{-1} . $$
However, $$ \left( \begin{array}{c} a+b \\ a \end{array} \right) = \left(\frac{b+1}{1}\right)\left(\frac{b+2}{2}\right) ... \left(\frac{b+a}{a}\right) \leq (b+1)^a . $$
Therefore, using $a=x$ and $b=y-1$, we have that $$ B(x,y) \geq \frac{y^{-x}}{x}. $$
Is the final inequality true for real values $x \geq 1$ and $y\geq 1$? If so, how is this proved?
OK, I finally figured it out. This is for integer $x\geq 1$ and real valued $y \geq 1$, which is actually all that I needed. Note that you can swap x and y due to the symmetry of the beta function. I believe that the inequality should hold when neither $x$ nor $y$ are integers (but still at least 1), but I did not actually need this result.
We have that \begin{equation*} B(x,y) = \frac{\Gamma(x)\Gamma(y)}{\Gamma(x+y)}. \end{equation*} However, note that $\Gamma(x) = (x-1)! = \frac{x!}{x}$ for integers $x \geq 1$. We also have that $$ \begin{split} \Gamma(x+y) & = (x+y-1) \Gamma(x+y-1) \\ &= (x+y-1)(x+y-2) \Gamma(x+y-2) \\ &= \cdots \\ &= \left({\prod_{j=1}^x (x+y-j)}\right) \Gamma(x+y-x) \\ &= \Gamma(y) \left({\prod_{j=1}^x (y-1+j)}\right) . \end{split} $$ Putting all of this together, we have that (Equation (1)) $$ \begin{split} B(x,y) &= \frac{x! \Gamma(y)}{x \Gamma(y) \left({\prod_{j=1}^x (y-1+j)}\right) } \\ &= \frac{1}{x} \frac{x!}{\left({\prod_{j=1}^x (y-1+j)}\right) } \\ &= \frac{1}{x\left({\prod_{j=1}^x \frac{y-1+j}{j} }\right) } . \end{split} $$ However, for $j\geq 1$, note that $$ \begin{split} y \geq 1 &\implies y-1 \geq 0 \\ &\implies (y-1)(j-1) \geq 0 \\ &\implies 1 - y - j + jy \geq 0 \\ &\implies y-1+j \leq jy \\ &\implies \frac{y-1+j}{j} \leq y . \end{split} $$ We also have that $0 < \frac{y-1+j}{j}$, and therefore we have that $$ {\prod_{j=1}^x \frac{y-1+j}{j} } \leq \prod_{j=1}^x y = y^x . $$ Combining this inequality with Equation (1), we have that $$ B(x,y) \geq \frac{y^{-x}}{x} . \phantom{mmmm}\blacksquare $$