I am reading about $2$-categories of bordisms, denoted $Cob_2(n)$. This paper by Lurie (see page 10) states that the objects of such a category are closed $(n-2)$-manifolds and and for $M, N \in Cob_2(n)$, the category of maps $Map_{Cob_2(n)}(M, N)$ has objects given by bordisms $B: M\to N$ and morphisms given by oriented diffeomorphism classes of oriented bordisms $X: B\to B'$. However, I don't understand how to define a bordism $X: B\to B'$ when $B$ and $B'$ are already manifolds with boundary.
For instance, we would like the cylinder $B\times [0, 1]$ to be a bordism from $B$ to itself, but its (unoriented) boundary consists of $B\times\{0\} \cup \partial B \times [0, 1] \cup B\times \{1\}$, not just $B\times \{0\} \cup B\times \{1\}$ as one might prefer. Using this example as inspiration, I have come up with the following attempt at a definition:
A bordism $X: B\to B'$ between bordisms $B, B': M \to N$ is an oriented $n$-manifold with corners together with an orientation-preserving diffeomorphism from $\partial X$ to $$\overline{B} \coprod_{\overline{\partial B}}(\partial B \times [0, 1]) \coprod_{\partial B} B'$$ where we have used the given orientation-preserving diffeomorphism $\partial B\simeq \partial B'$.
Is my definition equivalent to the standard one? Or is there something I'm missing here?
Yes, you got that right as far as I can tell. More generally given two integers $0 \le k < n$, one can define a $(n-k)$-category $\mathsf{Cob}_k(n)$ of "$n$-dimensional cobordisms extended down to dimension $k$" (in your case, $k=2$). The definition is as follows:
The image you need to think about is this one, taken from Freed's paper The cobordism hypothesis (Bull. Amer. Math. Soc. (N.S.), 2013, vol. 50, pp. 57--92, DOI):
This represents a cobordism $I \sqcup I \to I \sqcup I$, where $I = [0,1]$ is the interval. The boundary of $I \sqcup I$ is four points, and you see that the cobordism does contain something of the form $\{\text{four points}\} \times [0,1]$. The source is the "bottom" with the two parallel intervals, and the target is the "top".