There are Kirby diagrams for 3- and 4-manifolds which consist of framed links corresponding to 1- and 2-handles attached to a single 0-handle. Any such diagram will give a unique closed manifold since the 3- (and 4-) handles that close the 0-1-2-handlebody are unique.
Is there a good notation for Kirby diagrams for manifolds with boundaries, i.e. where maybe parts of the 0-1-2-handlebody aren't attached to a 3-4-handlebody?
One suggestion for the 4-dimensional case might be drawing a circle with one half in a different colour for 2-handles with boundary. The different colour would mean "boundary here". But I'm not sure whether this is unambiguous.
Recall that if $X$ is an oriented $4$-manifold with boundary $\partial X = \overline{\partial_- X}\amalg \partial_+ X$, then a handlebody decomposition of $X$ relative to $\partial_- X$ is an identification of $X$ with a manifold obtained by attaching handles to $\partial_- X \times I$ along $\partial_- X \times \{1\}$. When $\partial_- X = \varnothing$, we get the usual notion of a handlebody decomposition.
Now by the work of Kirby, we know that any $3$-manifold may be obtained by surgery on a framed link in $S^3$. Surgeries on $S^3$ and handle attaching are related as follows: Adding handles to $S^3 \times I$ along the framed link $L \times \{0\}$ in $S^3 \times \{0\}$ gives a cobordism between the result of surgery on $L$ in $S^3$ and $S^3$. So for any $3$-manifold $Y$ we have a cobordism from $Y$ to $S^3$, i.e. a $4$-manifold with boundary $\bar{Y} \amalg S^3$.
So now we're in the setup for looking at a relative handlebody decomposition of a $4$-manifold $X$ with boundary $Y$. We can look at it as adding handles to the $S^3$ boundary component of the above constructed cobordism between $Y$ and $S^3$. To get a Kirby diagram for this whole thing, we first draw the framed link $L$ that determines the $3$-manifold $Y$, then we draw the Kirby diagram for the handles we attach to the $S^3$ boundary component of the cobordism. To distinguish the components of $L$ from the attaching circles for $2$-handles, we can put brackets around the framing coefficients of $L$, for example write $\langle k \rangle$ for the framing coefficient $k$ in $L$.
You can do some Kirby calculus in these relative Kirby diagrams. See section 5.5 of Gompf and Stipsicz's 4-Manifolds and Kirby Calculus for a brief discussion.