I know if we have a compact simply connected 3-manifold with boundary, then the boundary will be $S^2$. The argument is as following
1) $H_1(M)\simeq 0$;
2) Poincare duality $H_2(M,\partial M)\simeq H^1(M)\simeq 0$;
3) Long exact sequence we have
$$\to H_2(M,\partial M)\to H_1(\partial M)\to H_1(M)\to$$
Thus $H_1(\partial M)\simeq 0$ and $\partial M\cong S^2$. Now if we do same argument for 4-dimensional compact simply connected manifold with boundary we cannot say anything about 1st and 2nd homology groups, like we could above. I think $\partial M\cong S^3, S^1\times S^2, L(p,q)$, but the argument doesn't work. How can I show this?
There are many more 3-manifolds that arise as the boundary of compact contractible 4-manifolds than just those, including some Brieskorn spheres like $\Sigma(2,3,13)$. The keyword you want is Mazur manifold. It is my impression that, and I would be surprised if it weren't true, there is not much known about what 3-manifolds arise as the boundary of a Mazur manifold.
Now if you just want simply connected, every orientable 3-manifold bounds a simply connected 4-manifold. First recall the Lickorish-Wallace theorem that every oriented 3-manifold $M$ arises from surgery on a link in $S^3$. This defines a cobordism $W: S^3 \to M$ that's obtained by attaching 2-handles to $S^3 \times I$; this remains simply connected. Now simply cap off the copy of $S^3$ with a ball. This gives a simply connected 4-manifold that bounds $M$.
(Of course, a non-orientable 3-manifold cannot bound a simply connected 4-manifold. For simple connectedness implies the 4-manifold is orientable; and the boundary of an orientable manifold is orientable.)