I was reading Tom Apostol calculus volume 2 and came across theorem 10.8 (page 350)
My doubt is, What is the definition of the right hand side of the underlined equation that is, What is the definition of $$\int_{a}^b \psi(\vec x,t)dt $$, this integral formally? I know integral of real valued function formally to some extent? But what is the FORMAL DEFINITION OF THIS INTEGRAL?

You're confused because $\psi(\vec{x},\,t)$ depends on multiple variables. But for each value of $\vec{x}$, $\psi(\vec{x},\,t)$ becomes just a function of $t$; you know how to integrate that. This recipe gives each $\varphi(\vec{x})$, thus defining the function $\varphi$.