$X$ is $\sigma$-compact then let $X=\cup K_n$ where $K_n$'s are compact. We denote borel $\sigma$-algebra on $X$ by $\mathfrak{m}$.
Let $E\in\mathfrak{m}$. We have to show that $$\mu(E)=\text{sup}\{\mu(K):\ K\subset E,\ K\text{ is compact}\}$$ Choose $\epsilon>0$
I know that (from Rudin's Real & Complex Analysis) there is a closed set $F$ and open set $V$ such that $F\subset E\subset V$ and $\mu(V-F)<\epsilon$
First of I observe that any closed set $F$ is $\sigma$-compact since $F=\cup F\cap K_n$ and $F\cap K_n$ is compact.
Now if $\mu(E)<\infty$, then $\mu(F)<\infty$.
So, $\mu(E-F)\le \mu(V-F)<\epsilon\implies \mu(E)-\mu(F)<\epsilon$ (note we can write $\mu(E-F)=\mu(E)-\mu(F)$ as $\mu(F)$ is finite here).
Hence $\mu(E)-\epsilon<\mu(F)$
This shows that $$\mu(E)=\text{sup}\{\mu(K):\ K\subset E,\ K\text{ is }\sigma\text{-compact}\}$$
Now if $\mu(E)=\infty$ then we must have $\mu(F)=\infty$, (if not then $\mu(E)<\mu(F)+\epsilon<\infty$)
Hence we have $$\mu(E)=\text{sup}\{\mu(K):\ K\subset E,\ K\text{ is }\sigma\text{-compact}\}$$
But we need to prove that $\mu(E)$ can be approximated by compact sets from below. But we have proved for only $\sigma$-compact sets.
It's not explained in Rudin's book. Can anyone complete this proof? Thanks for your help in advance.
Now we only need to prove that $$\mu(E)\le\text{sup}\{\mu(A):\ A\subset E,\ A\text{ is compact}\},$$ i.e. there exists a sequence of compact sets $\{A_i\}$ satisfying $$A_i\subset E,\quad \lim\limits_{i\to\infty}\mu(A_i)=\mu(E).$$
Since $\mu(E)=\text{sup}\{\mu(F):\ F\subset E,\ F\text{ is closed}\}$, let $\{F_i\}$ be the limiting sequence, thus we choose $$A_i:=F_i\cap(\bigcup\limits_{k=1}^i K_k),$$ it is easy to see that $A_i\subset E$ is compact and $\lim\limits_{i\to \infty}\mu(A_i)=\lim\limits_{i\to \infty}\mu(F_i)=\mu(E)$.