My initial thought for this exercise was to first find a potential collection of non-open sets who's union would form $\mathbb{R}$, to satisfy being a $\sigma$-finite measure, and it seemed a nice choice for this would be to define $A_{i}=[-i,i]$.
Now my next thought was that maybe I could define the measure $\mu$ to be the Lebesgue measure for non-open sets (and the empty set), and for open sets simply define the measure to be $+\infty$.
However firstly I am not sure if this is even well defined, and secondly I seem to have a problem where no matter what I try I'm not sure I can fix, which is satisfying $\sigma$-additivity on $\mu$, as if one of the sets is open it instantly makes the sum $+\infty$ yet the union may not necessarily be open, and so may not be $+\infty$.
Any guidance or help would be appreciated thanks!
(I'm pretty new to measure theory so if anything is particularly wrong I apologise)
You could take the counting measure on the rationals, i.e., the measure of a subset equals the number of rationals in it. Then the measure of an open interval is infinite (there are infinitely many rationals in any open interval), but you can write $\mathbb R$ as the countable union of the set of irrational numbers and singleton sets of rationals.