Exercise 3-2 from Calculus on Manifolds by Spivak:
Let $A\subset R^n,\ f:A\rightarrow R$ an integrable (in the sense of Darboux) function. Let $g=f$ except at finitely many points. Prove that $g$ is also integrable and $\int_Af=\int_A g$.
Note that there is a similar question show that $g$ is integrable, with $f=g$ except in finite set and $f$ integrable but the answers assume the knowledge of measure theory whereas Spivak doesn't.
I guess I need to use the criterion saying that $f$ is integrable iff there is a partition $P$ of $A$ such that $U(f,P)-L(f,P)< \epsilon$ for any $\epsilon < 0$. But I don't know how to apply it to both functions. I thought about considering $f-g$ (which should be integrable except finitely many points), but Spivak doesn't even state that the sum of two integrable functions is integrable, so perhaps I'm not supposed to use this. (Even if I consider $f-g$, I don't know how to proceed).
Your idea of considering the difference is a good one, since your problem is then seen to be equivalent to proving that a function which is $0$ except at finitely many points is integrable and must have integral equal to $0$ (try to understand why if this isn't clear)
In order to prove this statement, taking partitions with small diameter will show that you can let your upper and lower sums be as near zero as you want (they will be bounded by $\pm N \cdot\mathrm{diam}(P)^n\cdot\max |f|$, where $N$ is the number of points where the function is not zero).
Now, to see why $g$ is integrable, notice that $g=(g-f)+f$.
Therefore, $\int g=\int(g-f)+\int f=\int f$.
You mention in the question that the book does not say that sum of integrable functions is integrable before this point. If you are uncomfortable with using this fact, try to adapt the proof above to avoid using it. The core idea is in the second paragraph of this answer.