Actually this question may look simple and basic, but it is about something which bugs me for a long time, since when I took my first calculus classes. The limit $\lim_{x \to a}f(x) = L$ is defined when for each $\epsilon >0$ it is always $|f(x)-L|<\epsilon$ for some $\delta$ neighborhood of $a$, $0 < |x-a| < \delta$. This means that it doesn't matter how small $\epsilon$ gets, we are always able to find $f(x)$ values around $L$ in the neighborhood of $a$ within a $\delta$ distance.
In the definition of limit, the point $a$ is not needed to be included in the domain of $f(x)$. This is the part which I always find strange, when it comes to the definition of "solid" values which actually exist; but only at the "limit" and cannot be evaluated directly. For example, the derivative $f'(x)=\lim_{h\to 0}\dfrac{f(x+h)-f(x)}{h}$. Assume that this limit exists. Here $g(h) =\dfrac{f(x+h)-f(x)}{h}$ is a function of $h$ and it is not defined when $h=0$. The most instinctive thing would be to set $h=0$ and directly get derivative's exact value. But we can never directly put $h=0$ into $g(h)$ and find the exact value of the derivative. Despite this, the derivative clearly exists at $x$ and $h=0$ as a actual value, in the shape of the slope of the tangent line, passing through $(x,f(x))$. In this sense, I interpret here the limit operation as some kind of inference mechanism which fills in the "gap" at $g(h=0)$ by using the following logic : "We cannot exactly evaluate the function $g(h)$ at $h=0$ but we observe that there is a $L$ value such that for each $\epsilon > 0$, we can give a $\delta$ value such that all $g(h)$ values corresponding to $0 < |h| < \delta$ are contained between $L-\epsilon$ and $L+\epsilon$. The function cannot be evaluated as $g(0)=L$ but since all "nearby" values tend to get closer and closer to $L$ constantly, without any interruption, $L=f'(x)$ is the limit and can be used as a placeholder value for undefined $g(h=0)$". The placeholder is the critical word for me here: The derivative's exact value cannot be computed directly at $h=0$ since it results with a division by zero, but since the limit exists at $h=0$, it is considered as the most appropriate value, because all $h$ close to $0$ results in close $g(h)$ values s.t. $|g(h)-L| < \epsilon$.
I apply the same thinking to definite integral as well. Let the Riemann Sum between $a$ and $b$ for a function $f(x)$ is defined like that: $S(\delta x) = \sum_{i=1}^{n}f(x_i)\delta x + Excess$, where $n = \lfloor \frac{b-a}{\delta x} \rfloor$ and $Excess$ is the last partial rectangle in case $\delta x$ does not evenly divide $b-a$. Again, the exact area under $f(x)$ between $a$ and $b$ would be given if would be possible to evaluate $S(\delta x=0)$ but this is not defined, so we evaluate $\lim_{\delta x \to 0^{+}} S(\delta x)=A$ and if this limit exists, we deduce that the exact area under $f(x)$, between $a$ and $b$ would be the value of this limit $A$, since it would be the most natural value for $S(\delta x=0)$ if it were defined at that point.
My question is, is my thinking here true? Can we interpret the limit definitions of derivative and definite integral like that, assuming the limit at zero as the "most natural" value for them, since we cannot compute them analytically at $h=0$ and $\delta x=0$?
Yes. What you call the "most natural" value is the value that ensures continuity (when possible).
A function is continuous at a point when it has no "jump" there; this is formalized by the $\epsilon/\delta$ neighborhoods approach. When a function is undefined at a point, but nearby values are available, it is a natural choice to assign the value that ensures continuity. This is the very definition of a limit: we expect $f(x)$ to be the same as $f(x\pm\delta)$ with a small $\epsilon$ error.
Take some function defined as $f(x)=2+x$ for $x>0$ and undefined otherwise. If you want to extend the definition to $x=0$, the obvious choice will be $f(0)=2$. This extension is possible because you know the values of the function for points arbitrarily close to $0$. On the opposite, an extension of $f$ at $-1$ is not well defined.
This is the exact situation you face when defining the derivative or integral: they can only be defined in terms of limits and are the values that respect continuity. For example, let $f(h)$ be the slope of the chord of the parabola $y=x^2$ between the points $(1,1)$ and $(1+h,(1+h)^2)$: it equals $\frac{\Delta y}{\Delta x}=\frac{1+2h+h^2-1}{1+h-1}=2+h$, for $h>0$. But for $h=0$, this slope is undefined (we are missing a second point to define the straight line). If we are interested in the slope of the tangent, then we use continuity and admit that $f(0)=2$.