I'm having trouble deriving a finite difference scheme that calculates the second derivative of a function on the boundaries of a non-uniform grid and makes use of a known first derivative at the boundaries.
Notation
Suppose I have $N$ ordered points ${x_i}$ ($i \in 1, 2, 3...N$) where $x_i < x_{i+1}$, and a function $f(x)$ evaluated at each of those $N$ points: $f_i \equiv f(x_i)$.
Suppose I also know the function's derivative at the endpoints: $g_1 \equiv \frac{\partial f}{\partial x}\big|_{i=1}$ and $g_N \equiv \frac{\partial f}{\partial x}\big|_{i=N}$.
Uniform Grid
IF the grid was uniformly spaced (say, with spacing $h$), a possible 1st order accurate scheme at $i=1$ is:
\begin{equation} \frac{\partial^2 f}{\partial x^2}\big|_{i=1} \;=\; \frac{2}{h^2}\left(f_2 - f_1\right) - \frac{2}{h} g_1 \end{equation}
which can be derived from this Taylor expansion: $f(x+h) = f(x) + h f'(x) + \frac{h^2}{2} f''(x)$, which in our notation is: $f_2 = f_1 + h g_1 + \frac{h^2}{2} \frac{\partial^2 f}{\partial x^2}\big|_{i=1}$
Non-Uniform Grid
Is there a finite difference scheme to calculate the second derivative at the endpoints in a form like this?
\begin{equation} \frac{\partial^2 f}{\partial x^2}\big|_{i=1} \;=\; A g_1 + B f_1 + C f_2 + D f_3 + E f_4\text{,} \end{equation}
where $A$, $B$, $C$, $D$ and $E$, are all constants that depend on the grid spacing.
You formula for the uniform grid is only of the first order. $$ \frac{f(h) - f(0)}{h^2} - \frac{2}{h}f'(0) = f''(0) + \frac{h}{3} f'''(0) + O(h^2). $$ It is not possible to construct a formula of the second order using $f(0), f'(0)$ and $f(h)$. Indeed: $$ \frac{A}{h} f'(0) + \frac{B f(h) + C f(0)}{h^2} = \frac{B + C}{h^2} f(0) + \frac{A + B}{h} f'(0) + \frac{B}{2} f''(0) + \frac{Bh}{6} f'''(0) + O(h^2). $$ Solving $$ B + C = 0\\ A + B = 0\\ B = 2\\ $$ gives the only one possible formula and that is of $O(h)$ order.
For the irregular grid (assuming $h_i = O(h)$, i.e. $h$ is some average stepsize) let $\alpha = \frac{h_1}{h} = O(1)$, $\beta = \frac{h_1 + h_2}{h} = O(1)$, $\gamma = \frac{h_1 + h_2 + h_3}{h} = O(1)$ $$ \frac{A}{h} f'(0) + \frac{Bf(0) + Cf(\alpha h) + D f(\beta h) + E f(\gamma h)}{h^2} = \\ = \frac{B+ C+ D+ E}{h^2} f(0) + \frac{A + \alpha C + \beta D + \gamma E}{h} f'(0)+\\ + \frac{\alpha^2 C + \beta^2 D + \gamma^2 E}{2} f''(0)+ \frac{\alpha^3 C + \beta^3 D + \gamma^3 E}{6} hf'''(0)+ O(h^2). $$ Now the second order conditions are $$ B + C + D + E = 0\\ A + \alpha C + \beta D + \gamma E = 0\\ \alpha^2 C + \beta^2 D + \gamma^2 E = 2\\ \alpha^3 C + \beta^3 D + \gamma^3 E = 0\\ $$ These conditions have one more degree of freedom that can be used either for third order accurate formula, or for elimination of $f_4$ ($E = 0$), or for some other properties. In any case, the solution is $$ A = 2\frac{\gamma (\alpha -\gamma ) (\gamma -\beta )}{\alpha \beta }E -2\frac{\alpha +\beta }{\alpha \beta }\\ C = 2\frac{\beta }{\alpha ^2 (\beta -\alpha )}+2\frac{\gamma ^2 (\beta -\gamma)}{\alpha ^2 (\alpha -\beta )}E\\ D = 2\frac{\alpha }{\beta ^2 (\alpha -\beta )}+2\frac{\gamma ^2 (\gamma -\alpha )}{\beta ^2 (\alpha -\beta )}E\\ B = -C - D - E $$ For $E = 0, \alpha = 1, \beta = 2$ we get the formula for the uniform grid $$ f''(0) = -\frac{3}{h}f'(0) + \frac{-7f(0) + 8f(h) -f(2h)}{2h^2} + O(h^2) $$